12.14 PWFA¶
Overview¶
- The PWFA experiment rents computing resources from the public computing platform of the Slurm cluster to complete its computing tasks. Available resources are as follows:
| Resource Type | Resource | Description |
|---|---|---|
| Login node | pvm057 | Dedicated login node for PWFA |
| Slurm | Compute nodes of the spub partition | spub[001-016] and gpu036, with the highest usage priority |
| home directory | /ihepfs/pwfa/user/{username} | Dedicated storage |
| working directory | /ihepfs/pwfa/data/ | Dedicated storage |
12.14.1 A Brief Introduction of the Slurm Cluster¶
12.14.1.1 register¶
- Seeapplication for your cluster user nameto get registered in the cluster.
12.14.1.2 Log in¶
sshcommand is used to log in the cluster,stands for your real user name.
# lxlogin: shared login nodesssh@lxlogin.ihep.ac.cn # pvm057 : dedicated login nodes for group pwfassh@pvm057.ihep.ac.cn
12.14.1.3 Cluster usage¶
- There are two ways to use cluster
- Interactive mode : Log into the node pvm057 and debug your program.
- Job mode : submit slurm jobs with command
sbatchand run your fine-tuned program.- Jobs can be submitted to partition pwfadebug and spub.
- More detailed usage can be found in theSlurm User Manual
12.14.2 Available computing resources¶
- The PWFA experiment is renting computing services of the spub computing platform.
- Available computing resources are in the Slurm cluster and can be classified into three catagories.
| Partition | Node | Function | Description |
|---|---|---|---|
| login | pvm057 | Dedicated login node | Log in with commandssh, interactive mode |
| pwfadebug | pvm058 | Dedicated worker node | Submit jobs with commandsbatch, job mode |
| spub | spub[001-016],gpu036 | Shared worker nodes | Submit jobs withsbatchcommand, job mode |
12.14.3 Accessable storage¶
- /ihepfs is used as data or home directory for the PWFA experiment, details can be found in the following table.
stands for your cluster user name
| Directory | Functionality | quota limit |
|---|---|---|
/ihepfs/pwfa/user/ |
home directory | 100GB,200,000 files |
/ihepfs/pwfa/data/ |
data directory | 1TB,200,000 files |
12.14.4 Installed software¶
| Software | Installation path | Code source |
|---|---|---|
| fbpic | /cvmfs/slurm.ihep.ac.cn/centos7.9/anaconda3/envs/fbpic | github fbpic |
| QuickPIC | /cvmfs/pwfa.ihep.ac.cn/software/QuickPIC-OpenSource-dev/source/qpic.e | github QuickPIC |
12.14.5 Slurm job samples¶
- Users of the PWFA expriments can submit to partition pwfadebug and spub
- To submit jobs, some job parameters including partition, account and qos must be provided.
| partition | account | qos |
|---|---|---|
| pwfadebug | pwfa | spubpwfa |
| spub | pwfa | spubpwfa |
12.14.5.1 QuickPIC¶
- The following sample job script
qpic_sample_job.shshow how to submit a QuickPIC job to partition spub.
$catqpic_sample_job.sh#! /bin/bash#SBATCH --partition=spub#SBATCH --account=pwfa#SBATCH --qos=spubpwfa#SBATCH --ntasks=32#SBATCH --mem-per-cpu=4GB#SBATCH --job-name=quickPICmoduleuse/cvmfs/slurm.ihep.ac.cn/alma9/modulefilesmoduleloadquickpic/developexportUCX_NET_DEVICES=mlx5_0:1exportOMP_NUM_THREADS=1echo$SLURM_NTASKSmpirun-np$SLURM_NTASKS/cvmfs/slurm.ihep.ac.cn/alma9/software-gcc11/QuickPIC-OpenSource-dev/source/qpic.e
Explanation:
module use /cvmfs/slurm.ihep.ac.cn/alma9/modulefilesandmodule load quickpic/developare used to load the software environment required to run QuickPICUCX_NET_DEVICESandOMP_NUM_THREADSset the network device and thread environment variables required to run QuickPICqpic.eis the excecutable program of QuickPIC, the defauld input file isqpinput.jsonwhich should be put in the current directory to submit jobs.- Details about
qpinput.jsoncan be found with the link ofInput File for QuickPIC.
- To submit QuickPIC job, type the following command
12.14.5.2 fbpic¶
- The following job script
fbpic_cpu.shshows how to submit a fbpic job to partition spub.
$catfbpic_cpu.sh#! /bin/bash#=====================================================# FBPIC CPU job on one spub CPU node## Usage:# sbatch fbpic_cpu.sh# sbatch fbpic_cpu.sh fbpic_script.py## Tune --cpus-per-task and THREADS together if you do# not want to use all 56 CPU cores on the node.#=====================================================#SBATCH --partition=spub#SBATCH --account=pwfa#SBATCH --qos=spubpwfa#SBATCH --job-name=fbpic_cpu#SBATCH --nodes=1#SBATCH --ntasks=1#SBATCH --cpus-per-task=56#SBATCH --mem=240G#SBATCH --output=fbpic_cpu_%j.out#SBATCH --error=fbpic_cpu_%j.errset-euopipefailsource/ihepfs/pwfa/data/public/env_FBPIC.shexportNUMBA_CACHE_DIR=/ihepfs/pwfa/data/${USER}/cachemkdir-p"${NUMBA_CACHE_DIR}"# FBPIC CPU multi-threading. Keep these equal to --cpus-per-task.THREADS="${SLURM_CPUS_PER_TASK:-56}"exportMKL_NUM_THREADS="${THREADS}"exportNUMBA_NUM_THREADS="${THREADS}"exportOMP_NUM_THREADS="${THREADS}"exportOPENBLAS_NUM_THREADS="${THREADS}"exportNUMEXPR_NUM_THREADS="${THREADS}"# Keep all worker threads on the CPU cores allocated by Slurm.exportOMP_PROC_BIND=closeexportOMP_PLACES=coresFBPIC_SCRIPT="${1:-fb-in.py}"echo"Job ID:${SLURM_JOB_ID}"echo"Node list:${SLURM_JOB_NODELIST}"echo"FBPIC script:${FBPIC_SCRIPT}"echo"CPU threads:${THREADS}"echo"Memory requested:${SLURM_MEM_PER_NODE:-240G}"srun--cpu-bind=corespython"${FBPIC_SCRIPT}"
Explanation:
- This script runs an fbpic CPU job on one spub CPU node. Submit it with
sbatch fbpic_cpu.sh [fbpic_script.py]; iffbpic_script.pyis not specified,fb-in.pyis run by default.- If you do not want to use all 56 CPU cores on the node, adjust
--cpus-per-taskand the script'sTHREADSvariable together.source /ihepfs/pwfa/data/public/env_FBPIC.shis used to load the environment required to run fbpic.NUMBA_CACHE_DIRsets numba's cache directory, avoiding cache conflicts when multiple users share the same installation.MKL_NUM_THREADS,NUMBA_NUM_THREADS,OMP_NUM_THREADS,OPENBLAS_NUM_THREADS, andNUMEXPR_NUM_THREADSset the number of threads used by each of fbpic's computing libraries; keep these equal to--cpus-per-task.OMP_PROC_BINDandOMP_PLACESbind worker threads to the CPU cores allocated by Slurm, avoiding thread migration that would hurt performance.
- To submit the job, run the following command
AI 助手
快速提问
×
有问题?输入后跳转到对话页继续交流