Skip to content

12.14 PWFA

Overview

  • The PWFA experiment rents computing resources from the public computing platform of the Slurm cluster to complete its computing tasks. Available resources are as follows:
Resource Type Resource Description
Login node pvm057 Dedicated login node for PWFA
Slurm Compute nodes of the spub partition spub[001-016] and gpu036, with the highest usage priority
home directory /ihepfs/pwfa/user/{username} Dedicated storage
working directory /ihepfs/pwfa/data/ Dedicated storage

12.14.1 A Brief Introduction of the Slurm Cluster

12.14.1.1 register

12.14.1.2 Log in

  • sshcommand is used to log in the cluster,stands for your real user name.
# lxlogin: shared login nodesssh@lxlogin.ihep.ac.cn# pvm057 : dedicated login nodes for group pwfassh@pvm057.ihep.ac.cn

12.14.1.3 Cluster usage

  • There are two ways to use cluster
  • Interactive mode : Log into the node pvm057 and debug your program.
  • Job mode : submit slurm jobs with commandsbatchand run your fine-tuned program.
    • Jobs can be submitted to partition pwfadebug and spub.
  • More detailed usage can be found in theSlurm User Manual

12.14.2 Available computing resources

  • The PWFA experiment is renting computing services of the spub computing platform.
  • Available computing resources are in the Slurm cluster and can be classified into three catagories.
Partition Node Function Description
login pvm057 Dedicated login node Log in with commandssh, interactive mode
pwfadebug pvm058 Dedicated worker node Submit jobs with commandsbatch, job mode
spub spub[001-016],gpu036 Shared worker nodes Submit jobs withsbatchcommand, job mode

12.14.3 Accessable storage

  • /ihepfs is used as data or home directory for the PWFA experiment, details can be found in the following table.
  • stands for your cluster user name
Directory Functionality quota limit
/ihepfs/pwfa/user/ home directory 100GB,200,000 files
/ihepfs/pwfa/data/ data directory 1TB,200,000 files

12.14.4 Installed software

Software Installation path Code source
fbpic /cvmfs/slurm.ihep.ac.cn/centos7.9/anaconda3/envs/fbpic github fbpic
QuickPIC /cvmfs/pwfa.ihep.ac.cn/software/QuickPIC-OpenSource-dev/source/qpic.e github QuickPIC

12.14.5 Slurm job samples

  • Users of the PWFA expriments can submit to partition pwfadebug and spub
  • To submit jobs, some job parameters including partition, account and qos must be provided.
partition account qos
pwfadebug pwfa spubpwfa
spub pwfa spubpwfa

12.14.5.1 QuickPIC

  • The following sample job scriptqpic_sample_job.shshow how to submit a QuickPIC job to partition spub.
$catqpic_sample_job.sh#! /bin/bash#SBATCH --partition=spub#SBATCH --account=pwfa#SBATCH --qos=spubpwfa#SBATCH --ntasks=32#SBATCH --mem-per-cpu=4GB#SBATCH --job-name=quickPICmoduleuse/cvmfs/slurm.ihep.ac.cn/alma9/modulefilesmoduleloadquickpic/developexportUCX_NET_DEVICES=mlx5_0:1exportOMP_NUM_THREADS=1echo$SLURM_NTASKSmpirun-np$SLURM_NTASKS/cvmfs/slurm.ihep.ac.cn/alma9/software-gcc11/QuickPIC-OpenSource-dev/source/qpic.e

Explanation:

  • module use /cvmfs/slurm.ihep.ac.cn/alma9/modulefilesandmodule load quickpic/developare used to load the software environment required to run QuickPIC
  • UCX_NET_DEVICESandOMP_NUM_THREADSset the network device and thread environment variables required to run QuickPIC
  • qpic.eis the excecutable program of QuickPIC, the defauld input file isqpinput.jsonwhich should be put in the current directory to submit jobs.
  • Details aboutqpinput.jsoncan be found with the link ofInput File for QuickPIC.
  • To submit QuickPIC job, type the following command
$sbatchqpic_sample_job.sh

12.14.5.2 fbpic

  • The following job scriptfbpic_cpu.shshows how to submit a fbpic job to partition spub.
$catfbpic_cpu.sh#! /bin/bash#=====================================================# FBPIC CPU job on one spub CPU node## Usage:# sbatch fbpic_cpu.sh# sbatch fbpic_cpu.sh fbpic_script.py## Tune --cpus-per-task and THREADS together if you do# not want to use all 56 CPU cores on the node.#=====================================================#SBATCH --partition=spub#SBATCH --account=pwfa#SBATCH --qos=spubpwfa#SBATCH --job-name=fbpic_cpu#SBATCH --nodes=1#SBATCH --ntasks=1#SBATCH --cpus-per-task=56#SBATCH --mem=240G#SBATCH --output=fbpic_cpu_%j.out#SBATCH --error=fbpic_cpu_%j.errset-euopipefailsource/ihepfs/pwfa/data/public/env_FBPIC.shexportNUMBA_CACHE_DIR=/ihepfs/pwfa/data/${USER}/cachemkdir-p"${NUMBA_CACHE_DIR}"# FBPIC CPU multi-threading. Keep these equal to --cpus-per-task.THREADS="${SLURM_CPUS_PER_TASK:-56}"exportMKL_NUM_THREADS="${THREADS}"exportNUMBA_NUM_THREADS="${THREADS}"exportOMP_NUM_THREADS="${THREADS}"exportOPENBLAS_NUM_THREADS="${THREADS}"exportNUMEXPR_NUM_THREADS="${THREADS}"# Keep all worker threads on the CPU cores allocated by Slurm.exportOMP_PROC_BIND=closeexportOMP_PLACES=coresFBPIC_SCRIPT="${1:-fb-in.py}"echo"Job ID:${SLURM_JOB_ID}"echo"Node list:${SLURM_JOB_NODELIST}"echo"FBPIC script:${FBPIC_SCRIPT}"echo"CPU threads:${THREADS}"echo"Memory requested:${SLURM_MEM_PER_NODE:-240G}"srun--cpu-bind=corespython"${FBPIC_SCRIPT}"

Explanation:

  • This script runs an fbpic CPU job on one spub CPU node. Submit it withsbatch fbpic_cpu.sh [fbpic_script.py]; iffbpic_script.pyis not specified,fb-in.pyis run by default.
  • If you do not want to use all 56 CPU cores on the node, adjust--cpus-per-taskand the script'sTHREADSvariable together.
  • source /ihepfs/pwfa/data/public/env_FBPIC.shis used to load the environment required to run fbpic.
  • NUMBA_CACHE_DIRsets numba's cache directory, avoiding cache conflicts when multiple users share the same installation.
  • MKL_NUM_THREADS,NUMBA_NUM_THREADS,OMP_NUM_THREADS,OPENBLAS_NUM_THREADS, andNUMEXPR_NUM_THREADSset the number of threads used by each of fbpic's computing libraries; keep these equal to--cpus-per-task.
  • OMP_PROC_BINDandOMP_PLACESbind worker threads to the CPU cores allocated by Slurm, avoiding thread migration that would hurt performance.
  • To submit the job, run the following command
$sbatchfbpic_cpu.sh
AI 助手
快速提问 ×

有问题?输入后跳转到对话页继续交流

Baidu
map