Skip to content

12.9 HEPS

Overview

Resource type Resource amount Notes
Login nodes
HTCondor
Slurm
Home directory
Work directory

12.9.1 New Users

  • Apply for a computing cluster account.

Refer to the account application chapter. After approval from the HEPS experiment contact, users can obtain an AFS account.

  • Initialize the computing environment.

After logging in for the first time, run:

$ln-s/sharefs/heps/data/opt/config/bash_profile~/.bash_profile$source~/.bash_profile

12.9.2 Computing Resources

HEPS computing resources include the following nodes:

Node Logical CPU cores per node Node type Scheduler Notes
accap002.ihep.ac.cn 28 Login node
accap001.ihep.ac.cn - accap009.ihep.ac.cn 56 Compute node Slurm Excluding accap002
accap010.ihep.ac.cn - accap015.ihep.ac.cn 72 Compute node Slurm
accap016.ihep.ac.cn - accap017.ihep.ac.cn 88 Compute node Slurm
accap019.ihep.ac.cn - accap053.ihep.ac.cn 36 Compute node Slurm SSH login is not allowed

After the 2025 summer system upgrade, all compute servers were upgraded to ALMA-9. Users need to reset the login configuration:

$rm~/.bash_profile$ln-s/sharefs/heps/data/opt/config/bash_profile~/.bash_profile

Log out and log in again for the setting to take effect.

For VNC access toaccap002, disable screen blanking after login:Applications -> System Tools -> Settings -> Power -> Blank Screen Never.

12.9.3 Storage Directories

Directory Quota / file limit Quota command
/afs/ihep.ac.cn/users/a-z/username 500 MB / none fs listquota /afs/ihep.ac.cn/users/a-z/username
/workfs2/heps/username 5 GB / 50,000 files lfs quota -u username /workfs2
/scratchfs/heps/username 500 GB / none; data kept for two weeks lfs quota -u username /scratchfs
/sharefs/heps/user/username 3 TB / none lfs quota -u username /sharefs

Some users may have personal directories under/workfs2/csns/username,/workfs2/others/username,/scratchfs/csns/username, or/scratchfs/others/username.

Usage recommendations:

  • Keep only important personal configuration files under/afs.
  • /workfs2can be read and written on login nodes, but Slurm jobs on compute nodes cannot create or modify files there.
  • Use/sharefsfor important results and data that need to be read and written by login and compute nodes.
  • Use/scratchfsfor temporary large files and intermediate results.
  • Back up important results to your personal computer in time.

12.9.4 Programs and Examples

HEPS computing programs are installed under/sharefs/heps/data/APcodesand/sharefs/heps/data/opt/. See/sharefs/heps/data/APcodes/README.mdfor a detailed program list.

12.9.4.1 Run Parallel Programs Through Slurm

Submit jobs after logging in tolxlogin.ihep.ac.cnoraccap002.ihep.ac.cn. The recommended number of parallel cores is a multiple of 36.

Examplesbatch.sh:

#! /bin/bash#SBATCH --partition=heps#SBATCH --account=heps#SBATCH --job-name=DA#SBATCH --qos=regular#SBATCH --output=slurm-%j.txt#SBATCH --ntasks=36mpiexec-np36PelegantparTrack2.ele

Submit the job:

$sbatchsbatch.sh

For debugging, use thehepsdebugqueue. It allows jobs with no more than 36 tasks and a maximum run time of 30 minutes:

#SBATCH --qos=hepsdebug

Useful commands:

$sinfo-pheps$squeue$scanceljobid

12.9.4.3 Run Licensed MATLAB

HEPS MATLAB is installed under/afs/ihep.ac.cn/soft/hepsap/matlab. Log in toaccap002.ihep.ac.cnby SSH or VNC and run:

$matlab

The installed MATLAB supports parallel computing with up to 512 parallel cores. MATLAB parallel jobs do not need Slurm; request parallel cores inside MATLAB:

>>parpool(512)

12.9.4.4 Run Licensed Mathematica

HEPS Mathematica is installed under/sharefs/heps/data/opt/Mathematica/12.0:

$/sharefs/heps/data/opt/Mathematica/12.0/bin/mathematica

The license supports up to 17 concurrent Mathematica main programs and up to 136 cores for parallel computing. Examples are available under:

/sharefs/heps/data/opt/examples/MathematicaExamples/

For single-node parallel jobs onaccap002, add the following before the parallel section:

LaunchKernels[20]

For cross-node parallel jobs, submit resources through Slurm. Non-interactive Wolfram Language scripts can be run with:

math-scripttest.wls

Interactive jobs can first allocate resources:

salloc-pheps--qos=regular--ntasks=72-t01:00:00

Then launch Mathematica onaccap002and run the parallel computation.

12.9.5 FAQ

12.9.5.1 Cannot Write to the AFS Personal Directory

Users need to renew tokens about every 10 hours:

$echo'password'|kintusername$aklog-d

You can also create a scheduled task:

$sh/sharefs/heps/data/opt/config/generate_crontab_klog.shusernamepassword

If the AFS directory is full, check quota and disk usage:

$fslistquota/afs/ihep.ac.cn/users/a-z/username$du-h/afs/ihep.ac.cn/users/a-z/username

MATLAB users should clear cache files regularly:

$rm~/.felix

12.9.5.2 Cannot Log In by SSH from a Linux Host

If SSH reportsHost key verification failed, remove the lines containing the target host, such asaccap002, from~/.ssh/known_hosts.

12.9.5.3 MATLAB Plots Do Not Display Correctly on accap002

Start MATLAB with software OpenGL:

$matlab-softwareopengl
AI 助手
快速提问 ×

有问题?输入后跳转到对话页继续交流

Baidu
map