12.12 HERD¶
Overview¶
| Service Type | Resource | Notes |
|---|---|---|
| Login node | lxlogin.ihep.ac.cn | |
| Computing resources | 804 cores; idle or remote resources can be shared | HTCondor: HepJob |
| VNC and related services | herdtb01 | Use containers |
| Home directory | /afs/ihep.ac.cn/users/ | 500 MB |
| Working directories | /herdfs/data, /herdfs/user | See 12.12.2 |
12.12.1 Account Application¶
- Apply for a computing cluster account.
Refer toChapter 3. After approval from the HERD experiment computing contact person, the user can obtain an AFS account.
12.12.2 Storage Directories¶
Available storage directories are divided into two types: public storage provided by the computing cluster and HERD project-owned storage. Public storage includes the personal user directory under AFS, with a 500 MB quota.
HERD project-owned storage has two parts:
| Storage Path | Quota | Features | Usage Notes |
|---|---|---|---|
| /herdfs/data | Dynamically controlled according to project needs | Working directory, large capacity | User directories are created underUSER; do not create directories directly underdatawithout permission |
| /herdfs/user | User directory | Stores personal files |
12.12.3 Job Management¶
HERD uses theHepJobtoolset for job management. HepJob wraps native HTCondor commands and is optimized for the IHEP cluster. Users should use HepJob instead of native HTCondor commands.
Environment Setup¶
Add the HepJob command directory toPATH:
Job Submission:hep_sub¶
Common parameters:
| Parameter | Description |
|---|---|
-g herd |
Use computing resources of the HERD group |
-o |
Standard output file; default isjob_name.out |
-e |
Standard error file; default isjob_name.err |
-n |
SubmitNjobs in a batch |
-mem |
Request memory in MB |
-np |
Request number of cores, used for MPI jobs |
-os |
Specify operating system version, such as SL7 |
Note: The job script, such as
job.sh, must have executable permission. Usechmod +x job.shto grant permission.
Job Query:hep_q¶
# View all jobs of the current userhep_q# View jobs of a specified userhep_q-u # View a specified job IDhep_q-i3745232.1# View running jobshep_q-run
Output fields:JOBIDis the job ID,OWNERis the owner,SUBMITTEDis the submission time,RUN_TIMEis the runtime,STis the status (I= idle,R= running),PRIis the priority,SIZEis virtual memory usage, andCMDis the job program name.
Job Deletion:hep_rm¶
# Delete a specified jobhep_rm3745232.1# Delete a whole job clusterhep_rm3745232# Delete multiple jobs at the same timehep_rm37452323745233.0# Delete all jobs of the current userhep_rm-a# Force deletion of jobs in X statehep_rm-a-forcex
All HepJob commands support
-hfor help, for examplehep_sub -h.
Use Local Resources¶
Submit jobs to the IHEP local cluster, which is the default method:
If the-rmtparameter is not specified, jobs are scheduled by default to IHEP local HTCondor compute nodes (herdws001-012, 804 cores in total).
Use Dongguan Resources¶
Submit jobs to the Dongguan site by specifying the remote site with-rmt:
For query and deletion commands, add the-rmtparameter to the default commands in the same way.
The Dongguan site provides additional computing resources and is suitable when the local cluster is busy or when larger-scale computing is needed.
12.12.4 Grid Authentication¶
HERD usesINDIGO IAMfor grid authentication management:https://herd-iam.ihep.ac.cn.
IAM supports OIDC/WLCG Token, SAML/eduGain, IHEP-SSO, and X.509 certificate plus VOMS proxy authentication.
Register an IAM Account¶
- Visithttps://herd-iam.ihep.ac.cn.
- Choose a login method. IHEP-SSO or eduGain is recommended.
- Fill in the registration form and wait for administrator approval.
- After approval, log in to IAM to obtain a Token or VOMS proxy.
Obtain Token / VOMS Proxy¶
- WLCG Token: IAM supports WLCG JWT Profile by default and can generate it directly from the web interface.
- VOMS Proxy: VOMS AA service address is
herd-iam.ihep.ac.cn:8443, and the VO name isherd.
12.12.5 Grid Data Management¶
HERD grid data management is based onRucio:https://herd-rucio.ihep.ac.cn.
Rucio is used for HERD dataset registration, distribution, replication, and lifecycle management. It is currently open to production users.
Environment Setup¶
Common Operations¶
# List available RSEs (storage endpoints)ruciolist-rses# List datasetsruciolist-dids--filtertype=DATASET# Download dataruciodownload
Storage Endpoints¶
HERD grid storage endpoints are provided by collaborating sites. Administrators should contactzhangxuantong@ihep.ac.cnto register SEs in Rucio.
12.12.6 Software and Environment¶
HERD offline softwareHERDOShas been deployed on CVMFS. It includes a complete toolchain for detector simulation (Geant4), digitization, reconstruction, and analysis.
Use Through CVMFS¶
# CentOS 7 / AlmaLinux 9source/cvmfs/herd.ihep.ac.cn/HERDOS/centos7_amd64_gcc850/Release/v00-10/ExternalLibs/bashrc.sh
| Version | Environment Command |
|---|---|
| v00-10 | source /cvmfs/herd.ihep.ac.cn/HERDOS/centos7_amd64_gcc850/Release/v00-10/setup.sh |
| v00-06 | source /cvmfs/herd.ihep.ac.cn/HERDOS/SLC7/Pre-release/v00-06/setup.sh |
| v00-05 | source /cvmfs/herd.ihep.ac.cn/HERDOS/SLC7/Pre-release/v00-05/setup.sh |
For detailed release notes, see/cvmfs/herd.ihep.ac.cn/HERDOS/README.
Build from Source¶
gitclonegit@code.ihep.ac.cn:herdos/offline.gitcdofflinegitcheckoutv00-10./build_RelWithDebInfo.shsourceinstall/setup.sh
12.12.7 Other Management Services¶
HERD provides the following auxiliary management services on theherdtb01node. These services are deployed in containers.
Other Services¶
| Service | Description |
|---|---|
| VNC | Remote graphical desktop for ROOT/Geant4 visualization and related use cases |
To add more services, contact the HERD computing administrator.
12.12.8 Service Support¶
HERD computing administrators:
| Administrator | |
|---|---|
| Xiaowei Jiang | jiangxw@ihep.ac.cn |
| Xuantong Zhang | zhangxuantong@ihep.ac.cn |
有问题?输入后跳转到对话页继续交流