Skip to content

12.12 HERD

Overview

Service Type Resource Notes
Login node lxlogin.ihep.ac.cn
Computing resources 804 cores; idle or remote resources can be shared HTCondor: HepJob
VNC and related services herdtb01 Use containers
Home directory /afs/ihep.ac.cn/users/ 500 MB
Working directories /herdfs/data, /herdfs/user See 12.12.2

12.12.1 Account Application

  • Apply for a computing cluster account.

Refer toChapter 3. After approval from the HERD experiment computing contact person, the user can obtain an AFS account.

12.12.2 Storage Directories

Available storage directories are divided into two types: public storage provided by the computing cluster and HERD project-owned storage. Public storage includes the personal user directory under AFS, with a 500 MB quota.

HERD project-owned storage has two parts:

Storage Path Quota Features Usage Notes
/herdfs/data Dynamically controlled according to project needs Working directory, large capacity User directories are created underUSER; do not create directories directly underdatawithout permission
/herdfs/user User directory Stores personal files

12.12.3 Job Management

HERD uses theHepJobtoolset for job management. HepJob wraps native HTCondor commands and is optimized for the IHEP cluster. Users should use HepJob instead of native HTCondor commands.

Environment Setup

Add the HepJob command directory toPATH:

exportPATH=/cvmfs/common.ihep.ac.cn/software/hepjob/bin:$PATH

Job Submission:hep_sub

hep_sub-gherdjob.sh

Common parameters:

Parameter Description
-g herd Use computing resources of the HERD group
-o Standard output file; default isjob_name.out
-e Standard error file; default isjob_name.err
-n SubmitNjobs in a batch
-mem Request memory in MB
-np Request number of cores, used for MPI jobs
-os Specify operating system version, such as SL7

Note: The job script, such asjob.sh, must have executable permission. Usechmod +x job.shto grant permission.

Job Query:hep_q

# View all jobs of the current userhep_q# View jobs of a specified userhep_q-u# View a specified job IDhep_q-i3745232.1# View running jobshep_q-run

Output fields:JOBIDis the job ID,OWNERis the owner,SUBMITTEDis the submission time,RUN_TIMEis the runtime,STis the status (I= idle,R= running),PRIis the priority,SIZEis virtual memory usage, andCMDis the job program name.

Job Deletion:hep_rm

# Delete a specified jobhep_rm3745232.1# Delete a whole job clusterhep_rm3745232# Delete multiple jobs at the same timehep_rm37452323745233.0# Delete all jobs of the current userhep_rm-a# Force deletion of jobs in X statehep_rm-a-forcex

All HepJob commands support-hfor help, for examplehep_sub -h.

Use Local Resources

Submit jobs to the IHEP local cluster, which is the default method:

hep_sub-gherdjob.sh

If the-rmtparameter is not specified, jobs are scheduled by default to IHEP local HTCondor compute nodes (herdws001-012, 804 cores in total).

Use Dongguan Resources

Submit jobs to the Dongguan site by specifying the remote site with-rmt:

hep_sub-gherd-rmtjob.sh

For query and deletion commands, add the-rmtparameter to the default commands in the same way.

The Dongguan site provides additional computing resources and is suitable when the local cluster is busy or when larger-scale computing is needed.

12.12.4 Grid Authentication

HERD usesINDIGO IAMfor grid authentication management:https://herd-iam.ihep.ac.cn.

IAM supports OIDC/WLCG Token, SAML/eduGain, IHEP-SSO, and X.509 certificate plus VOMS proxy authentication.

Register an IAM Account

  1. Visithttps://herd-iam.ihep.ac.cn.
  2. Choose a login method. IHEP-SSO or eduGain is recommended.
  3. Fill in the registration form and wait for administrator approval.
  4. After approval, log in to IAM to obtain a Token or VOMS proxy.

Obtain Token / VOMS Proxy

  • WLCG Token: IAM supports WLCG JWT Profile by default and can generate it directly from the web interface.
  • VOMS Proxy: VOMS AA service address isherd-iam.ihep.ac.cn:8443, and the VO name isherd.

12.12.5 Grid Data Management

HERD grid data management is based onRucio:https://herd-rucio.ihep.ac.cn.

Rucio is used for HERD dataset registration, distribution, replication, and lifecycle management. It is currently open to production users.

Environment Setup

source/cvmfs/herd.ihep.ac.cn/grid/env.sh

Common Operations

# List available RSEs (storage endpoints)ruciolist-rses# List datasetsruciolist-dids--filtertype=DATASET# Download dataruciodownload

Storage Endpoints

HERD grid storage endpoints are provided by collaborating sites. Administrators should contactzhangxuantong@ihep.ac.cnto register SEs in Rucio.

12.12.6 Software and Environment

HERD offline softwareHERDOShas been deployed on CVMFS. It includes a complete toolchain for detector simulation (Geant4), digitization, reconstruction, and analysis.

Use Through CVMFS

# CentOS 7 / AlmaLinux 9source/cvmfs/herd.ihep.ac.cn/HERDOS/centos7_amd64_gcc850/Release/v00-10/ExternalLibs/bashrc.sh
Version Environment Command
v00-10 source /cvmfs/herd.ihep.ac.cn/HERDOS/centos7_amd64_gcc850/Release/v00-10/setup.sh
v00-06 source /cvmfs/herd.ihep.ac.cn/HERDOS/SLC7/Pre-release/v00-06/setup.sh
v00-05 source /cvmfs/herd.ihep.ac.cn/HERDOS/SLC7/Pre-release/v00-05/setup.sh

For detailed release notes, see/cvmfs/herd.ihep.ac.cn/HERDOS/README.

Build from Source

gitclonegit@code.ihep.ac.cn:herdos/offline.gitcdofflinegitcheckoutv00-10./build_RelWithDebInfo.shsourceinstall/setup.sh

12.12.7 Other Management Services

HERD provides the following auxiliary management services on theherdtb01node. These services are deployed in containers.

Other Services

Service Description
VNC Remote graphical desktop for ROOT/Geant4 visualization and related use cases

To add more services, contact the HERD computing administrator.

12.12.8 Service Support

HERD computing administrators:

Administrator Email
Xiaowei Jiang jiangxw@ihep.ac.cn
Xuantong Zhang zhangxuantong@ihep.ac.cn
AI 助手
快速提问 ×

有问题?输入后跳转到对话页继续交流

Baidu
map