12.6 CMS¶
Overview¶
| Resource Type | Capacity | Notes |
|---|---|---|
| Login Node | lxlogin.ihep.ac.cn | General-purpose login node |
| HTCondor | T2: 1728 cores / T3: 836 cores | T2 shares endpoint with ATLAS, HTCondor-CE v24 |
| Slurm | 1×A100 + V100 shared pool + RTX 5090 nodes | cmsgpu group, see 7.6.7 |
| home Directory | /afs/ihep.ac.cn/users/ | 500MB, see 7.6.3 |
| Working Directory | T2 EOS: 1.7 PB / T3 publicfs: 849.1 TB | cceos.ihep.ac.cn, see 7.6.3/7.6.5 |
12.6.1 Login and Authentication¶
Login:
Grid Authentication:
- Since 2023, CMS has been migrating from X.509 certificates toWLCG Token authentication. CERN CMS IAM:https://cms-auth.web.cern.ch
- IHEP T2 HTTP/XRootD/HTCondor all support Token authentication. GSI (voms-proxy-init) is being phased out.
12.6.2 Tier3 Local Computing¶
CMS local computing resources:836 cores, managed by HTCondor. Access via lxlogin.
HTCondor Job Submission:
T3 Site Production(2025 data):
- Jobs completed: ~11,192,024 (-8.0% YoY)
- CPU hours: ~3,788,453 (+141.0% YoY)
12.6.3 Tier3 Local Storage¶
Public Data Area¶
Total capacity: 849.1 TB (Lustre filesystem), ~97% used, nearly full. Availability >99%.
Check usage:
Public Software Area¶
Personal Storage Areas¶
| Directory | Quota | Features | Notes |
|---|---|---|---|
/afs/ihep.ac.cn/users/ |
500MB | Backed up, readable from all nodes | Config files, scripts |
/workfs/cms/ |
5GB / 50,000 files | Backed up, read-only on compute nodes | Important personal files |
/scratchfs/cms/ |
500GB / 2-week retention | Read/write from all nodes | Temporary files |
12.6.4 Tier2 Grid Computing¶
System:HTCondor-CE + Batch HTCondor, upgraded to HTCondor v24.
CreamCE, mentioned in older documentation, was fully decommissioned in 2023. All resources now use HTCondor-CE.
Resources:1728 cores (48.6k HS23), sharing the same CE endpoint with ATLAS.
T2 Site Production(2025 data):
- Jobs completed: ~2,933,904 (-46.3% YoY, due to shift toward analysis job types)
- CPU hours: ~3,139,908 (+4.8% YoY)
- Data uploaded: 158.0 TB (-51.5% YoY)
- Data downloaded: 779.7 TB (-28.5% YoY)
Site Availability/Reliability:Ranked 15thamong all 49 CMS T2 sites globally (Jul 2025 – Jul 2026).
Maintenance and Procurement Project(approved 2023, completed Dec 2024):
| Resource Type | Before | After | Multiplier |
|---|---|---|---|
| T2 CPU | 420 cores | 1728 cores | 3.8x |
| T2 Storage | 673.5 TB | 1.7 PB | 2.5x |
12.6.5 Tier2 Grid Storage¶
System:EOS storage (shared instance with ATLAS, Belle2, Biomend, OPS).
In May 2023, T2 storage was migrated from DPM to EOS. dCache/SRM has been decommissioned. The old address
ccsrm.ihep.ac.cnhas been replaced bycceos.ihep.ac.cn.
Resources:1.7 PB total
- Central managed data: ~540 TB
- User data area: ~350 TB capacity, ~216.3 TB currently used (61.8%)
T2 EOS Access:
EOS XRootD endpoint:root://cceos.ihep.ac.cn:1094/
EOS WebDAV endpoint:https://cceos.ihep.ac.cn:9000/eos/ihep/cms/
Common commands:
# Grid tools (recommended)gfal-lsroot://cceos.ihep.ac.cn:1094//store/user/gfal-lshttps://cceos.ihep.ac.cn:9000/eos/ihep/cms/store/user/# xrdfs accessxrdfsroot://cceos.ihep.ac.cn:1094/ls/store/# davix-ls accessdavix-lshttps://cceos.ihep.ac.cn:9000/eos/ihep/cms/store/# EOS client (set environment variable first)exportEOS_MGM_URL=root://cceos.ihep.ac.cn:1094eosls/store/
User Space Quota Policy(updated 2026):
| Condition | Quota |
|---|---|
| Usage < 9 TB | No quota(default) |
| Usage exceeds 9 TB | 9 TB quota automatically applied at 08:00 next morning |
| Over-quota | Email notification sent, please clean up |
| Persistent over-quota ~6 months | Oldest files cleaned first |
12.6.6 Grid Job Submission¶
CMSSW Environment Setup¶
exportVO_CMS_SW_DIR=/cvmfs/cms.cern.ch/source$VO_CMS_SW_DIR/cmsset_default.sh# List installed CMSSW versionsscramv1list-cCMSSW# First time: create local CMSSW directoryscramv1projectCMSSWCMSSW_X_Y_Z# Initialize runtime environmentcdCMSSW_X_Y_Z/src/eval`scramv1runtime-sh`
CRAB Job Submission¶
source/cvmfs/cms.cern.ch/crab/crab.(c)sh# Create and submit jobscrabsubmit[-c] # Check job statuscrabstatus[-d] # Retrieve logs and resultscrabgetlog[-d]
More details:https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookCRAB3Tutorial
T2/T3 Bidirectional Data Access (new in 2025)¶
T3 jobs accessing T2 data: Add certificate options in
hep_subto enable T3 jobs to access T2 EOS storage via XRootD protocol.T2 jobs accessing T3 data: After linking your CERN certificate athttp://ccs.ihep.ac.cn/jeeplus/a, T2 jobs can access T3 data via XRootD proxy.
12.6.7 GPU Resources¶
CMS Priority GPU¶
| Item | Description |
|---|---|
| Resource | 1×NVIDIA A100 GPU, priority access for cmsgpu group |
| Management | Slurm shared pool (includes V100 GPUs), QOS:cmsnormal |
| Suitable for | Scientific/HPC workloads requiring higher numerical precision |
| Apply | Request cmsgpu group approval athttps://ink.ihep.ac.cn/dashed |
| Submit | #SBATCH --qos=cmsnormal |
IHEP-AI Shared GPU Platform¶
| Item | Description |
|---|---|
| Resource | Slurm cluster: 2×8-GPU RTX 5090 nodes; Kubernetes cluster: 1×8-GPU RTX 5090 node |
| Management | hepai group |
| Suitable for | AI training, large model workloads (mixed precision) |
| Apply | Request hepai group approval |
| Platform | https://ai.ihep.ac.cn |
Slurm documentation:http://afsapply.ihep.ac.cn/cchelp/zh/local-cluster/jobs/slurm/
12.6.8 Notes¶
/publicfs/cmsis ~97% full. Please remove unused files promptly to free space./workfsis read-only on compute nodes. Jobs writing to this directory will fail; use/scratchfsfor job output.- Keep files per directory under 1000; create subdirectories for large file collections.
- T2 EOS user storage exceeding 9 TB will be automatically restricted. Watch for quota alert emails in your CERN inbox.
- X.509 grid certificates are being replaced by WLCG Tokens. New users should use Token authentication directly.
- Contact site coordinator for questions:zhangxuantong@ihep.ac.cn.
有问题?输入后跳转到对话页继续交流