Skip to content

12.6 CMS

Overview

Resource Type Capacity Notes
Login Node lxlogin.ihep.ac.cn General-purpose login node
HTCondor T2: 1728 cores / T3: 836 cores T2 shares endpoint with ATLAS, HTCondor-CE v24
Slurm 1×A100 + V100 shared pool + RTX 5090 nodes cmsgpu group, see 7.6.7
home Directory /afs/ihep.ac.cn/users/ 500MB, see 7.6.3
Working Directory T2 EOS: 1.7 PB / T3 publicfs: 849.1 TB cceos.ihep.ac.cn, see 7.6.3/7.6.5

12.6.1 Login and Authentication

Login:

sshlxlogin.ihep.ac.cn

Grid Authentication:

  • Since 2023, CMS has been migrating from X.509 certificates toWLCG Token authentication. CERN CMS IAM:https://cms-auth.web.cern.ch
  • IHEP T2 HTTP/XRootD/HTCondor all support Token authentication. GSI (voms-proxy-init) is being phased out.

12.6.2 Tier3 Local Computing

CMS local computing resources:836 cores, managed by HTCondor. Access via lxlogin.

HTCondor Job Submission:

condor_submitsubmit-groupcms

T3 Site Production(2025 data):

  • Jobs completed: ~11,192,024 (-8.0% YoY)
  • CPU hours: ~3,788,453 (+141.0% YoY)

12.6.3 Tier3 Local Storage

Public Data Area

/publicfs/cms

Total capacity: 849.1 TB (Lustre filesystem), ~97% used, nearly full. Availability >99%.

Check usage:

lfsquota-u/publicfs

Public Software Area

/afs/ihep.ac.cn/soft/CMS//cvmfs/cms.cern.ch/ (maintained by CERN)

Personal Storage Areas

Directory Quota Features Notes
/afs/ihep.ac.cn/users/ 500MB Backed up, readable from all nodes Config files, scripts
/workfs/cms/ 5GB / 50,000 files Backed up, read-only on compute nodes Important personal files
/scratchfs/cms/ 500GB / 2-week retention Read/write from all nodes Temporary files

12.6.4 Tier2 Grid Computing

System:HTCondor-CE + Batch HTCondor, upgraded to HTCondor v24.

CreamCE, mentioned in older documentation, was fully decommissioned in 2023. All resources now use HTCondor-CE.

Resources:1728 cores (48.6k HS23), sharing the same CE endpoint with ATLAS.

T2 Site Production(2025 data):

  • Jobs completed: ~2,933,904 (-46.3% YoY, due to shift toward analysis job types)
  • CPU hours: ~3,139,908 (+4.8% YoY)
  • Data uploaded: 158.0 TB (-51.5% YoY)
  • Data downloaded: 779.7 TB (-28.5% YoY)

Site Availability/Reliability:Ranked 15thamong all 49 CMS T2 sites globally (Jul 2025 – Jul 2026).

Maintenance and Procurement Project(approved 2023, completed Dec 2024):

Resource Type Before After Multiplier
T2 CPU 420 cores 1728 cores 3.8x
T2 Storage 673.5 TB 1.7 PB 2.5x

12.6.5 Tier2 Grid Storage

System:EOS storage (shared instance with ATLAS, Belle2, Biomend, OPS).

In May 2023, T2 storage was migrated from DPM to EOS. dCache/SRM has been decommissioned. The old addressccsrm.ihep.ac.cnhas been replaced bycceos.ihep.ac.cn.

Resources:1.7 PB total

  • Central managed data: ~540 TB
  • User data area: ~350 TB capacity, ~216.3 TB currently used (61.8%)

T2 EOS Access:

EOS XRootD endpoint:root://cceos.ihep.ac.cn:1094/

EOS WebDAV endpoint:https://cceos.ihep.ac.cn:9000/eos/ihep/cms/

Common commands:

# Grid tools (recommended)gfal-lsroot://cceos.ihep.ac.cn:1094//store/user/gfal-lshttps://cceos.ihep.ac.cn:9000/eos/ihep/cms/store/user/# xrdfs accessxrdfsroot://cceos.ihep.ac.cn:1094/ls/store/# davix-ls accessdavix-lshttps://cceos.ihep.ac.cn:9000/eos/ihep/cms/store/# EOS client (set environment variable first)exportEOS_MGM_URL=root://cceos.ihep.ac.cn:1094eosls/store/

User Space Quota Policy(updated 2026):

Condition Quota
Usage < 9 TB No quota(default)
Usage exceeds 9 TB 9 TB quota automatically applied at 08:00 next morning
Over-quota Email notification sent, please clean up
Persistent over-quota ~6 months Oldest files cleaned first

12.6.6 Grid Job Submission

CMSSW Environment Setup

exportVO_CMS_SW_DIR=/cvmfs/cms.cern.ch/source$VO_CMS_SW_DIR/cmsset_default.sh# List installed CMSSW versionsscramv1list-cCMSSW# First time: create local CMSSW directoryscramv1projectCMSSWCMSSW_X_Y_Z# Initialize runtime environmentcdCMSSW_X_Y_Z/src/eval`scramv1runtime-sh`

CRAB Job Submission

source/cvmfs/cms.cern.ch/crab/crab.(c)sh# Create and submit jobscrabsubmit[-c]# Check job statuscrabstatus[-d]# Retrieve logs and resultscrabgetlog[-d]

More details:https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookCRAB3Tutorial

T2/T3 Bidirectional Data Access (new in 2025)

  • T3 jobs accessing T2 data: Add certificate options inhep_subto enable T3 jobs to access T2 EOS storage via XRootD protocol.

    hep_sub--invomsproxy>--certjob>

  • T2 jobs accessing T3 data: After linking your CERN certificate athttp://ccs.ihep.ac.cn/jeeplus/a, T2 jobs can access T3 data via XRootD proxy.

12.6.7 GPU Resources

CMS Priority GPU

Item Description
Resource 1×NVIDIA A100 GPU, priority access for cmsgpu group
Management Slurm shared pool (includes V100 GPUs), QOS:cmsnormal
Suitable for Scientific/HPC workloads requiring higher numerical precision
Apply Request cmsgpu group approval athttps://ink.ihep.ac.cn/dashed
Submit #SBATCH --qos=cmsnormal

IHEP-AI Shared GPU Platform

Item Description
Resource Slurm cluster: 2×8-GPU RTX 5090 nodes; Kubernetes cluster: 1×8-GPU RTX 5090 node
Management hepai group
Suitable for AI training, large model workloads (mixed precision)
Apply Request hepai group approval
Platform https://ai.ihep.ac.cn

Slurm documentation:http://afsapply.ihep.ac.cn/cchelp/zh/local-cluster/jobs/slurm/

12.6.8 Notes

  1. /publicfs/cmsis ~97% full. Please remove unused files promptly to free space.
  2. /workfsis read-only on compute nodes. Jobs writing to this directory will fail; use/scratchfsfor job output.
  3. Keep files per directory under 1000; create subdirectories for large file collections.
  4. T2 EOS user storage exceeding 9 TB will be automatically restricted. Watch for quota alert emails in your CERN inbox.
  5. X.509 grid certificates are being replaced by WLCG Tokens. New users should use Token authentication directly.
  6. Contact site coordinator for questions:zhangxuantong@ihep.ac.cn.
AI 助手
快速提问 ×

有问题?输入后跳转到对话页继续交流

Baidu
map