Skip to content

8.3 Lustre File Storage

8.3.1 Overview

LUSTRE is an open-source, scalable, high-performance, object-oriented parallel file system that is widely used on supercomputing platforms. More than 70% of the world's supercomputers use the LUSTRE file system. LUSTRE consists of metadata servers (MDS), object storage servers (OSS), and clients. Through metadata servers, LUSTRE combines storage devices on all object storage servers into a unified storage space and provides it to clients for mounting. It is fully compatible with POSIX semantics. LUSTRE is highly scalable, can easily manage tens of PB of storage, and provides very high performance, with aggregate bandwidth reaching tens or even hundreds of GB per second. Therefore, it is widely used in high-performance computing. However, due to metadata server limitations, LUSTRE has no advantage in managing massive numbers of small files, and it does not support erasure coding or replicas. Therefore, the LUSTRE file system is mainly used to store massive experiment data. LUSTRE is currently the Computing Center's main disk file system.

LUSTRE is the largest-scale file system used in the high-performance computing platform. It provides massive storage services for BES, DYB, JUNO, and other experiments. Current Lustre file system information is listed in Table 6.

Total Space (TB) Purpose User Quota Backup Status
/bes3fs 2500 BESIII production data Raw data is backed up to tape; others are not backed up
/besfs4 2500 BESIII production data Raw data is backed up to tape; others are not backed up
/besfs5 2400 BESIII production data, BESIII group data, user data 50 GB per user under theusersdirectory; group directory quotas vary by group Raw data is backed up to tape; others are not backed up
/besfs7 3600 BESIII production data Raw data is backed up to tape; others are not backed up
/besfs8 3300 BESIII production data Raw data is backed up to tape; others are not backed up
/besfs9 4000 BESIII production data Raw data is backed up to tape; others are not backed up
/besfs10 4100 BESIII production data Raw data is backed up to tape; others are not backed up
/publicfs 3000 ATLAS, CMS, LHCb, and UCAS sub-pool shared data disk 5 TB and 300,000 files per user Resources are isolated by group; no backup
/sharefs 1500 ALICPT, BES, HBKG, HEPS, and MBH sub-pool shared data disk Resources are isolated by group; no backup
/scratchfs 572 User temporary files 500 GB and 200,000 files per user No backup
/scratchfs2 1300 User temporary files, gradually replacing/scratchfs 500 GB and 200,000 files per user No backup
/workfs2 22 User personal files, recommended for important files or results 5 GB and 50,000 files per user Full backup
/cefs 2500 CEPC experiment data and user data No backup
/junofs 3100 JUNO experiment data 500 GB and 300,000 files per user No backup
/lustre/juno26 2600 JUNO experiment data 500 GB and 300,000 files per user No backup
/lustre/junoflash 88 JUNO experiment data No backup
/dybfs2 1400 DYB experiment data and user data 1 TB and 300,000 files per user Raw data is backed up to tape; others are not backed up
/gecamfs 2200 GECAM experiment data No backup
/hxmtfs 1300 HXMT experiment data 500 GB and 50,000 files per user home directory; 5 TB and 1,000,000 files per user data directory No backup
/hxmtfs2 2200 HXMT experiment data No backup
/hpcfs 1800 Slurm GPU application data No backup
/lhaasofs 610 LHAASO user data 200 GB and 500,000 files per user No backup
/epfs 610 EP experiment data and user data No backup
/extpfs 610 EXTP experiment data and user data No backup
/herdfs 1700 HERD experiment data and user data 200 GB and 300,000 files per user No backup
/hepsfs 645 BSRF experiment data and user data No backup
/hepsdatafs 29000 HEPSBL central storage experiment data No backup
/lustrefs/atlas 635 ATLAS experiment data and user data No backup
/lustrefs/alicpt 1300 ALICPT experiment data and user data No backup
/aifs 194 IHEPAI artificial intelligence application data and user data 500 GB and 500,000 files per user home directory No backup
/sdcfs 1700 SDC submitted data and user data Submitted data is backed up; others are not backed up

Table 6. Lustre file system information.

  • The Lustre file system fully supports Linux POSIX semantics. Users can access Lustre with Linux file system commands and APIs.
  • To use user ACLs and extended attributes, submit a request toics@ihep.ac.cn.
  • /workfsis dedicated to interactive use. It is readable and writable on login nodes, but users cannot create or modify files under/workfsfrom compute nodes. In other words, user jobs cannot write to or modify files under/workfswhile running.

8.3.2 Common Commands

Check User Resource Quota

Command format:

lfsquota-uusernamedirname

Example:

lfsquota-uzhangsan/publicfsDisk

Example output:

quotas for user zhangsan (uid XXXX):Filesystem kbytes quota limit grace files quota limit grace/publicfs 338626804 524288000 524288000 - 232010 300000 300100(used space) (soft capacity quota) (hard capacity quota) (used files) (soft file quota)

Check Quota Usage of Directories Under/besfs5/userand/besfs5/group

  1. In a specific directory, runlsattr -pon any regular file (not a directory) to view the project ID of that directory.
#lsattr -p nohup.out1097-----------------Pnohup.out

This shows that the project ID of the directory is1097.

  1. Runlfs quota -p 1097 /besfs5to view quota usage of that directory.
# lfs quota -p 1097 -h /besfs5Diskquotasforprj1097(pid1097):Filesystemusedquotalimitgracefilesquotalimitgrace/besfs51.362T*50G50G-29324900-

Check Space Usage of Each Disk Pool in a Lustre File System

Command format:

lfsdf-h-ppoolnamedirname

The-pparameter is followed by the disk pool name. For example, the ATLAS group hasacfs.atlaspool, the CMS group hasacfs.cmspool, and the LHCb group hasacfs.lhcbpool.

Examples:

lfsdf-h-pacfs.lhcbpool/publicfslfsdf-pacfs.atlaspool/publicfs|tail-n3

Example output:

filesystem summary: 934854246652 267931989532 619787497356 30% /publicfs

Set File Striping to Improve Concurrent Access Performance for a Single File

Command format:

lfssetstripedirname-cn

nis the stripe count.

Example:

lfssetstripeyourdirname-c6

Files under the directory inherit the directory striping attribute. To specify striping for a single file, create an empty file first withtouch, and then set its striping attribute.

Set Directory ACLs

For example, the directory/besfs4/wanglutestis owned byroot:rootand has permission750; userwanglu(60007:600) has no access to this directory.

-bash-4.2$ll/besfs4|grepwanglutestdrwxr-x---+2rootroot4096May1115:24wanglutest-bash-4.2$cd/besfs4/wanglutest-bash:cd:/besfs4/wanglutest:Permissiondenied

Grant userwanglupermission on this directory:

setfacl-muser:wanglu:rwx/besfs4/wanglutest

Checkwanglu's permissions:

-bash-4.2$getfacl/besfs4/wanglutestgetfacl:Removingleading'/'fromabsolutepathnames# file: besfs4/wanglutest# owner: root# group: rootuser::rwxuser:wanglu:rwxgroup::r-xmask::rwxother::---

The user can now read and write in the directory.

-bash-4.2$pwd/afs/ihep.ac.cn/users/w/wanglu-bash-4.2$cd/besfs4/wanglutest-bash-4.2$cp/etc/hosts.-bash-4.2$ls-lhtotal3.1M-rw-r--r--1wangluu073.0MMay1115:22f1-rw-r--r--1wangluu07292May1115:38hosts

Delete a user's ACL permission:

setfacl-xuser:wanglu/besfs4/wanglutest

Notes:

  1. Only the owner of a directory can modify ACL permissions on that directory.
  2. After ACL permissions are set,ls -lshows a+after the permission field. At this point, the original Linux permission rules no longer fully apply.
  3. To add an ACL for a group, usesetfacl -m group:xxx /xxx/xxx.

To restore the original Linux permission display and clear the+afterls -l, usesetfacl -b.

setfacl-b/besfs4/wanglutest

For complete ACL syntax, see:

http://wiki.lustre.org/images/5/57/PosixAccessControlInLinux.pdf

Or refer to Linux man pages:

mangetfaclmansetfacl

FAQ

SeeFAQ.

AI 助手
快速提问 ×

有问题?输入后跳转到对话页继续交流

Baidu
map