15.1 Common Linux Commands¶
This chapter summarizes common Linux commands for daily work on the computing cluster. In command examples,$represents a normal user shell prompt. Do not copy$when copying commands.
Warning
Login nodes are mainly used for logging in, editing files, compiling small programs, submitting jobs, and checking results. Do not run long-running, high-CPU, high-memory, or heavy I/O computing tasks on login nodes. Submit these tasks to the HTCondor or Slurm job systems.
1. Get Current Environment Information¶
| Task | Command Example |
|---|---|
| Show current user | whoami |
| Show user ID and groups | id |
| Show user groups | groups |
| Show current host name | hostname |
| Show current directory | pwd |
| Show current time | date |
| Show system uptime and load | uptime |
| Show kernel and architecture | uname -a |
| Show operating system version | cat /etc/os-release |
Examples:
2. Directory and Path Operations¶
| Task | Command Example |
|---|---|
| Change to a directory | cd /path/to/dir |
| Return to home directory | cdorcd ~ |
| Go to the parent directory | cd .. |
| Go back to the previous working directory | cd - |
| Show the absolute path of the current directory | pwd |
| Create a directory | mkdir dirname |
| Create nested directories | mkdir -p project/logs |
| View a directory tree | tree |
Iftreeis not installed, usefindto view directory levels:
3. View Files and Directories¶
| Task | Command Example |
|---|---|
| List files in the current directory | ls |
| Show detailed information | ls -l |
| Show hidden files | ls -a |
| Sort by time, newest first | ls -lt |
| Sort by file size | ls -lhS |
| Show human-readable sizes | ls -lh |
| Show file type | file filename |
| Show the first lines of a file | head filename |
| Show the last lines of a file | tail filename |
| Follow log output in real time | tail -f job.out |
| Page through a file | less filename |
| Print a file directly | cat filename |
Common combinations:
4. Create, Copy, Move, and Remove Files or Directories¶
| Task | Command Example |
|---|---|
| Create an empty file | touch file.txt |
| Copy a file | cp source.txt target.txt |
| Copy a directory | cp -r source_dir target_dir |
| Preserve attributes while copying | cp -a source_dir target_dir |
| Move a file or directory | mv old_path new_path |
| Rename a file | mv old.txt new.txt |
| Remove a file | rm file.txt |
| Remove a file interactively | rm -i file.txt |
| Remove an empty directory | rmdir empty_dir |
| Remove a non-empty directory | rm -r dir |
Warning
Files removed withrmusually cannot be restored directly. Before runningrm -rorrm -rf, carefully confirm the path and variable expansion. Do not use recursive deletion when a variable may be empty or a path is uncertain.
Safer deletion examples:
5. Search Files and Text¶
| Task | Command Example |
|---|---|
| Find files by name | find . -name "*.txt" |
| Find directories | find . -type d -name "logs" |
| Find regular files | find . -type f |
| Find large files | find . -type f -size +1G |
| Find recently modified files | find . -type f -mtime -1 |
| Search text in a file | grep "keyword" file.txt |
| Search text recursively | grep -R "keyword" . |
| Show matching line numbers | grep -n "keyword" file.txt |
| Search case-insensitively | grep -i "keyword" file.txt |
| Count matching lines | grep -c "keyword" file.txt |
Ifrgis installed, prefer it for faster recursive searches:
Common examples:
# Find all .out files under the current directory$find.-typef-name"*.out"# Find files larger than 1 GB$find.-typef-size+1G-lh# Search logs for errors$grep-Rni"error"logs/
6. Check File Size, Disk Usage, and Quotas¶
| Task | Command Example |
|---|---|
| Show file system usage | df -h |
| Show total size of current directory | du -sh . |
| Show sizes of subdirectories | du -h --max-depth=1 |
| Sort directories by size | du -h --max-depth=1 | sort -h |
| Sort files by size | ls -lhS |
| Check AFS quota | fs listquota /afs/ihep.ac.cn/users/... |
| Check Lustre user quota | lfs quota -u username /scratchfs |
| Check Lustre file system space | lfs df -h /scratchfs |
Examples:
$df-h$du-sh*$du-h--max-depth=1|sort-h$fslistquota/afs/ihep.ac.cn/users/a/username$lfsquota-uusername/scratchfs$lfsquota-uusername/workfs
When disk space is insufficient, first locate large directories and files:
7. File Permissions and ACLs¶
| Task | Command Example |
|---|---|
| View permissions | ls -l file.sh |
| Add execute permission | chmod +x file.sh |
| Set file permissions to owner read/write only | chmod 600 file.txt |
| Set directory permissions to owner access only | chmod 700 private_dir |
| Set file permissions to owner read/write and others read-only | chmod 644 file.txt |
| Set common directory access permissions | chmod 755 dir |
| View ACL permissions | getfacl file_or_dir |
| Set ACL permissions | setfacl -m u:username:rwx dir |
Common numeric permissions:
| Permission | Meaning |
|---|---|
7 |
read, write, and execute |
6 |
read and write |
5 |
read and execute |
4 |
read only |
0 |
no permission |
Examples:
$chmod+xrun.sh$chmod600id_rsa$chmod700~/.ssh$chmod644~/.ssh/authorized_keys$getfacl/workfs/group/project
8. Text Editing and Text Processing¶
| Task | Command Example |
|---|---|
| Edit with vim | vim file.txt |
| Edit with nano | nano file.txt |
| Count lines, words, and bytes | wc file.txt |
| Count lines only | wc -l file.txt |
| Sort lines | sort file.txt |
| Remove adjacent duplicate lines | uniq file.txt |
| Sort and remove duplicates | sort file.txt | uniq |
| Extract columns | awk '{print $1}' file.txt |
| Replace text | sed 's/old/new/g' file.txt |
| Compare two files | diff old.txt new.txt |
| Inspect binary or hidden characters | od -c file.txt |
Examples:
# Show the first 20 lines$head-n20data.txt# Show lines 10 to 20$sed-n'10,20p'data.txt# Extract the first column and remove duplicates$awk'{print $1}'data.txt|sort|uniq# Count occurrences of error in a log$grep-i"error"job.log|wc-l
9. Compression and Extraction¶
| File Type | Extraction Command |
|---|---|
.tar |
tar -xvf file.tar |
.tar.gzor.tgz |
tar -zxvf file.tar.gz |
.tar.bz2 |
tar -jxvf file.tar.bz2 |
.tar.xz |
tar -Jxvf file.tar.xz |
.gz |
gunzip file.gz |
.zip |
unzip file.zip |
Create archives:
# Archive a directory without compression$tar-cvfproject.tarproject/# Create a gzip-compressed archive$tar-zcvfproject.tar.gzproject/# Exclude selected files or directories$tar--exclude="*.tmp"--exclude="output/"-zcvfproject.tar.gzproject/# Create a zip archive of a directory$zip-rproject.zipproject/
View archive contents:
Note
Large numbers of small files can affect shared file system performance. Before backup or transfer, consider packaging them withtar.
10. File Transfer¶
| Task | Command Example |
|---|---|
| Copy from local machine to cluster | scp local.txt username@lxlogin.ihep.ac.cn:/path/ |
| Copy from cluster to local machine | scp username@lxlogin.ihep.ac.cn:/path/file.txt ./ |
| Copy a directory | scp -r local_dir username@lxlogin.ihep.ac.cn:/path/ |
| Resume or synchronize a directory | rsync -avP source/ target/ |
| Use sftp | sftp username@lxlogin.ihep.ac.cn |
| Download a file from a URL | wget URL |
| Download a file with curl | curl -O URL |
Examples:
# Upload a file$scpinput.datusername@lxlogin.ihep.ac.cn:/workfs/group/username/# Download results$scpusername@lxlogin.ihep.ac.cn:/workfs/group/username/result.out./# Synchronize a directory, useful for large directories or unstable networks$rsync-avP./project/username@lxlogin.ihep.ac.cn:/workfs/group/username/project/# Preview what rsync would synchronize$rsync-avP--dry-run./project/username@lxlogin.ihep.ac.cn:/workfs/group/username/project/
11. View and Manage Processes¶
| Task | Command Example |
|---|---|
| Show processes in current terminal | ps |
| Show all processes | ps -ef |
| Show your own processes | ps -u username |
| Search processes by keyword | pgrep -af python |
| View processes dynamically | top |
| View processes dynamically with htop | htop |
| Terminate a process | kill PID |
| Force-terminate a process | kill -9 PID |
| Run a command in the background | command & |
| Show background jobs | jobs |
| Bring a background job to the foreground | fg %1 |
| Continue a stopped job in the background | bg %1 |
Examples:
Warning
Do not usekill -9as the default option. A normalkill PIDgives the program a chance to exit cleanly. Usekill -9 PIDonly when the process cannot terminate normally.
12. View System Resources¶
| Task | Command Example |
|---|---|
| View CPU and memory usage | top |
| View memory usage | free -h |
| View process, memory, and system load | vmstat 1 |
| View disk I/O | iostat -x 1 |
| View GPU status | nvidia-smi |
| Refresh GPU status continuously | watch -n 1 nvidia-smi |
| Show logged-in users | who |
| Show current terminal and load information | w |
Examples:
Ifiostatorhtopis unavailable, the corresponding tool is not installed on the current system. Use basic commands such astopandfreeinstead.
13. Network and Connection Checks¶
| Task | Command Example |
|---|---|
| Test host reachability | ping hostname |
| Show network route path | traceroute hostname |
| Test whether a port is reachable | nc -vz hostname port |
| Show network addresses | ip addr |
| Show routes | ip route |
| Show listening ports | ss -lntp |
| Show all TCP connections | ss -ant |
Examples:
14. SSH and Remote Sessions¶
| Task | Command Example |
|---|---|
| Log in with SSH | ssh username@lxlogin.ihep.ac.cn |
| Log in with a specified port | ssh -p 2222 username@hostname |
| Log in with a private key | ssh -i ~/.ssh/id_rsa username@hostname |
| Enable X11 forwarding | ssh -XYC username@hostname |
| Generate an SSH key | ssh-keygen -t ed25519 |
| Copy a public key to a remote host | ssh-copy-id username@hostname |
Examples:
For long interactive sessions, usetmuxorscreento avoid losing the session when the network disconnects:
# Create a new tmux session$tmuxnew-swork# Reattach to an existing tmux session$tmuxattach-twork# List sessions$tmuxls
Note
tmuxandscreenare suitable for editing, compiling, and small interactive tests. Formal computing tasks should still be submitted to the job system.
15. Software Environment and Command Lookup¶
| Task | Command Example |
|---|---|
| Show environment variables | env |
| Show one environment variable | echo $PATH |
| Set an environment variable temporarily | export VAR=value |
| Locate a command | which python |
| Show how a command is resolved | type module |
| Show dynamic library dependencies | ldd program |
| Show current shell | echo $SHELL |
| Show command help | command --help |
| Show manual page | man command |
The cluster commonly usesmoduleto manage software environments:
# List available modules$moduleavail# Load a module$moduleloadpython/3.7# List loaded modules$modulelist# Unload a module$moduleunloadpython/3.7# Clear all loaded modules$modulepurge# Show module help$modulehelppython/3.7
Locate programs and dynamic library issues:
16. AFS and Kerberos Authentication¶
When reading or writing files under AFS directories, a valid token is often required.
| Task | Command Example |
|---|---|
| Obtain a Kerberos ticket | kinit username |
| Obtain an AFS token | aklog |
| View Kerberos tickets | klist |
| View AFS tokens | tokens |
| Remove AFS tokens | unlog |
| Change Kerberos password | kpasswd |
Examples:
If you cannot write files under/afs, check these first:
17. Shell History, Aliases, and Scripts¶
| Task | Command Example |
|---|---|
| View command history | history |
| Search command history | history | grep keyword |
| View current aliases | alias |
| Set a temporary alias | alias ll='ls -lh' |
| Remove an alias | unalias ll |
| Trace script execution | bash -x script.sh |
| Check script syntax | bash -n script.sh |
Common script header:
Notes:
set -e: exit the script when a command fails.set -u: exit the script when an undefined variable is used.
18. Input/Output Redirection and Pipes¶
| Task | Command Example |
|---|---|
| Write standard output to a file and overwrite it | command > output.txt |
| Append standard output to a file | command >> output.txt |
| Write standard error to a file | command 2> error.txt |
| Write standard output and error to the same file | command > all.log 2>&1 |
| Pipe output to another command | command1 | command2 |
| Show output and save it to a file | command | tee output.txt |
| Append output while showing it | command | tee -a output.txt |
Examples:
19. Common Log Troubleshooting Commands¶
| Task | Command Example |
|---|---|
| View the end of a log | tail -n 100 job.out |
| Follow a log in real time | tail -f job.out |
| Search for errors | grep -i "error" job.out |
| Search for failure messages | grep -Ei "error|failed|fatal|killed" job.out |
| View output file sizes | ls -lh job.out |
| View recently generated files | ls -lt |
| View the exit code of the previous command | echo $? |
Examples:
Common keywords:
| Keyword | Possible Meaning |
|---|---|
No such file or directory |
The path is wrong or the file does not exist |
Permission denied |
Permission is insufficient or the script is not executable |
command not found |
The command is not loaded, or the command path is not inPATH |
Killed |
The process may have been terminated because it used too much memory |
out of memory/OOM |
Insufficient memory |
Disk quota exceeded |
Storage quota exceeded |
20. Common Checks Before and After Jobs¶
Before submitting a job, check:
# Which directory am I in?$pwd# Does the input file exist?$ls-lhinput.dat# Is the script executable?$ls-lrun.sh$chmod+xrun.sh# Is the software environment correct?$modulelist$whichpython$python--version# Is there enough storage space?$du-sh.$df-h.
After a job finishes, check:
# View output files$ls-lh$tail-n100job.out# Search for errors$grep-Ei"error|failed|fatal|killed|oom"job.out# Check output directory size$du-shoutput/
21. Get Help¶
| Task | Command Example |
|---|---|
| View brief command help | command --help |
| View the manual page | man command |
| Show command path | which command |
| View help for a shell built-in command | help cd |
| View module help | module help module_name |
Examples:
有问题?输入后跳转到对话页继续交流