Show Username & full process command with nvidia-smi

enhanced-nvidia-smi.md

For typical use cases, I prefer using nvitop to view detailed information. This script offers a dependency-free implementation.

The script enhances the functionality of nvidia-smi and provides the following info:

• Username
• full process Command
• GPU ID
• PID

This is useful on multi-user servers and can be used to quickly identify which user is using the GPU and running what kind of program.

nvidia-smi && (nvidia-smi |tr -s ' '|grep -Eo "| [0123456789]+ N/A N/A [0-9]{3,} .*"|awk -F' ' '{system("s=$(cat /proc/"$4"/cmdline| tr \"\\0\" \" \");u=$(ps -o uname= -p "$4");echo "$1"sep"$4"sep$u sep"$7"sep$s" ) }'|sed 's/sep/\t/g')

The output is like this

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      3296      C   python                                      5825MiB |
|    0     21266      C   /opt/conda/bin/python                       6319MiB |
|    1      3296      C   python                                       137MiB |
|    1      8063      C   /usr/bin/python                              273MiB |
|    1     21266      C   /opt/conda/bin/python                       4735MiB |
|    1     30439      C   /usr/bin/python                              199MiB |
|    2      3296      C   python                                       137MiB |
|    2     21266      C   /opt/conda/bin/python                       4735MiB |
+-----------------------------------------------------------------------------+
0   3296    user1   5825MiB python playground2.py
0   21266   user2   6319MiB /opt/conda/bin/python -u /cail/model.py
1   3299    root    137MiB  python playground2.py
1   8063    root    273MiB  /usr/bin/python -u /sentsim/cli_predict.py
1   21266   root    4735MiB /opt/conda/bin/python -u /cail/model.py
1   30439   root    199MiB  /usr/bin/python -u /sentsim/train.py
2   3296    root    137MiB  python playground2.py
2   21266   root    4735MiB /opt/conda/bin/python -u /cail/model.py

For convenience, you can alias the command.

That's extremely helpful, thanks

Awesome. thanks