Windows

列出文件夹里的子文件夹的每个的大小:

Get-ChildItem -Path "." -Directory | ForEach-Object { $_.FullName + ": " + ((Get-ChildItem $_.FullName -Recurse | Measure-Object -Property Length -Sum).Sum / 1MB) + " MB" }

Linux

查看zip文件里的总文件个数:

 zipinfo  -h file.zip  |  tr '\n' ':'  | awk -F':'  '{print $2 , $5 , "files"}'

查看第一个jsonl最后两个内容的某个key:

tail -n 2 $(ls *.jsonl | head -n 1) | while read line; do echo "$line" | jq -r .id; done

查看指定jsonl最后两个内容的某key:

file=dbr_meta_790002-0.jsonl && tail -n 2 "$file" | while read line; do echo "$line" | jq -r .id; done

[S3] 复制当前目录下所有zip文件到s3 uri:

for file in *.zip; do aws s3 mv "$file" s3://YOUR_BUCKET_NAME/YOUR_DIRECTORY/; done

列出当前文件夹里各个的大小:

du -sh *

磁盘管理:

# sudo apt-get install ncdu
ncdu ~/

解压并删除文件夹里的所有zip文件:

for file in *.zip; do unzip "$file" && rm "$file"; done

列出当前目录的所有文件到txt:

find . -type f -exec basename {} \; > filenames.txt

列出当前文件count:

ls -p | grep -v / | wc -l

转换图片到webp95:

# sudo apt install imagemagick
find ./ -type f -exec convert '{}' -quality 95 '{}'.webp \;

删除所有invalid packages:

# which python
ls /home/ubuntu/miniconda3/lib/python3.10/site-packages | grep '^~' | xargs -I {} rm -rf /home/ubuntu/miniconda3/lib/python3.10/site-packages/'{}'

# delete curr python's directly:
# ls "$(dirname $(dirname $(which python)))/lib/python3.10/site-packages" | grep '^~' | xargs -I {} rm -rf "$(dirname $(dirname $(which python)))/lib/python3.10/site-packages/'{}'"

Setup

配置swap (4g):

# sudo apt install jq
# sudo swapoff /swapfile  
sudo dd if=/dev/zero of=/swapfile bs=1G count=4 && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile && sudo swapon --show

安装pip:

curl -O https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py --user && echo 'export PATH=$PATH:/home/ubuntu/.local/bin' >> ~/.bashrc && source ~/.bashrc

安装awscli:

sudo apt install unzip && curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip awscliv2.zip && sudo ./aws/install && rm awscliv2.zip

安装awscli (ARM64):

sudo apt install unzip 
curl -O 'https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip'
unzip awscli-exe-linux-aarch64.zip
sudo ./aws/install

Monitor

始终更新的nvidia-smi:

watch -n 0.5 nvidia-smi

trojblue/one-liners.md

Windows

Linux

Setup

Monitor