programming TIPS

useful_tips.md

Bash / Unix admin

https://www.cheatsheet.wtf/Nano/

• useful prompt (when working in git repos):

# Show Git branch in the prompt
parse_git_branch() {
   git branch 2>/dev/null | grep '*' | sed 's/* //'
}

# Customize the prompt with different colors for user and hostname
export PS1="\[\033[34m\]\u\[\033[00m\]@\[\033[36m\]\h \[\033[32m\]\w\[\033[33m\] on git:\$(parse_git_branch)\[\033[00m\] $ "

• grep with logic or grep -e patter1 -e pattern1

• file stats: stat [file] and debugfs

• remove lines matching pattern from file

• how-to-go-to-the-previous-working-directory-in-terminal cd ~-

• bash: how-to-check-how-long-a-process-has-been-running/

• find process: pgrep -f [pattern] -- midn -f!; or ps aux | grep [pattern]

• variables with nested shell scripts https://stackoverflow.com/questions/9772036/pass-all-variables-from-one-shell-script-to-another

• n. of files in all subdir: https://unix.stackexchange.com/questions/4105/how-do-i-count-all-the-files-recursively-through-directories

• find files and list info e.g. with csv: find . -name *csv -exec ls -lsh {} \;

• line endings:

  • to check them: https://stackoverflow.com/a/3570574/5533078
  • to control how git handles them:
    • https://stackoverflow.com/a/5834094/5533078
    • https://stackoverflow.com/a/39461324/5533078

• checksums:

  • crc32 -t d4 -f yourfile (d4 is for decimal format)

• hexadecimal conversion $((16#$hexNum)) or in your bashrc

	hex2dec(){

        	hexnum="$1" 
	        echo $((16#$hexnum))
	}

• diff --color=always -y --suppress-common-lines tips: https://phoenixnap.com/kb/diff-color

• diff for directories

  • tar -cf - somedir | md5sum
  • find somedir -type f -exec md5sum {} \; | sort -k 2 | md5sum
  • diff -qr dir1 dir2

• update directories -- mind tha cp only looks at timestamp -> https://superuser.com/a/434101/687355

• (re)directing stdout / stderr to files:

  • https://linuxize.com/post/bash-redirect-stderr-stdout/
  • https://www.howtogeek.com/435903/what-are-stdin-stdout-and-stderr-on-linux/

• run shell command on remote host -- detach from ssh session:

  • https://askubuntu.com/questions/8653/how-to-keep-processes-running-after-ending-ssh-session
  • https://stackoverflow.com/questions/4549489/can-i-change-the-name-of-nohup-out

• detach jobs:

  • Press Ctrl+Z to pause the currently running process.
  • Type bg and press Enter
  • disown jobID (list with jobs -l or ps aux)

• reown / reattach: https://github.com/nelhage/reptyr

• compare two files -- SO question

  • cmp --silent $old $new || echo "files are different"
  • use a checksum

• compares files contained in two folders -> https://gist.github.com/ggrrll/7ab0107182c30e9154dd3e7d6e48b62a

• ssh connection timeout

  • https://www.tecmint.com/increase-ssh-connection-timeout/ & https://man.openbsd.org/ssh_config
  • https://stackoverflow.com/questions/4936807/how-to-set-ssh-timeout

• copy files from server, that need sudo rights > https://askubuntu.com/a/872537/641871

• pipeline to open package location in sublime :) sublime $(pip show [your_package] | grep Location | cut -d ' ' -f 2)

• update-a-file-in-hdfs

• get FQDN server

• working on remote with sublime :) > https://github.com/randy3k/RemoteSubl

• touchpad behaviour (xinput)

  • how_to_invert_buttons_below_the_touchpad/
  • change scrolling speed

• close background ssh tunnel (ssh -f) : https://unix.stackexchange.com/a/164656/143800

• fast (de)compression of archives: https://stackoverflow.com/questions/12313242/utilizing-multi-core-for-targzip-bzip-compression-decompression , in particular https://github.com/facebook/zstd

• replace string in filenames: find . -name '*jpg' -exec bash -c ' mv $0 ${0/\old/new}' {} \;

• replace whitespaces in filenames: https://vitux.com/how-to-replace-spaces-in-filenames-with-underscores-on-the-linux-shell/

• fuzzy search in bash history https://github.com/junegunn/fzf

• colorized ls https://askubuntu.com/a/973569/641871

networks

• get host's IP dig +short myip.opendns.com @resolver1.opendns.com

useful bashrc / bash_profile

# some more ls aliases
alias ll='ls -larth -F'
alias la='ls -Al'
alias l='ls -CF'
alias nano='nano -L -N -c'
#alias less='less -N'
alias diff='diff --suppress-common-lines --color=always -y'

if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto -n'
    alias fgrep='fgrep --color=auto -n'
    alias egrep='egrep --color=auto -n'
fi

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoredups

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=
HISTFILESIZE=

GPU monitornig

• lspci | grep VGA
• pip install gpustat (only NVIDIA)
• apt-get install nvtop

file names

• detox -r [dir] https://stackoverflow.com/a/14266025/5533078

partitions and disks

• fix filesystem corruption (one approach) https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/Mount-mount-dev-mapper-vg00-lvol1-on-mnt-StorageMedia-failed-Structure-needs-cleaning-error-when-trying-to-mount-storage-in-Flame.html
• sudo lsblk -a also allows to see which are mounted

disks sizes

• average file size! ls -l | gawk '{sum += $5; n++;} END {print sum/n;}'

• list files by size find . -type f -printf "%s\t%p\n" | sort -n

• du -hsc * | sort -hr https://askubuntu.com/questions/1224/how-do-i-determine-the-total-size-of-a-directory-folder-from-the-command-line

• df -h or ncdu for usage %

• tree -h

• count files by type! :) find . -type f | sed -n 's/..*\.//p' | sort | uniq -c (credis to https://www.2daygeek.com/how-to-count-files-by-extension-in-linux/ )

• count files in each sub-dir : du --inodes

cleaning up space

• journalctl https://unix.stackexchange.com/questions/130786/can-i-remove-files-in-var-log-journal-and-var-cache-abrt-di-usr

network monitoring

• https://askubuntu.com/questions/257263/how-to-display-network-traffic-in-the-terminal

desktop env

• check des manager wmctrl -m

users & groups

proper permissions -- chmod

• home: 750
• .ssh/authorized_keys: 644
• private ssh key: 400

create new user

• sudo useradd -m username
• sudo passwd username
• usermod -aG sudo username

check groups and members

• groups [user]

• members [group-name] or grep 'group-name-here' /etc/group

• cat /etc/group | cut -d: -f1 -- list all groups

• find /home -group test -- files owned by group 'test'

check open ports

from https://www.cyberciti.biz/faq/unix-linux-check-if-port-is-in-use-command/

• sudo lsof -i -P -n | grep LISTEN -- less /etc/services
• netstat -tulpn | grep LISTE

Security

• tools for scanning repo for credentials: https://geekflare.com/github-credentials-scanner/

Ubuntu / Mint

• change config of progressive apps (browser apps), edit files in ~.local/share/applications

Python

profilers

https://stackoverflow.com/questions/4544784/how-can-you-get-the-call-tree-with-python-profilers

parallelization

• joblib: https://joblib.readthedocs.io/en/latest/
• RAY:
  • https://betterprogramming.pub/python-use-ray-to-run-functions-in-parallel-871f7a6678c4
  • https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1

code analysis

• UML diagram tools: see discussion and comment on GH and this good review; see also this description of UML diagram conventions
  • pynsource -- nice , but needs pro
  • pylint: FOSS, not interactive (webp cannot be generated...to double check)
  • epydoc : does not work with python3

(sub) plotting

• https://engineeringfordatascience.com/posts/matplotlib_subplots/
• seaborn:
  • xticks rotation in facegrid https://stackoverflow.com/a/72099702/5533078

debugging

• ✅ ipdb has autocompletion, loads variables.. : ipdb3 my_script.py
• reload(my_package) -- credits to this SO psot
• pip update selected packages (bash pipe): e.g. pip freeze | grep jupy | cut -d'=' -f1 | cut -d":" -f2 | xargs pip install -U

miscellanea

• file open modes

• extract numbers from strings (e.g. filenames) https://stackoverflow.com/a/4703409/5533078

• get variable name :)!

  • in list: https://stackoverflow.com/a/52038959/5533078
  • simple variable: https://stackoverflow.com/a/58451182/5533078

VS code

• list of python symbols https://code.visualstudio.com/docs/editor/intellisense
• to activate the right env/kernel, in interactive script: sclick at the symbol , at the top right, in the interactive tab, and select your interpreter

Jupyter notebook

layout

• decrease side space -- mind the different css code for notebook 7 https://stackoverflow.com/a/76778615/5533078 -- see also jupyter/notebook#7152

%load_ext autoreload

this is problematic...many things tried (none worked):

• %autoreload 1 + %aimport tiqcovid
• %autoreload 2
• removing the module from subfolder > put in external folder
• sys.path.append('path_to_package')

debugging

• %debug -- here is an intro

• from IPython.core.debugger import set_trace -> better to use pdb.set_trace()

here are a description of commands + GUI

• in iPython, better to use %autoreload (see the doc)
• code profiling https://jakevdp.github.io/PythonDataScienceHandbook/01.07-timing-and-profiling.html

plotting

• matplotlib backends https://stackoverflow.com/a/36021869/5533078
• save plotly graphs inside nb, after export https://community.plotly.com/t/exporting-jupyterlab-notebook-as-html-plotly-loses-interactiveness/15310

saving outputs

save cell to file :)

%%writefile example.txt

from interactive prompt

• first capture the output:

%%capture my_var
my_command

like explained in the 2nd answer here

• to save do:

%%store my_var.stdout >> my_file.txt

from script in terminal

mind the buffering! -.-

https://unix.stackexchange.com/questions/182537/write-python-stdout-to-file-immediately

print all attributes of python object

for attr in dir(obj):
    print "obj.%s = %s" % (attr, getattr(obj, attr))

shut down all notebooks servers kept open

• list all servers with jupyter notebook list
• try: jupyter notebook stop PORTID
• else: deleted the related file in /Library/Jupyter/runtime
• find the PIDs related to each port with ps aux | grep PORTID
• kill the process with kill -9 PID

find kernel version

from sys import executable

notebook on remote server

on local host ssh -L localhost:8888:localhost:8889 username@your_remote_host_name (-N -f flags are not needed)

and on remote: jupyter notebook --no-browser --port=8889

refs: https://amber-md.github.io/pytraj/latest/tutorials/remote_jupyter_notebook https://towardsdatascience.com/running-jupyter-notebooks-on-remote-servers-603fbcc256b3

presentation

https://voila.readthedocs.io/en/stable/using.html is a nice tool to make nb nore suitable for presentations

tricks:

• show code cells with --strip_sources=False

python packaging

• nice tutorial and collection of resources > https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html

• https://medium.com/38th-street-studios/creating-your-first-python-package-181c5e31f3f8

generate static html

• https://www.fullstackpython.com/static-site-generator.html
• https://stackoverflow.com/questions/6748559/generating-html-documents-in-python
• jinjia2 https://towardsdatascience.com/using-python-to-create-static-web-pages-the-easy-way-6eb16c997571

GitHub

actions

• mind to use lowercase letters only in yml! -> https://github.com/orgs/community/discussions/69617

Docker

manage-docker-as-a-non-root-user

general good practises

https://docs.docker.com/develop/develop-images/instructions/

https://docs.docker.com/develop/develop-images/guidelines/

tips

• add a label to existing image https://stackoverflow.com/a/54598281/5533078
• tool to visualize containers & images relations https://github.com/justone/dockviz

docker on github

• github working-with-the-container-registry

some useful commands

• images

  • docker image ls
  • docker image rm [image_ID]
  • docker build --network=host -t [full_tab_incl_webhost] -f [docker_file] (build an image to be set on a remote host)
  • docker push [full_tag]
  • docker inspect [image_ID]
  • docker run -it [full_tag] (to run a container on that image)
  • docker exec -it <mycontainer> bash to log into a container shell (add -u 0 to login as root)

• containers

  • docker ps -a
  • docker container rm [cont_ID]
  • docker restart <container_id>
  • where to find container logs: docker inspect --format='{{.LogPath}}' containername https://stackoverflow.com/a/41147654/5533078

Kubernetes

• kubectl get pods -A
• kubectl delete pod [pod_ID] -n [namespace]
• kubectl describe pod [pod_ID] -n [namespace]
• kubectl exec -it [pod_ID] -n [namespace] engine -- bash

Git

intro tutorials

nice cheatsheet https://www.quora.com/What-is-the-best-Git-cheat-sheet https://learngitbranching.js.org/.
https://gitimmersion.com/.
https://githowto.com/.

long list: https://medium.com/javarevisited/11-best-online-places-to-learn-git-for-beginners-in-2021-6dc2b7c6ef48

tips

• store safely credentials (use cache)

• working on shared folder:

  • https://serverfault.com/questions/26954/how-do-i-share-a-git-repository-with-multiple-users-on-a-machine
  • https://git-scm.com/docs/git-init#Documentation/git-init.txt---sharedfalsetrueumaskgroupallworldeverybodyltpermgt
  • if the repo already exists: git config core.sharedRepository true, after setting up the dedicated group

• pull file from another branch https://www.freecodecamp.org/news/git-checkout-file-from-another-branch/

• verbose log git log flags: -p , --stat, --patch-with-stat

• remove file from last commit (reset + amend) https://stackoverflow.com/a/42450337/5533078

• git add -u to stage only tracked files

• make private fork of public repo: https://stackoverflow.com/questions/10065526/github-how-to-make-a-fork-of-public-repository-private

• checkout new branch: use new command git switch [branch] https://stackoverflow.com/a/9537923/5533078

• mind possiblity to udpate-index: https://stackoverflow.com/a/20241145/5533078

• careful when working with ignored files on different branches!

  • https://stackoverflow.com/questions/30643119/git-checkout-recover-lost-files
  • https://stackoverflow.com/questions/1836742/using-git-how-do-i-ignore-a-file-in-one-branch-but-have-it-committed-in-another

• to avoid messing up after fetching (merge conflicts), check the diff first!:

git diff origin/master..

• modify past commit message: https://help.github.com/en/github/committing-changes-to-your-project/changing-a-commit-message

• git diff in same branch > git diff --no-index [file1] [file2]

• git diff specific commit: git diff sha~ sha https://stackoverflow.com/a/17563740/5533078

• commit description : you need 2 new lines to be correctly shown in github desktop (see also this SO discussion)

• for notebooks comparison: nbdiff-web -M [branch1] [branch2] [notebook_name]
  (git diff across branches works only after commit!)

• apply .gitignore: git rm -r --cached . & git add .

• untrack files (without necessary having them in gitignore) git rm --cached [file]

'going back'

mind! revert VS reset VS checkout

https://www.w3docs.com/snippets/git/how-to-reset-or-revert-a-file-to-a-specific-revision.html

merge

• pulling with rebase

  • https://www.atlassian.com/git/tutorials/merging-vs-rebasing
  • https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/about-merge-methods-on-github

• for selective merge :

  • git checkout --patch B f
  • or try also
  • git merge --no-ff --no-commit branchX

• notebook merge (follow also instructino from github) (e.g. branch branch_b into branch_a):

  • open a PR!
  • git checkout branch_b
  • git merge branch_a > these creates/show conflicts
  • nbdime mergetool

git LFS

• nice intro tutorial https://sabicalija.github.io/git-lfs-intro/

• nice reading on pros and cons https://stackoverflow.com/questions/75946411/how-does-git-lfs-track-and-store-binary-data-more-efficiently-than-git

• in case export does not work for reverting, I wrote a script git-lfs/git-lfs#5463 (comment)

• cons:

  • if you convert to file (lfs pull) and back to pointer (see tutorial) it generates a file change, in principle to be commited

Android

flash a ROM with fastboot

do these steps, in this order, before restoring your stuff...

0. OEM + UBS from dev options
1. check adb devices
2. adb reboot bootloader
3. fastboot oem unlock
4. sh flash_your_ROM.sh
5. fastboot oem lock

source: https://forum.xda-developers.com/mi-a2/how-to/guide-official-stock-android-10-t4033835

adb sideload

TWRP issue

I got Failed to mount '/system' error while trying to wipe

• channging TWRP v. did not work (as suggested here https://android.stackexchange.com/a/214375/210859)
• solution: force again ext4 partition https://www.youtube.com/watch?v=ysBfcasLnGM

OTA installation

https://www.androidsage.com/2017/11/10/how-to-install-ota-updates/

https://fossbytes.com/install-ota-updates-using-android-recovery-and-adb-sideload/

https://www.theandroidsoul.com/how-to-install-ota-updates-manually-using-recovery-and-adb-sideload/

SQL

remove annoying crtl + C:

use --sigint-ignore

credtits https://dba.stackexchange.com/questions/20566/mysql-console-ctrlc-driving-me-nuts

Regex

easy regex generators

https://www.autoregex.xyz/
https://simple-regex.com/
https://regex-generator.olafneumann.org

R

clean code

(thanks to this SO post )

• packages:
  • install.packages("formatR")
  • install.packages("styler") -- styler doc and this tutorial
  • https://github.com/jimhester/lintr

Latex

tools to prettify latex code:

• install latest texstudiousing these commands

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null ; brew install caskroom/cask/brew-cask 2> /dev/null

then

brew cask install texstudio

• https://latexindentpl.readthedocs.io/en/latest/sec-how-to-use.html
• to see nice diff, use --> --word-diff

S2 GT-I9100
Android 4.1.2

https://lineageosroms.com/i9100/

https://forum.xda-developers.com/t/rom-unofficial-11-0-0-r43-i9100-lineageos-18-1.4199759/

https://forum.xda-developers.com/t/rom-unofficial-10-0-0-r41-i9100-lineageos-17-1.4022733/

download-android-11-gapps

help with heimdall > https://forum.xda-developers.com/t/how-to-flashing-with-odin-heimdall.1922461/

I am asking for help here > https://android.stackexchange.com/questions/244656/adb-sideload-stuck-at-36

GALAXY TAB S2 - sm-t813

https://www.droidword.com/2018/04/05/update-samsung-galaxy-tab-s2-9-7-sm-t813-to-android-8-0-oreo-via-resurrection-remix/