Tools, notes

apt.md

Apt

Missing pubkey

After a error such as

Err:4 https://apt.releases.hashicorp.com focal InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY AA16FCBCA621E701

Import key with

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7FCC7D46ACCC4CF8

azure-cli.md

az

https://docs.microsoft.com/en-us/cli/azure/

Defaults configuration

az config set core.output=table
az config set defaults.group=<resource group>

Clouds

az cloud list
az cloud set --name AzureUSGovernment

Subscriptions

List

az account show
az account list

Set active

az account set --subscription "name"

Locations

List locations supported by current subscription

az account list-locations

Resource groups

List

az group list

Deployments

Create

az deployment group create --resource-group <group> --template-file azuredeploy.json --name <deployment name>

cron.md

cron

list user jobs

crontab -l

edit user jobs

crontab -e

job spec

 * * * * *  command to execute
 │ │ │ │ │
 │ │ │ │ │
 │ │ │ │ └───── day of week (0 - 6) (0 to 6 are Sunday to Saturday, or use names; 7 is Sunday, the same as 0)
 │ │ │ └────────── month (1 - 12)
 │ │ └─────────────── day of month (1 - 31)
 │ └──────────────────── hour (0 - 23)
 └───────────────────────── min (0 - 59)

view cron logs in CentOS

vim /var/log/cron

curl.md

Curl

post json

--data-binary sends the data as-is

curl -i -X POST <url> -H "Content-Type: application/json" --data-binary @data.jsonl

docker.md

Docker

Context

docker context create dmachine \
  --docker "host=tcp://192.168.99.101:2376,ca=$HOME/.docker/machine/machines/dmachine/ca.pem,cert=$HOME/.docker/machine/machines/dmachine/cert.pem,key=$HOME/.docker/machine/machines/dmachine/key.pem"
docker context use dmachine

Run

Run interactive

docker run -it $IMAGE /bin/bash

Run expose ports

docker run -p 18080:80 $IMAGE

Run detached

docker run -d $IMAGE

Running container

Exec

docker exec -it $CONTAINER_ID bash

Follow logs

docker logs -f $CONTAINER_ID

Find container IP

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $CONTAINER_NAME_OR_ID

Host connectivity

docker container run --rm -it alpine sh
apk update && apk add iputils busybox-extras
ping
telnet

Azure

Authenticate docker CLI to push images

az login
az acr login -n $REGISTRY

Authenticate docker CLI to run on ACI (Azure Container Instances)

docker login azure
docker context create aci $CONTEXT_NAME
docker context use $CONTEXT_NAME

GCloud

Authenticate docker CLI to push images

gcloud auth configure-docker

executables_and_libraries.md

Linux

Discover newly installed libraries

ldconfig

Show locations of shared libraries (also discover newly installed)

ldconfig -v

Locations configured for shared libraries

cat /etc/ld.so.conf

Show libraries used by executable

ldd $EXECUTABLE

List strings in a file

strings $EXECUTABLE | less

fiddler.md

Capture remote powershell session

Setup fiddler as a proxy, and reset afterwards

netsh winhttp set proxy localhost:8888
netsh winhttp reset proxy

https://www.michev.info/Blog/Post/2946/using-fiddler-to-examine-remote-powershell-sessions

https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/configure-proxy-internet?view=o365-worldwide#configure-the-proxy-server-manually-using-netsh-command

git-credential-manager.md

git credential manager on linux headless

Install through deb package, configure gpg cred store as documented here

https://github.com/git-ecosystem/git-credential-manager/blob/release/docs/credstores.md#gpgpass-compatible-files

Configure MS auth to use a device code

git config --global credential.msauthFlow devicecode

For azure devops

git config --global credential.https://dev.azure.com.useHttpPath true

For headless gpg

https://superuser.com/a/521027

In ~/.bashrc

export GPG_TTY=$(tty)

Install pinentry-tty

sudo apt-get install pinentry-tty

Add these lines to ~/.gnupg/gpg-agent.conf

pinentry-program /usr/bin/pinentry-tty
default-cache-ttl 86400
max-cache-ttl 86400

Reload configuration

gpg-connect-agent reloadagent /bye

Debug

GCM_TRACE=1 GIT_TRACE=1 git fetch

git-rebase.md

git rebase [A]
    replay commits of current branch on top of A
    
git rebase --onto [A] [B]
    replay commits of current branch on top of A, excluding anything found in B
    (so starting from all commits in current branch that come after the commits found in B)

    
git rebase --onto branch1_squashed branch1


current: branch2

git rebase master


git rebase --onto master branch1_unsquashed

git-submodules.md

git submodules

List submodules

git submodule status --recursive

Add a submodule

git submodule add [email protected]:org/repo.git <submodule path>

Remove submodule

git rm <submodule path>

Re-add a removed submodule

git submodule add --force [email protected]:org/repo.git <submodule path>

Update submodules to latest commit from remote

git submodule update --remote --init --recursive

gpg.md

GPG

List public keys

gpg2 --list-keys --keyid-format=long

List private keys

gpg2 --list-secret-keys --keyid-format=long

They keyid is the part after pub rsa4096/

Export public key

gpg2 --armor --export --output public.pgp $KEYID

Export private key

gpg2 --armor --export-secret-key --output private.pgp $KEYID

Git

Tell git which key to use and sign all commits

git config --global user.signingkey $KEYID
git config --global commit.gpgsign true

grep.md

Grep

Match recursive, case-insensitive, emit line number

grep -inr "pattern" path

Match specific filetype

grep --include \*.txt "pattern" filepath

Match several patterns

 grep -E "pattern1|pattern2" filepath

iptables.md

iptables

temporarily disable

https://help.ubuntu.com/community/IptablesHowTo

sudo iptables -P INPUT ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -F

allow a single port

To allow outgoing connections from server1 to server2 on TCP port 2194, use this on server1:

iptables -A OUTPUT -p tcp -d <server2ip> --dport 2194 -j ACCEPT

To allow incoming connections from server1 to server2 on TCP port 2194, use this on server2:

iptables -A INPUT -p tcp -s <server1ip> --dport 2194 -j ACCEPT

kubernetes.md

kubectl

View contexts

kubectl config get-contexts

Get resources

kubectl -n $namespace get pods

Describe individual resource

kubectl -n $namespace describe pod $pod

Restart pods

kubectl -n $namespace rollout restart deployment $deployment

linux.md

Linux utils

Disk usage

check disk usage of directory:

du -ch dir_name | grep total

top 5 largest files:

sudo find ~ -type f -ls | sort -n -r -k 7,7 | head -n 5

Search for files

find a file in directory (and subdirs):

find dir_name -name 'file name'

bash

variables:

export PATH=$PATH:<your new location>

redirect stdout and stderr:

<cmd> 2>&1 | tee file.txt

loop:

for i in {1..9};do wget http://cms.ipspace.net/slides/224d60ce-891d-11e1-8fb7-005056880254/00$i.jpg;done
for i in {10..18};do wget http://cms.ipspace.net/slides/224d60ce-891d-11e1-8fb7-005056880254/0$i.jpg;done

virtual box

VBoxManage convertfromraw /path/to/usb.img /path/to/usb.vdi --format vdi
VBoxManage internalcommands converttoraw /path/to/usb.vdi /path/to/usb.img

deb

install:

dpkg -i name.deb

search:

sudo apt-cache search name

list installed:

apt list --installed

list files installed by package:

dpkg -L name

rpm

install:

rpm -ivh name.rpm

find name:

rpm -qa | grep name

uninstall:

rpm -e package_name

display info:

rpm -qi name

display all related files:

rpm -qli name

network ports

to see ports used:

netstat -antpe | grep <port>

users

add user to secondary group

useradd -G {group-name} username
usermod -G [group1,group2,...] username

View live VM information on hosts

virsh list
virsh dumpxml <vm_deploy_id>
# (attach does not work in RHEL 6)
virsh attach-interface domain_id bridge bridge_name --target oif0
virsh attach-interface 35 bridge br1
virsh detach-interface 35 bridge --mac 02:00:c0:a8:20:01
virsh create domain.xml

linux_performance.md

Disk

Test write speed

 dd if=/dev/zero of=test.file bs=10MB count=1000 oflag=dsync

https://linuxreviews.org/HOWTO_Test_Disk_I/O_Performance

Test read speed

Flush caches

sync
echo 3 > /proc/sys/vm/drop_caches
echo 3 | sudo tee /proc/sys/vm/drop_caches

 dd if=test.file of=/dev/null bs=8k

Network

machine1

nc -vvlnp 5001 >/dev/null

machine2

dd if=/dev/zero bs=10M count=1K | nc -n machine1 5001

alternative:

machine1

sudo firewall-cmd --add-port=5001/tcp
iperf -s

machine2

iperf -c machine1

increase TCP window size

$ cat /proc/sys/net/ipv4/tcp_rmem
4096    87380   6291456
$ cat /proc/sys/net/ipv4/tcp_wmem
4096    16384   4194304
$ cat /proc/sys/net/core/rmem_max
212992
$ cat /proc/sys/net/core/wmem_max
212992

sysctl -w net.core.rmem_max=6291456
sysctl -w net.core.wmem_max=4194304

networking.md

Networking

To add a route to subnet 192.168.31.0/24 through the gw:

route add -net 192.168.31.0/24 gw 192.168.32.100 eth2

to see network routes:

netstat -nr
route -v

to check port:

netstat -antpe | grep <port>

add/remove ip address:

ip addr add 10.10.7.33/24 broadcast 10.10.7.255 dev eth0
ip addr delete 10.10.7.33/24 dev eth0
ip -4 link show up
ip -4 addr show up

add eth alias:

ifconfig eth0:1 192.168.31.2 up

linux bridge

to bridge a vlan device:

brctl addbr brname
ip link set up brname
ip link add link eth0 name eth0.20 type vlan id 20 (or vconfig add eth0 20)
ip link set up eth0.20
brctl addif brname eth0.20

ovswitch

!!! if exporting nfs to a host with the eth0 enslaved on the ovs bridge, the ovs bridge should have an IP address and the export should be done on that IP address !!!

ovs-vsctl list-br
ovs-vsctl add-br br0
ip link set up br0
ovs-vsctl del-br br0cat
ovs-vsctl list-ports br0
ovs-vsctl add-port br0 eth0
ovs-vsctl add-port br0 tap0 tag=1
ovs-vsctl del-port br0 tap0
ovs-vsctl port-to-br tap0
ovs-vsctl get Port vnet1 tag

Script to ensalve eth0 with a bridge br100:

#! /bin/bash

#ip addr del 10.80.80.25/24 dev eth0
#ip addr add 10.80.80.25/24 dev br100
ip link set up br100
brctl addif br100 eth0
/sbin/route del -net 10.80.80.0 netmask 255.255.255.0 dev eth0
/sbin/route del -net 0.0.0.0 gw 10.80.80.1 dev eth0
/sbin/route add -net 10.80.80.0 netmask 255.255.255.0 dev br100
/sbin/route add -net 0.0.0.0 gw 10.80.80.1 dev br100

enable IP forwarding

/etc/sysctl.conf
net.ipv4.ip_forward = 1

run:

sysctl -p
service network restart

check if it is applied:

cat /proc/sys/net/ipv4/ip_forward

wireshark

yum install wireshark wireshark-gnome

arp

arp -a

scan open ports

sudo nmap -sT -p- <host or ip>

postgres_parallel.md

Make sure

• no temp table, materialized CTE, or correlated subquery is used
• no INSERTs, only CREATE TABLE AS

May help:

• SET parallel_setup_cost = 0
• SET parallel_tuple_cost = 0

Parallel GIN it's possible to parallelize GIN access with partitions and parallel append

• partition the table containing the ts vectors, then try to get a parallel append from it
• if the plan does not parallelize per partition, you can force it by manually querying its partition seperately and combining with UNION ALL

profiling.md

Basic time/memory profile of a script

/usr/bin/time -v $SCRIPT

pyarrow.md

Install pyarrow from source on CentOS 7 Python 3.8

Install prerequisites

sudo yum install -y https://apache.bintray.com/arrow/centos/$(cut -d: -f5 /etc/system-release-cpe)/apache-arrow-release-latest.rpm
sudo yum install -y arrow-devel arrow-glib-devel arrow-dataset-devel parquet-devel parquet-glib-devel arrow-python-devel snappy snappy-devel brotli brotli-devel utf8proc utf8proc-devel 

Also install cmake from source

https://gitlab.kitware.com/cmake/cmake/-/blob/a0f2e8cbfaf62f2a15adf4bf58170f81b7fea250/README.rst

./bootstrap && make && sudo make install

Install in a virtualenv

python3.8 -m venv venv
source venv/bin/activate
pip install pyarrow --no-binary :all:

python.md

Install

yum install gcc openssl-devel bzip2-devel libffi-devel zlib-devel ncurses-devel xz-devel gdbm-devel db4-devel sqlite-devel tkinter python3-tkinter uuid-devel readline-devel
./configure --enable-optimizations
sudo make altinstall

python_profiling.md

Profile running process

Py-Spy

Install

pip install py-spy

Profile

py-spy record -o flamegraph.svg --pid 118460 --native --gil
py-spy record -o speedscope.svg --pid 118460 --format speedscope --native --gil

Upload speedscope to https://www.speedscope.app/

Py-Spy in docker process

Copy py-spy executable inside the container, and run it with privileges

docker cp py-spy $CONTAINER:/home/user/
docker exec -it --privileged $CONTAINER /home/user/py-spy ...

screen.md

GNU screen

http://aperiodic.net/screen/quick_reference

• screen -S name create a screen
• screen -ls list screens
• screen -r name attach to screen
• Ctrl-shift-A D get out of the screen
• Ctrl-shift-A Esc enter copy mode, allowing scroll. Esc again to exit

https://joshtronic.com/2018/12/10/gnu-screen-versus-tmux-hotkeys/

smb.md

Windows SMB server

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/direct-hosting-of-smb-over-tcpip

SMB over TCP, port 445

Check status

From windows terminal

net config server

Access from WSL2

telnet $(hostname).local 445
smbclient -L $(hostname).local

sudo.md

Allow user to run specific commands as sudo

sudo visudo -f /etc/sudoers.d/username

%user ALL= NOPASSWD: /command arg1 arg2
%user ALL= NOPASSWD: /command2 arg1 arg2
# allow sudo -i -u user2 w/o password
%user ALL=(user2) NOPASSWD: /bin/bash

tar.md

tar.gz

Untar

tar xzf archive.tar.gz

Create an archive containing the data folder

tar czf archive.tar.gz data

tmux.md

TMUX

tmux ls
tmux new -s test
tmux attach -t test
tmux kill-session -t test

Reload tmux.conf

tmux source-file ~/.tmux.conf

detach: Ctrl-b d
create window: Ctrl-b c
go to window 1: Ctrl-b 1
list windows: Ctrl-b f
rename window: Ctrl-b ,
scroll: Ctrl-b [

vertical pane: Ctrl-b %
horizontal pane: Ctrl-b "
cycle panes: Ctrl-b o
show pane numbers: Ctrl-b q
go to pane number 1: Ctrl-b q 1
zoom into/out of pane: Ctrl-b z
exit pane: Ctrl-b x
enter command: Ctrl-b :
resize

https://joshtronic.com/2018/12/10/gnu-screen-versus-tmux-hotkeys/

https://mutelight.org/practical-tmux

Conf

~/.tmux.conf

#set -g default-terminal "screen-256color"
#set -g mouse on
setw -g aggressive-resize on
set -g base-index 1
set-window-option -g window-status-current-style bg=red
set -g mouse off
set -g default-terminal "screen-256color"
#bind -n WheelUpPane copy-mode
set-option -g allow-rename off
set-window-option -g mode-keys vi

CentOS

https://github.com/tmux/tmux/wiki/Installing

sudo yum install libevent-devel ncurses-devel gcc make pkg-config
wget https://github.com/tmux/tmux/releases/download/3.0a/tmux-3.0a.tar.gz
tar xzf tmux-3.0a.tar.gz
cd tmux-3.0a
./configure
make && sudo make install

Issues

protocol version mismatch (client 7, server 6)

https://unix.stackexchange.com/a/126578

Connect to existing session and exit it

$ pgrep tmux
3429
$ /proc/3429/exe attach

venv.md

Virtualenv

Create in home directory

python -m venv ~/.virtualenvs/project-name --prompt venv