Some useful tools and tweaks for new systems. Probably only useful for me, @jtmoon79.
- Useful Tools
- System setup
- Program frameworks
- Recommended Linux system changes
- Recommended Windows system changes
- Miscellaneous
- TODO notes
Install my bash profile
curl --silent 'https://raw.githubusercontent.com/jtmoon79/dotfiles/master/install.sh' | bash --norc --noprofile
Configure locales
dpkg-reconfigure locales
First
apt update
Second, install apt helpers
apt install apt-utils
Package apt-utils is required for proper configuration of many other apt packages; install it first!
Suggested, more packages that will help building of other packages
build-essential autoconf automake libtool cmake pkg-config libssl-dev fakeroot
Suggested, more packages that will help installation of other packages
apt-transport-https apt-file aptitude software-properties-common gnupg debian-keyring debian-archive-keyring man-db manpages
Run apt-file update after installing apt-file.
Optional, update locales from default C.UTF-8 and POSIX
dpkg-reconfigure locales
Optional, an Apt UI
aptitude
Do this if seeing characters rendering incorrectly.
info info2man
information pagesman-dbmanpagesmanpages-dev
manmanual pagesmlocate nocache
programlocate,nocachehelps ittmux
GNUtmuxx11-apps
xeyesand othersmoreutils
see https://packages.debian.org/sid/moreutilschronic: runs a command quietly unless it failscombine: combine the lines in two files using boolean operationserrno: look up errno names and descriptionsifdata: get network interface info without parsing ifconfig outputifne: run a program if the standard input is not emptyisutf8: check if a file or standard input is utf-8lckdo: execute a program with a lock heldmispipe: pipe two commands, returning the exit status of the firstparallel: run multiple jobs at oncepee: tee standard input to pipessponge: soak up standard input and write to a filets: timestamp standard inputvidir: edit a directory in your text editorvipe: insert a text editor into a pipezrun: automatically uncompress arguments to command
util-linux
some of the programs among many otherslscpulsblkmoresublkidfsckmkfsmkswapflocklsipclslockslsmemlsloginslsnswhereis
bsdutils
programsloggerrenicescriptscriptlivescriptreplaywall
sharutils
uuencode,uudecode,shar,unshar
file
programfilefish
a better shellzoxide fzf
smarter cd programzreptyr
reparent an orphaned processuuid-runtime
programuuidgen
dropwatch
cmake make g++ gcc gdb gdb-doc libc6-dbg cpp autoconf automake build-essential pkg-config libtool gettext cmake
typical build + make requirements
htopbottom
cargo install --locked bottom(runbtm)bashtop
pretty TUI graphics system monitorapt install bashtop sysstat- or install manually
git clone https://github.com/aristocratos/bashtop && cd bashtop && make install
lshwpciutils lsscsi hdparm
programslspci,lsscsi, andhdparmusbutils usb.ids
programlsusbprocps lsof
various process tools;pstree,ps,lsoflsb-release lsb-core
programlsb_releasechkservice
systemd CUIwhowatch
Watch logins live.read-edid
programget-edidto display monitor info thatlshwmay not showlddtree
show shared library dependencies of a program in a tree formatcargo install --locked lddtreedisktest
disk read/write tester for SSD and NVMecargo install --locked disktestsystemctl-tui
Helpful TUI for viewing systemctl servicescargo install --locked systemctl-tui
debian-keyring debian-archive-keyring
for commandgpg --verifychkrootkit clamav
root kit scanner for clam AntiVirus see https://archive.ph/Gy4y2
curl wget httpienmap netcat socat
nmapandnc
netcatmight benetcat-traditionalornetcat-openbsdnet-tools
various small network utilities likeifconfig,arp,route, etc.inetutils-traceroute
traceroutebind9-dnsutils
dig
dog
dig on steriods a better dig
ipcacliperf3
measure net bandwidthiputils iputils-arping iputils-clockdiff iputils-tracepath ipv6calc ipv6toolkit
more various small network utilitiesnmap
nmapnethogs
net top toollftp
lightweight ftp clientlynx elinks w3m
CLI web browsers; elinks is the bestmosh autossh sshpass
resilient SSH connectionsethtool
ethernet interface settings tooltcpdumpmtr-tiny"My TraceRoute"
Call withmtr --curses.
mtris a combination ofpingandtraceroute.nast
network analyzer and other miscellaneous toolsiptraf-ng
live network monitortermscp
TUI file transfers
install withcurl --proto '=https' --tlsv1.2 -sSLf "https://git.io/JBhDb" | sh
or
cargo install --locked termscpsmbclient
SMB client
tree
treebroot
tree improved
parted fdisk
disk formatting: parted fdisk cfdisk sfdiskacl
getfacl, setfaclzip unzip xz-utils tar p7zip p7zip-full p7zip-rar lrzip lz4 lzip unar
zip, unzip, xz, tar; some may already be installedsshfs
ssh filesystem in fusezerofree
zero empty disk blocks; useful for archiving disk backupsduc-nox
interactive disk usage visualizer
Run commandduc-nox ui /ncdu
interactive disk usage visualizer
Run commandncdu /yazi
apt install jq unar poppler fzf ffmpegthumbnailer
ffmpegthumbnailer adds a lot
poppler cannot be found!?cargo install --locked fd-find ripgrep zoxidecargo install --locked yazi-fm
- fio disk benchmarking
apt install fio disktest
disk read/write tester for SSD and NVMecargo install --locked disktest
powershellpython3 python3.8 python3.9 python3.10 pypy pypy3 pypy-dev pypy3-dev
Pythonsopenjdk-21-jre
Java Runtime Engine
colordiff
diff with color!gettext
msgcat --color=testpdfgrep
pdfgreppoppler-utils poppler-data
pdftotext, among many other PDF toolsjq jo
json handlersfiglet
draw big lettershexcurse
hex editorshed
nano-like hex editornano
nano simple text editorvim
lame lame-doc flac oggz-tools vorbis-tools
mp3, flac, ogg toolsvlc
VLC media player, many dependencies!libimage-exiftool-perl
toolexiftoolis a scriptble file tag reader/writer, supports many file types
goaccess geoip-database
goaccess web server log analyzerpv
Pipe Viewer, e.g.dd in=/dev/sda1 | pv | 7z -a -si sda1.imgfq
Binary file viewer inspired by jqrsyslog
See section rsyslog settingsmultitailgit-crecord
interactive chunk staging for gitcurl -sL https://git.io/_has | sudo tee /usr/local/bin/has >/dev/null
hascommand to check for installed programshstrbetter command history search
after install runhstr --show-configuration >> ~/.bashrc.local.postbinds tocrtrl-rby defaulttenki
nice terminal screensaver with weathercargo install --git https://github.com/ckaznable/tenki.git- Install Docker, groups and users
- Install Powershell
ct
bash command tracergit clone https://github.com/JB63134/bash_ct.git ~/.local/bin/bash_ct echo "bash_source_file ~/.local/bin/bash_ct/.bash_ct" >> ~/.bashrc
cd /tmp
git clone https://github.com/VirusTotal/vt-cli
cd vt-cli
make install
vt init
vt scan file /tmp/suspicious-file
vt analysis
( cat /var/log/{dpkg.log,dpkg.log.1} && zcat /var/log/dpkg.*.gz ) \
| grep -i "status installed" \
| sort -k5 -k1 -k2 \
| cut -f5 -d' ' \
| uniq
apk add \
bash bash-completion \
tmux \
file \
diffutils `# diff` \
coreutils `# sort cp cat mv ... (many Linux utils)` \
htop \
curl wget \
zip unzip tar xz p7zip \
git \
vim \
python3 \
gcc g++ \
lsof lshw \
alpine-conf \
util-linux procps \
man-pages man-db `# man pages` \
tzdata `# setup instructions https://wiki.alpinelinux.org/wiki/Setting_the_timezone` \
pkg install \
htop \
pstree \
sudo \
arping \
iftop \
iperf3 \
SKIP THIS SECTION, CAUSES BOOT ERRORS
Derived from https://docs.netgate.com/pfsense/en/latest/recipes/freebsd-pkg-repo.html
In file /usr/local/etc/pkg/repos/pfSense.conf change to
FreeBSD: { enabled: yes }
In file /usr/local/etc/pkg/repos/FreeBSD.conf change to
FreeBSD: { enabled: yes }
The full list of FreeBSD packages is at https://pkg.freebsd.org/FreeBSD:12:amd64/
For speedtest.net CLI app
pkg install -g libidn2 ca_root_nss
pkg add "https://install.speedtest.net/app/cli/ookla-speedtest-1.2.0-freebsd13-x86_64.pkg"
- Sysinternals Suite
- PowerToys
- ExifTool General purpose image tag reader, many formats supported
- Event Log Inspector Send Windows events as syslog messages to a syslog server (archived locally)
- gopeed Download Manager
- sniffnet Very pretty network sniffer and monitor
See section chocolatey.
See the section Enable Long Paths (Powershell)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env
# or
echo 'source "$HOME/.cargo/env"' >> ~/.bashrc.local.post
# or
echo 'bash_path_add "$HOME/.cargo/bin"' >> ~/.bashrc.local.post
Install cargo-cache and use it to clean the cache after these installs.
cargo install --locked cargo-cache
Some useful rust programs
cargo install --locked \
lsd `# improved 'ls'` \
bottom `# improved 'top'` \
ripgrep `# faster recursive grep` \
bat `# improved 'cat'` \
procs `# improved 'ps'` \
sd `# intuitive find+replace`\
cargo-update \
hexyl `# improved 'hexdump'` \
coreutils `# improved GNU Linux utils` \
macchina `# neofetch system viewer` \
rustscan `# fast nmap` \
bingrep \
exa `# improved 'ls'` \
hyperfine `# process perf measurement` \
systeroid systeroid-tui `# sysctl TUI` \
diskonaut `# fast disk usage TUI` \
rhit `# nginx log analyzer` \
jaq `# improved 'jq' clone`\
fd-find `# improved 'find'` \
du-dust `# improved 'du'` \
tealdeer `# tldr; shorter 'man', 'info' pages` \
dog `# dig on steriods` \
broot `# improved tree` \
gping `# TUI ping histogram` \
cargo-outdated `# run cargo outdated; requires apt "libssl-dev"` \
gitoxide `# fast git` \
termscp `# TUI file transfers` \
After the installs
cargo cache --autoclean
Periodically run cargo-update
cargo install-update -a
Also see
apt install python-is-python3 python3-pip python3-virtualenv python3-venv python3-wheel python3-build pipx
curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
You can also use pip to get the latest versions since the apt versions are often a few years old.
python -B -O -s -m pip install \
--no-cache-dir --disable-pip-version-check \
--upgrade wheel pip setuptools venv virtualenv
If the command fails try without -s.
To create a virtual environment run venv-create.sh
curl --silent 'https://raw.githubusercontent.com/jtmoon79/dotfiles/master/utilities/venv-create.sh' | bash --norc --noprofile
pipx install \
ipython `# popular debugger and interactive shell` \
ptpython `# popular debugger and interactive shell` \
six `# most downloaded pip package, might as well` \
black `# code formatter` \
Procpath `# powerful process monitor` \
dns-benchmark-tool `# great looking DNS Benchmark` \
To install, see https://go.dev/doc/install
pdfcpu
PDF manipulation toolgo install github.com/pdfcpu/pdfcpu/cmd/pdfcpu@latest
Update-Help -UICulture en-US
Run update-help once per install of Powershell. Updates are rare.
Install-Module Pscx -Scope CurrentUserLater
Update-Module PscxFound in this SO Answer.
Check
Get-ExecutionPolicy -ListSet the most lacking policy restricted to current user
Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope CurrentUserTODO: add these turned off Bitlocker TPM (https://archive.ph/WKfOr) turned off automatic update reboot (http://archive.vn/BEs3r)
https://stackoverflow.com/questions/66701954/skip-powershell-startup-check-for-new-version
apt install logrotate
systemctl status logrotate
On Debian-based systems
apt install acct
systemctl enable acct
systemctl start acct
On RedHat-based systems
yum install psacct
systemctl start psacct
Accounting records are stored in /var/log/account/pacct.
See accounting totals with
ac --individual-totals
sa --print-users
sa --user-summary
Add an entry for logrotate.
echo '# acct
/var/log/account/acct
/var/log/account/pacct {
weekly
rotate 50
compress
delaycompress
missingok
notifempty
create 0660 root root
sharedscripts
postrotate
/usr/sbin/accton off
/usr/sbin/accton on
endscript
}' > /etc/logrotate.d/acct
Test logrotate with
logrotate -v -d /etc/logrotate.d/acct
apt install auditd
systemctl enable auditd
sudo visudo
add line
Defaults:USER timestamp_timeout=720
A snippet of settings to add to /etc/rsyslog.conf to send logs to a remote server.
# /etc/rsyslog.conf
module(load="imjournal") # added by jtmoon https://www.loggly.com/ultimate-guide/centralizing-with-syslog/
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
$ActionFileDefaultTemplate RSYSLOG_FileFormat
# from https://www.linuxtechi.com/setup-rsyslog-server-on-debian/
# Enable sending system logs over UDP to rsyslog server
*.* @192.168.X.Y:514
# Enable sending system logs over TCP to rsyslog server
#*.* @@192.168.X.Y:514
A hostname may be used in place of the IP address.
Then restart the service
systemctl restart rsyslog
On the rsyslog server, add an entry for host to /etc/logrotate.d/rsyslog
/var/log/rsyslog/NEWHOST/*
/var/log/rsyslog/newhost/*
/var/log/rsyslog/newhost.net/*
/var/log/rsyslog/newhost.local/*
{
rotate 999
notifempty
size 1M
compress
createolddir
copytruncate
olddir /var/log/rsyslog-old/newhost
}
For openssh sshd server.
Append to /etc/ssh/ssh_config
ServerAliveInterval 10
ServerAliveCountMax 12
TCPKeepAlive no
Append to /etc/ssh/sshd_config
ClientAliveInterval 10
ClientAliveCountMax 24
TCPKeepAlive no
IME, TCPKeepAlive causes too many dropped SSH sessions. The internal application Alive
mechanism is more forgiving of lossy connections.
From SO Q&A.
For openssh sshd server.
For sshd service exposed to The Internet, disallow well-known users from login (i.e. root). Only allow root login from a private network or localhost.
Replace secretuser with a non-obvious user. This is the user that may login from The Internet.
# this sshd is exposed to The Internet so limit allowed user logins
AllowUsers root@10.* root@192.168.* root@127.0.0.1 secretuser
Append to /etc/default/cron
EXTRA_OPTS="-L 7"
systemctl edit --full cron.service
then edit the ExecStart
From the instructions at htop-dev/htop.
# install build dependencies
apt install libncursesw5-dev autotools-dev autoconf automake build-essential libtool
# install extras
apt install libcap-dev libcap2 libcap2-bin libsensors-dev libhwloc15 libhwloc-common libhwloc-dev libhwloc-doc libtool-doc libhwloc-plugins libtool
# get the source code
VER=3.2.2
cd /tmp
wget https://github.com/htop-dev/htop/archive/refs/tags/${VER}.zip
unzip ${VER}.zip
cd htop-${VER}
# build and install
./autogen.sh
./configure --enable-unicode --enable-hwloc --enable-sensors --enable-capabilities
make
make install
On a yum-managed distribution
yum install ncurses ncurses-devel lm_sensors lm_sensors-devel hwloc hwloc-libs hwloc-devel libpcap libpcap-devel libcap libcap-devel
touch ~/.gnupg/gpg-agent.conf
echo default-cache-ttl 3600 >> ~/.gnupg/gpg-agent.confFrom this SO answer.
useradd -M iperf
touch /etc/systemd/system/iperf3.serviceCreate the service definition file
# copied from https://futurereboot.com/iperf3-as-a-service/
[Unit]
Description="iperf3 server"
After=network.target
[Service]
Type=simple
User=iperf
ExecStart=/usr/bin/iperf3 -s
KillMode=process
Restart=on-abort
[Install]
WantedBy=multi-user.targetenable the service
systemctl enable --now iperf3check the logs
journalctl -u iperf3 -fFrom this article (archived).
See https://github.com/jtmoon79/dotfiles/tree/master/services/endlessh.
Tested on a Windows 10 Pro system.
TODO: Fill in
gpedit.mscComputer ConfigurationAdministrative TemplatesAll Settings- Configure Automatic Updates
Set the login to use the default powershell
Set-ItemProperty -verbose -path "HKLM:\HKEY_LOCAL_MACHINE\SOFTWARE\OpenSSH" -Name "DefaultShell" -Value $(Get-Command powershell.exe).SourceSet the login back to the default cmd.exe invocation by the empty string.
Set-ItemProperty -verbose -path "HKLM:\HKEY_LOCAL_MACHINE\SOFTWARE\OpenSSH" -Name "DefaultShell" -Value ""New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `
-Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -ForceInstall choco from chocolatey.
As Administrator, run in powershell
Set-ExecutionPolicy Bypass -Scope Process -Force
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
iex ((New-Object System.Net.WebClient).DownloadString("https://community.chocolatey.org/install.ps1"))Check with
choco --version
First install the GUI and extensions
choco install ChocolateyGUI chocolatey-core.extension
Check packages installed
choco list --local-only
See chocolatey packages.
winaero-tweaker
system tweakerwindirstat
disk usage analyzersysinternals
system utilitieswinmerge
file diffcpu-z.install
system infohwinfo.install
system infocoretemp
system info for temperaturespowertoys
system utilitiesntop.portable
htop for Windowsgopeed
download managertermscp
terminal UI remote shell and file copy client
Install these independely from choco.
msys2
a whole 'nother linux-like subsystem packages undermysys2can be found usingpkg(I THINK?)cygwin
a whole 'nother linux-like subsystem usecygcheck -cto list installed packages usecygcheck -pto find which package a file belongs towsl
- use Turn Windows Features On Or Off
- Windows Subsystem For Linux
- Microsoft Store to choose the OS
- use Turn Windows Features On Or Off
7zip
file archiver7zip-zstd
with zstd support (more compression algorithms)
notepadplusplus
text editornotepad3
text editorhxd
hex editorwinmerge
file diffposhgit
git tab-completions in Powershell
gnuwin32-coreutils.portablenano
handy console text editorgawksedawkgrepdiffutilsfindutilstacjq
winscp.install
sftp clientstunnel
SSL tunnelingiperf3
network performance testingcurl
command line http clientwget
command line http clientlftp
command line ftp clienttermscp
terminal UI remote shell and file copy clientputtyorputty-portable
ssh clienttelnet
a basic telnet client
sumatrapdf.install
pdf viewer
imgburn
CD/DVD/ISO burningwin32diskimager.portable
disk copyingdvddecrypter
DVD ripping
irfanview
image viewer
eartumpet
volume controlfoobar2000
audio player
mpc-hc-clsid2
media player classicvlc
video player
Update to the latest WSL2 kernel.
In file $env:USERPROFILE\.wslconfig (notepad.exe "$env:USERPROFILE\.wslconfig") to set contents
[wsl2]
memory=500MB
processors=2Inside the distribution, edit file /etc/wsl.conf
[boot]
systemd=trueDo not regenerate /etc/resolv.conf per start.
Inside the distribution, edit file /etc/wsl.conf
[network]
generateResolvConf=falseInside the distribution, edit file /etc/wsl.conf
[network]
hostname=host-ubuntu22TODO:
https://awakecoding.com/posts/powershell-remoting-trusted-hosts-what-does-it-mean/
On Debian-derived requires packages (among others)
apt install libtool libssh2-1-dev zstd libgssapi-krb5-2 libzstd-dev nghttp2
Link to latest source code
wget https://github.com/curl/curl/archive/refs/tags/curl-7_79_1.zip
unzip curl-7_79_1.zip
cd curl-7_79_1
autoreconf -fi
./configure --with-openssl --with-nghttp2 --with-ngtcp2 --with-gssapi --with-libssh2 --with-zstd
make && make test && make install
Step make test is optional.
By default, installs to /usr/local. To change, pass --prefix=... to configure command.
docker run -it --rm curlimages/curl:latest --version
Add git alias dog for pretty graphs
git config --global alias.adog "log --all --decorate --oneline --graph"
Install git-crecord. Interactive git staging using ncurses.
apt install git-crecord
Like the old /etc/rc.local script, a custom systemd service to run
miscellaneous commands at system startup.
From Running a Linux Command on Start-Up
touch /lib/systemd/system/startup.service
Service file startup.service
[Unit]
Description=Startup Script
[Service]
ExecStart=/root/.local/bin/startup.sh
[Install]
WantedBy=multi-user.target
Startup script
touch /root/bin/startup.sh
chmod +x /root/bin/startup.sh
script startup.sh
#!/usr/bin/env bash
#
# manual script run by systemd at system start
set -eux
logger -s -t 'startup.sh' "running startup.sh"
add whatever other commands are needed to startup.sh
Enable and test
systemctl enable startup.service --now