Hetzner AX41-NVMe Setup Docs

00-hetzner-ax41-nvme-setup.md

Hetzner AX41-NVMe Setup

Setup docs for Hetzner AX41-NVMe dedicated server (AMD Ryzen 5 3600, 64 GiB RAM, 2x 512 GB NVMe).

export HETZNERIP="your ip address"

Order: B20260219-3371017-2946215 (19 Feb 2026) Server: AX41-NVMe #2855249, Helsinki (eu-west)

How to get a server like this

1. Go to https://www.hetzner.com/dedicated-rootserver/matrix-ax/ and pick an AX model
2. Pick a datacenter location (Helsinki, Falkenstein, etc.)
3. After ordering, the server is delivered in Rescue System (usually within minutes)
4. Hetzner emails you the IP, root password, and IPv6 subnet
5. SSH into the rescue system and run installimage to install your OS
6. After reboot, you have a clean Ubuntu server at the IP from the email

Pricing (as of Feb 2026): AX41-NVMe is ~38 EUR/mo for AMD Ryzen 5 3600, 64 GiB RAM, 2x 512 GB NVMe.

Reproducing the setup

Follow the docs below in order. Each file has copy-pasteable commands.

For agents: see gist.md for instructions on reading and updating this gist.

Docs

• 01-os-and-hardware.md — OS installation, disk layout
• 02-ssh-and-users.md — SSH keys, user creation, passwordless sudo, SSH config aliases, sshd hardening, fail2ban
• 03-dev-tools.md — nvm, node, rvm, ruby, uv, docker, gh, claude, codex
• 04-shell-config.md — PATH configuration, dotfiles
• 05-networking.md — UFW firewall, SSH tunnels
• 06-tmux.md — tmuxctl install, session management, memory isolation
• 07-agents-setup.md — Claude/Codex/OpenCode config and Z.AI-routed profiles (zlaude/zodex)
• 08-bots.md — Telegram bots, bot-master process manager, crontab

01-os-and-hardware.md

OS and Hardware

Hardware

• CPU: AMD Ryzen 5 3600 — 6 cores / 12 threads @ 3.6 GHz
• RAM: 64 GiB
• Storage: 2x 512 GB Samsung NVMe (MZVL2512HCJQ-00B00)

1. Connect to the instance

Server was delivered in Hetzner Rescue System. Connected via SSH with the provided root credentials:

ssh root@${HETZNERIP}

2. Install Ubuntu

Ran installimage with the following config (no RAID, second NVMe left untouched):

DRIVE1 /dev/nvme0n1
DRIVE2 /dev/nvme1n1
FORMATDRIVE2 0

SWRAID 0
SWRAIDLEVEL 1

BOOTLOADER grub

HOSTNAME hetzner

PART swap swap 8G
PART /boot ext4 1024M
PART /    ext4 all

IMAGE /root/.oldroot/nfs/images/Ubuntu-2404-noble-amd64-base.tar.gz

Installed Ubuntu 24.04 (Noble) on nvme0n1. Rebooted into the new OS.

3. Mount second NVMe at /data

mkfs.ext4 -L data /dev/nvme1n1
mkdir -p /data
mount /dev/nvme1n1 /data
chown alexey:alexey /data
echo '/dev/nvme1n1 /data ext4 defaults 0 2' >> /etc/fstab

Disk layout:

• nvme0n1 — 512 GB, Ubuntu (32G swap, 1G /boot, 444G /)
• nvme1n1 — 512 GB, ext4, mounted at /data (445 GB usable, owned by alexey)

02-ssh-and-users.md

SSH and Users

1. Create SSH key and copy to server

Generated a dedicated ed25519 key:

ssh-keygen -t ed25519 -f ~/.ssh/id_hetzner -C "hetzner" -N ""

Copied the public key to the server:

ssh-copy-id -i ~/.ssh/id_hetzner.pub root@${HETZNERIP}

2. Create user alexey with sudo

useradd -m -s /bin/bash -G sudo alexey
echo 'alexey:<password>' | chpasswd
mkdir -p /home/alexey/.ssh
cp /root/.ssh/authorized_keys /home/alexey/.ssh/
chown -R alexey:alexey /home/alexey/.ssh
chmod 700 /home/alexey/.ssh
chmod 600 /home/alexey/.ssh/authorized_keys

3. Passwordless sudo

# as root
echo "alexey ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/alexey
chmod 440 /etc/sudoers.d/alexey

4. SSH config aliases

~/.ssh/config on the local machine:

Host hetzner
    HostName ${HETZNERIP}
    User alexey
    IdentityFile ~/.ssh/id_hetzner
    StrictHostKeyChecking no

5. Harden sshd

Once SSH access and the new user are confirmed working, lock down the SSH daemon. Create a drop-in config:

sudo tee /etc/ssh/sshd_config.d/harden.conf > /dev/null << 'EOF'
PermitRootLogin prohibit-password
MaxStartups 10:30:100
EOF

• PermitRootLogin prohibit-password — root can only log in with an SSH key, never by password. A cracked root password is useless.
• MaxStartups 10:30:100 — limits simultaneous unauthenticated connections (starts dropping probabilistically at 10, drops everything at 100). Without this, bots can exhaust the handshake slots and lock out legitimate SSH.

Reload sshd (does not drop current sessions):

sudo systemctl reload ssh

6. fail2ban

Bans IPs after repeated failed login attempts, protecting against brute-force bots that constantly scan the internet.

sudo apt-get update -qq
sudo apt-get install -y fail2ban

Create /etc/fail2ban/jail.local:

[DEFAULT]
bantime = 1h
findtime = 5m
maxretry = 3
banaction = ufw

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 1h
findtime = 5m

banaction = ufw integrates bans with the firewall (see 05-networking.md).

Start and enable:

sudo systemctl enable fail2ban
sudo systemctl restart fail2ban

7. Optional: change the SSH port

Radically cuts down bot noise. Add to the same drop-in config:

echo 'Port 2222' | sudo tee -a /etc/ssh/sshd_config.d/harden.conf
sudo systemctl reload ssh

Open the port in UFW (see 05-networking.md) and connect with ssh -p 2222 user@server.

Verify

sudo fail2ban-client status sshd          # banned IPs, retry count
ss -tn '( sport = :22 )'                  # who is connected right now
sudo journalctl -u ssh -n 30 --no-pager   # recent login attempts

What this defends against:

• Brute force — bots scan the internet and try to guess root passwords; fail2ban bans them automatically.
• sshd DoS — without MaxStartups, bots can fill all handshake slots so sshd stops accepting new connections.
• Root compromise — prohibit-password blocks root login by password even if the password is guessed.

03-dev-tools.md

Dev Tools

All commands run on the server as the alexey user unless noted otherwise.

nvm (+ Node + npm)

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
source ~/.bashrc
nvm install --lts

This also installs nvm's PATH setup into ~/.bashrc automatically.

rvm + Ruby

gpg2 --keyserver hkp://keyserver.ubuntu.com \
  --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 \
              7D2BAF1CF37B13E2069D6956105BD0E739499BDB
curl -sSL https://get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm
rvm get stable              # update rvm's ruby database
rvm install 3.3.7           # latest stable (rvm list known may be stale, use explicit version)
rvm use 3.3.7 --default

uv

curl -LsSf https://astral.sh/uv/install.sh | sh

Docker (run as root)

curl -fsSL https://get.docker.com | sh
usermod -aG docker alexey

GitHub CLI

sudo apt install -y gh
gh auth login

Claude Code (after nvm/node are installed)

npm install -g @anthropic-ai/claude-code

Codex (after nvm/node are installed)

npm install -g @openai/codex

Node, Claude, and Codex all end up in the same bin dir (~/.nvm/versions/node/v24.13.1/bin/). Log in with codex login.

See 07-agents-setup.md for configuring Claude/Codex/OpenCode skills and settings, and the Z.AI-routed zodex profile.

04-shell-config.md

Shell Configuration (PATH)

The tool installers add their own entries to ~/.bashrc and ~/.bash_profile, but they can be messy (especially nvm's multi-line sourcing block). We replaced them with clean one-line-per-tool PATH exports.

Important: when managing dotfiles remotely from a Windows/MINGW shell, $HOME and $PATH get expanded locally before reaching the server. To avoid this, write the file via a python script on the server:

# SCP a python script to the server, then run it
scp fix_profile.py root@${HETZNERIP}:/tmp/
ssh root@${HETZNERIP} 'python3 /tmp/fix_profile.py'

The script writes ~/.profile with this content:

# Dev tools
export NVM_DIR="$HOME/.nvm"
export NODE_HOME="$NVM_DIR/versions/node/v24.13.1"
export PATH="$NODE_HOME/bin:$HOME/.local/bin:$HOME/bin:$PATH"

# rvm — static exports instead of `source ~/.rvm/scripts/rvm`
# avoids "GEM_HOME not set" warning in non-interactive shells
# to find paths for a new version: source ~/.rvm/scripts/rvm && rvm info
# update RUBY_VERSION after: rvm install <new> && rvm use <new> --default
export RUBY_VERSION="ruby-3.3.7"
export GEM_HOME="$HOME/.rvm/gems/$RUBY_VERSION"
export GEM_PATH="$HOME/.rvm/gems/$RUBY_VERSION:$HOME/.rvm/gems/$RUBY_VERSION@global"
export PATH="$HOME/.rvm/gems/$RUBY_VERSION/bin:$HOME/.rvm/gems/$RUBY_VERSION@global/bin:$HOME/.rvm/rubies/$RUBY_VERSION/bin:$HOME/.rvm/bin:$PATH"

Same block also appended to ~/.bashrc (after the non-interactive guard) so it works in both login and interactive shells.

Verified with su - alexey -c 'which node && which claude && which ruby && which uv && which docker':

/home/alexey/.nvm/versions/node/v24.13.1/bin/node
/home/alexey/.nvm/versions/node/v24.13.1/bin/claude
/home/alexey/.rvm/rubies/ruby-3.3.7/bin/ruby
/home/alexey/.local/bin/uv
/usr/bin/docker

Shell config chain

• ~/.bash_profile — sources ~/.profile
• ~/.profile — Dev tools + rvm exports + export BASH_ENV="$HOME/.bashenv"
• ~/.bashrc — interactive shell config, claude aliases, same Dev tools block at the end
• ~/.bashenv — rvm-only static exports (for non-interactive shells via BASH_ENV)

05-networking.md

Networking (Firewall)

Only port 22 (SSH) is open. All other ports are blocked. Use SSH tunnels to access services:

# as root
ufw allow 22/tcp
ufw --force enable

# access remote services via SSH tunnel from local machine
ssh -L 8080:localhost:8080 hetzner

Docker and UFW

By default, Docker bypasses UFW by manipulating iptables directly. This means ports published with -p become accessible from outside even if UFW blocks them.

Solution: Disable Docker's iptables manipulation

Create /etc/docker/daemon.json:

sudo tee /etc/docker/daemon.json << 'EOF'
{
  "iptables": false
}
EOF

Restart Docker:

sudo systemctl restart docker

This stops Docker from inserting its own firewall rules. Containers can still talk to each other via Docker networks, but published ports won't be accessible from outside. Services remain accessible via SSH port forwarding only.

06-tmux.md

tmux

This server uses tmuxctl as its tmux interface. It ships two executables, tmuxctl and t, where t is just the short alias for the same CLI.

1. Install tmuxctl

Primary install (from PyPI):

uv tool install tmuxctl

Or from a local clone:

git clone git@github.com:alexeygrigorev/tmuxctl.git ~/git/tmuxcli
cd ~/git/tmuxcli
uv sync
./install.sh        # adds the .venv to PATH and the tl alias to ~/.bashrc
source ~/.bashrc

2. Find and attach sessions

Run t with no arguments to see the most recent sessions with numeric indices:

IDX  SESSION               CREATED
1    git-llm-zoomcamp      2026-07-07 13:30:05
2    git-datamailer-queue  2026-07-03 22:43:12
...

Common usage:

t                        # list recent sessions
t l                      # list all sessions (alias: tl)
t <id>                   # attach by index from the list
t <session>              # attach by name
t :<name>                # create the session if needed, then attach
t -                      # create-or-attach a session named after the current folder
t - cy                   # same, running "cy" only in a freshly created session
t create-detached :<name> # create a memory-capped detached session without attaching
t kill <id|session>      # kill a session
t rename <session> <new> # rename a session
t jobs                   # manage recurring jobs (scheduled pings)
t doctor                 # diagnose OOM risk: RAM, cgroup kills, robust scopes
t strays                 # scan all tmux sockets for stray/orphan sessions
t reap                   # kill idle servers and remove dead sockets (guarded)

Rule of thumb: t codex attaches only; t :codex creates or attaches.

3. Memory isolation

tmux runs one server per socket. If one pane starts a runaway build or agent and exhausts RAM, the kernel can kill that shared server and take down every session with it. tmuxctl avoids this by starting each new session's shell under its own systemd user scope with memory caps:

tmuxctl-server.slice
└── tmuxctl-server.service
    └── tmux server

robust.slice
├── tmuxctl-project-a.scope  MemoryHigh=10.2G MemoryMax=12G
└── tmuxctl-project-b.scope  MemoryHigh=20.4G MemoryMax=24G

The shared tmux server stays in an uncapped slice, while each session lives in its own capped scope. As a scope crosses its soft MemoryHigh threshold the kernel throttles and reclaims (spilling cold pages to swap); only if it reaches the hard MemoryMax are processes inside that scope killed, leaving the tmux server and unrelated sessions untouched.

Defaults: MemoryMax=12G, MemorySwapMax=8G, MemoryHigh at 85% of MemoryMax. Override per-user in ~/.config/tmuxctl/cgroups.toml, or per-project in a cgroups.toml / [tool.tmuxctl] block.

Key bindings

All commands start with Ctrl+b (prefix), then a key:

Sessions

• d — detach (session keeps running)
• s — list/switch sessions
• $ — rename session

Windows (tabs)

• c — new window
• n / p — next / previous window
• 0-9 — switch to window by number
• , — rename window
• & — close window

Panes (splits)

• " — split horizontal
• % — split vertical
• arrow keys — move between panes
• z — toggle pane fullscreen (zoom)
• x — close pane
• { / } — swap pane left/right
• Ctrl+arrow — resize pane

Copy/scroll mode

• [ — enter scroll mode (use arrow keys, Page Up/Down)
• q — exit scroll mode

With set -g mouse on in ~/.tmux.conf, you can scroll with the mouse wheel.

07-agents-setup.md

Agent Setup: Claude, Codex, OpenCode

All AI assistant tooling (skills, settings, aliases, wrappers) is managed by the /home/alexey/git/.agents repo. Clone once, then configure targets.

Install

# one-liner (clones to ~/git/.agents, configures all default targets)
curl -sSL https://raw.githubusercontent.com/alexeygrigorev/.agents/main/installer.sh | bash
source ~/.bashrc

Or, if already cloned:

cd ~/git/.agents
./configure.sh --yes all
source ~/.bashrc

CLIs

Install the CLI binaries (after nvm/node are installed — see 03-dev-tools.md):

npm install -g @anthropic-ai/claude-code
npm install -g @openai/codex

Targets

./configure.sh claude      # symlink skills into ~/.claude, sync settings
./configure.sh codex       # sync Codex config and shared skills into ~/.codex
./configure.sh opencode    # symlink skills into ~/.config/opencode
./configure.sh all         # everything except zlaude/zodex

The default all target configures claude, codex, and opencode. CLI wrappers are installed to ~/bin and ~/bin is added early to PATH.

Claude aliases

The claude target symlinks shared skills into ~/.claude/skills and syncs settings. It adds these aliases to ~/.bashrc: c, cc, csp, ccsp, claude_init.

codex-zai-proxy (zodex)

zodex is an opt-in target that sets up a separate Codex profile under ~/.zodex routed to Z.AI (GLM models) through the zai-codex-proxy Responses-to-Chat bridge. It is not part of all because it prompts for an API key.

./configure.sh zodex

This writes Codex config to ~/.zodex/config.toml, proxy config to ~/.zodex/codex-proxy/config.json, and the Z.AI key to ~/.zodex/zai.env. Use it via the zodex wrapper; use zy for the same profile with Codex's bypass/yolo flag.

The proxy runs on 127.0.0.1:18765 and is started (or verified running) automatically by the zodex/zy wrappers via scripts/zodex_start_proxy.sh. The startup script:

• Checks whether a proxy is already serving; if so, keeps the running process alone so active sessions are not disconnected.
• Otherwise checks the latest zai-codex-proxy release, downloads it into ~/.zodex/bin when the local binary is missing or older, and starts it.
• Set ZODEX_PROXY_RESTART_ON_UPDATE=1 to force an immediate restart after an update.

Start or update the proxy manually:

~/git/.agents/scripts/zodex_start_proxy.sh

zlaude (Z.AI-routed Claude)

zlaude is the Claude counterpart: a separate Claude Code profile under ~/.zlaude routed to Z.AI via ANTHROPIC_BASE_URL. Also opt-in.

./configure.sh zlaude

Use it via the z / zc / zsp aliases (mirroring c / cc / csp).

Verification

codex --version            # Codex CLI
zodex --version            # Z.AI-routed Codex profile
ls ~/.claude/skills        # shared skills linked

08-bots.md

Telegram Bots

Bots live in ~/bots/ and are managed by bot-master, a process manager with a systemd-managed daemon and a Textual TUI for monitoring. Migrated from an AWS EC2 bot-farm to Hetzner.

Bots

• au-tomator — au-tomator-telegram-bot (Python 3.12, uv)
• zapier — zapier-telegram-bot (Python 3.12, uv)
• writing-assistant — telegram-writing-assistant (~/git/telegram-writing-assistant, Python 3.12, uv)

Each bot has:

• .env with bot token and chat ID
• .gitignore with .env
• pyproject.toml pinning Python 3.12, python-telegram-bot==13.11, urllib3<2, setuptools<81

bot-master

A process manager for Telegram bots with auto-restart, persistent logging, and a TUI. Lives in ~/bots/bot-master.

Install (prod)

uvx bot-master install

Interactive wizard: asks where to store config and logs (default ~/bots/bot-master), creates a bots.yaml template, and generates a systemd service file.

Install (dev, uses local source)

cd ~/bots/bot-master
uv sync
./install.sh          # installs systemd service pointing to local .venv
uv run bot-master     # connect TUI

Configuration

Edit ~/bots/bot-master/bots.yaml:

bots:
  au-tomator:
    directory: /home/alexey/bots/au-tomator-telegram-bot
    command: uv run python main.py

  zapier:
    directory: /home/alexey/bots/zapier-telegram-bot
    command: uv run python main.py

  writing-assistant:
    directory: /home/alexey/git/telegram-writing-assistant
    command: uv run python main.py

The daemon automatically loads environment variables from .envrc and .env files in each bot's directory, so export TELEGRAM_TOKEN=... style files work out of the box.

systemd Service

Install the service file:

sudo cp ~/bots/bot-master/bot-master.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now bot-master

Manage:

sudo systemctl status bot-master
sudo systemctl restart bot-master

TUI Keybindings

Key	Action
s / x / r	Start / stop / restart selected bot
a / z	Start all / stop all
arrows / j/k	Navigate
tab	Switch focus between bot list and logs
q	Quit TUI (daemon keeps running)

Logs

Logs are stored in ~/bots/bot-master/logs/ (one file per bot, 10 MB rotation with 5 backups). The daemon also keeps the last 5000 lines per bot in memory for fast streaming.

Dependency fixes

• imghdr removed in Python 3.13 — pinned to Python 3.12 via uv python pin 3.12
• urllib3 v2 removed contrib.appengine — pinned urllib3<2
• setuptools 82+ removed pkg_resources — pinned setuptools<81

Crontab

Zapier bot scheduled messages (newsletters, slack invites, trello, slack dump), plus rds-export pipelines. Check with crontab -l.