Securing Linux Secrets

howto-dotfile-encryption-keyring.md

Encrypting Dotfile Secrets with the Linux Kernel Key Retention Service

The Linux kernel key retention service (keyrings) lets you store secrets in kernel memory — never on disk, automatically discarded when your session ends. This makes it a solid backing store for the encryption key used to protect a dotfile like .app.conf.

How it works

The kernel exposes a hierarchy of keyrings tied to different lifetimes:

Keyring	Lifetime	Typical use
@u (user)	Until user logs out	Persistent within a login session
@s (session)	Until session ends	Shell sessions, SSH connections
@p (process)	Until process exits	Short-lived, per-process secrets

You interact with it via keyctl (from the keyutils package).

Prerequisites

# Debian/Ubuntu
sudo apt install keyutils openssl

# Fedora/RHEL
sudo dnf install keyutils openssl

Step 1 — Generate a strong encryption key and store it in the keyring

# Generate a 256-bit random key (base64-encoded so keyctl can accept it as a string)
APP_KEY=$(openssl rand -base64 32)

# Store it in the user keyring under the name "app.conf.key"
KEY_ID=$(keyctl add user app.conf.key "$APP_KEY" @u)

echo "Key ID: $KEY_ID"

The key now lives in kernel memory. You can verify it's there:

keyctl list @u
keyctl describe $KEY_ID

Step 2 — Encrypt your dotfile

# Write your plaintext secrets to a temp file (or pipe them directly)
cat > /tmp/app.conf.plain <<'EOF'
api_key=supersecret123
db_password=hunter2
token=eyJhbGciOi...
EOF

# Retrieve the key from the keyring and encrypt with AES-256-GCM
keyctl print $KEY_ID \
  | openssl enc -aes-256-cbc -pbkdf2 -iter 600000 \
      -pass stdin \
      -in  /tmp/app.conf.plain \
      -out ~/.app.conf.enc

# Wipe the plaintext
shred -u /tmp/app.conf.plain

Your secrets are now stored only in ~/.app.conf.enc. The plaintext is gone.

Step 3 — Decrypt on demand

# Retrieve the key from the keyring and decrypt in-memory (never touches disk)
keyctl print $KEY_ID \
  | openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 \
      -pass stdin \
      -in ~/.app.conf.enc

To source the decrypted values directly into your shell environment:

eval "$(keyctl print $KEY_ID \
  | openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 \
      -pass stdin \
      -in ~/.app.conf.enc \
  | sed 's/^/export /')"

Step 4 — Persist the key across logins (PAM integration)

The keyring is emptied on logout, so you need a way to re-load the key on login. Two options:

Option A — Prompt on first use (interactive)

Add a shell function to your .bashrc / .zshrc:

load_app_key() {
  if ! keyctl request user app.conf.key 2>/dev/null; then
    read -rs -p "app.conf passphrase: " APP_KEY
    echo
    keyctl add user app.conf.key "$APP_KEY" @u
  fi
}

Call load_app_key before any script that needs the secrets.

Option B — Store the key in a file, load via PAM

If the passphrase itself must survive reboots, store it in a locked-down file and load it via ~/.pam_environment or a systemd user service.

# Store the key string in a tightly-permissioned file
install -m 600 /dev/null ~/.app.conf.keyfile
echo -n "$APP_KEY" > ~/.app.conf.keyfile

# Load it at login via ~/.bashrc or a systemd user service
keyctl add user app.conf.key "$(cat ~/.app.conf.keyfile)" @u

Note: Option B shifts the trust boundary to ~/.app.conf.keyfile. Protect it with filesystem permissions and, ideally, full-disk encryption (LUKS).

Step 5 — Revoke the key when done

# Revoke immediately (key is invalidated in kernel memory)
keyctl revoke $KEY_ID

# Or unlink it from the keyring (removed when ref count hits 0)
keyctl unlink $KEY_ID @u

Optional: Hardware-backed key persistence (TPM 2.0 / YubiKey)

The passphrase approach in Step 4 leaves a secret in your head (or a file). Hardware tokens eliminate that — the private key material never leaves the chip. The workflow changes from:

passphrase → keyring → decrypt dotfile

to:

hardware token (PIN/touch) → unwrap key → keyring → decrypt dotfile

---

TPM 2.0 — Option A: Kernel trusted keys (simplest)

The kernel has a native trusted key type wired directly to the TPM. The kernel performs sealing and unsealing transparently; you never handle raw key material.

Prerequisites:

# Debian/Ubuntu
sudo apt install tpm2-tools

# Fedora/RHEL
sudo dnf install tpm2-tools

# Verify the kernel supports trusted keys
keyctl add trusted _test "new 32" @s && keyctl unlink $? @s \
  && echo "trusted keys available" || echo "not supported (check CONFIG_TRUSTED_KEYS)"

Create and seal a new key:

# Generate a 32-byte key sealed to the local TPM
KEY_ID=$(keyctl add trusted app.conf.key "new 32" @u)

# Export the sealed blob (safe to store on disk — the TPM is required to unseal it)
keyctl pipe $KEY_ID | xxd -p -c 0 > ~/.app.conf.tpmblob
chmod 600 ~/.app.conf.tpmblob

echo "Key ID: $KEY_ID — sealed blob saved to ~/.app.conf.tpmblob"

Reload the sealed blob at next login:

KEY_ID=$(keyctl add trusted app.conf.key \
  "load $(cat ~/.app.conf.tpmblob)" @u)

Put that keyctl add trusted line in your ~/.bashrc, a PAM exec, or a systemd user service — no passphrase needed. The TPM unseals it automatically as long as the firmware state hasn't changed.

PCR policy binding (optional): Bind the sealed key to specific PCR values so it can only be unsealed on the exact same machine with the same firmware/bootloader configuration:

# Create a policy that requires PCRs 0,1,2,3,7 to match
tpm2_createpolicy --policy-pcr -l sha256:0,1,2,3,7 -f policy.pcr

# When creating the trusted key, pass the policy file
KEY_ID=$(keyctl add trusted app.conf.key "new 32 policydigest=$(xxd -p -c 0 policy.pcr)" @u)

If the bootloader or firmware changes, the TPM refuses to unseal — useful for tamper detection, but you must re-seal after intentional firmware updates.

---

TPM 2.0 — Option B: Explicit sealing with tpm2-tools

Use this when you need more control — e.g. existing tpm2-tools scripting, persistent handles, or non-root access to a specific TPM object hierarchy.

Seal the key into a TPM object:

# Create a primary key under the Owner hierarchy (persists across reboots)
tpm2_createprimary -C o -c ~/.tpm/primary.ctx
mkdir -p ~/.tpm && chmod 700 ~/.tpm

# Generate the dotfile encryption key and seal it as a TPM data object
APP_KEY=$(openssl rand -base64 32)
echo -n "$APP_KEY" \
  | tpm2_create -C ~/.tpm/primary.ctx \
      -i - \
      -u ~/.tpm/app.seal.pub \
      -r ~/.tpm/app.seal.priv
chmod 600 ~/.tpm/app.seal.*

Unseal and load into keyring at login:

tpm2_load -C ~/.tpm/primary.ctx \
  -u ~/.tpm/app.seal.pub \
  -r ~/.tpm/app.seal.priv \
  -c ~/.tpm/app.seal.ctx

APP_KEY=$(tpm2_unseal -c ~/.tpm/app.seal.ctx)
keyctl add user app.conf.key "$APP_KEY" @u
unset APP_KEY

The sealed blobs (app.seal.pub, app.seal.priv) are bound to this specific TPM. Copying them to another machine produces unusable ciphertext.

---

YubiKey — Option A: PIV slot / PKCS#11 asymmetric wrapping

The YubiKey's PIV application holds an RSA or EC private key on-chip. You encrypt (wrap) the dotfile key with the YubiKey's public key; decryption requires the YubiKey to be present and the PIN to be entered.

Prerequisites:

# Debian/Ubuntu
sudo apt install yubico-piv-tool opensc libykcs11

# Fedora/RHEL
sudo dnf install yubico-piv-tool opensc ykpers

One-time setup — generate a key in PIV slot 9d (Key Management):

# Generate RSA-2048 key in slot 9d
yubico-piv-tool -a generate -s 9d -A RSA2048 -o ~/.yk-app.pub.pem

# Issue a self-signed certificate (required by PKCS#11 layer)
yubico-piv-tool -a verify-pin -a selfsign-certificate \
  -s 9d -S "/CN=dotfile-key/" \
  -i ~/.yk-app.pub.pem -o ~/.yk-app.cert.pem

# Import the certificate back into the YubiKey
yubico-piv-tool -a import-certificate -s 9d -i ~/.yk-app.cert.pem
chmod 600 ~/.yk-app.pub.pem

Wrap (encrypt) the dotfile key with the YubiKey's public key:

APP_KEY=$(openssl rand -base64 32)

# Encrypt APP_KEY with the PIV public key — output is safe to store on disk
echo -n "$APP_KEY" \
  | openssl pkeyutl -encrypt -pubin -inkey ~/.yk-app.pub.pem \
      -pkeyopt rsa_padding_mode:oaep \
      -pkeyopt rsa_oaep_md:sha256 \
  > ~/.app.conf.ykwrap
chmod 600 ~/.app.conf.ykwrap
unset APP_KEY

Unwrap and load into keyring (YubiKey touch + PIN required):

# Find the correct PKCS#11 module path for your distro
YKCS11=/usr/lib/x86_64-linux-gnu/libykcs11.so          # Debian/Ubuntu
# YKCS11=/usr/lib64/libykcs11.so                        # Fedora/RHEL

APP_KEY=$(pkcs11-tool --module "$YKCS11" \
  --decrypt \
  --slot-index 0 \
  --id 04 \
  --mechanism RSA-PKCS-OAEP \
  --hash-algorithm SHA256 \
  --input-file ~/.app.conf.ykwrap)

keyctl add user app.conf.key "$APP_KEY" @u
unset APP_KEY

The YubiKey performs the RSA private-key operation internally. The plaintext key appears only briefly in the shell variable before being handed to the kernel keyring.

Slot 9d vs 9a: Slot 9d (Key Management) is designed for encryption/decryption. Slot 9a (Authentication) is intended for signing. Use 9d here.

---

YubiKey — Option B: HMAC-SHA1 challenge-response (simpler)

Older YubiKey firmware and the YubiKey 5 series all support HMAC-SHA1 challenge-response in slot 2. This derives a deterministic encryption key from a per-machine challenge stored on disk, without any asymmetric crypto or certificate management.

Prerequisites:

# Debian/Ubuntu
sudo apt install yubikey-manager ykchalresp

# Fedora/RHEL
sudo dnf install yubikey-manager ykchalresp

One-time setup — program slot 2 for HMAC-SHA1:

# Generate a random HMAC secret and program it into YubiKey slot 2
ykman otp chalresp --generate 2
# (or use the GUI: ykman-gui → Applications → OTP → Configure → Challenge-response)

Generate and store a per-machine challenge:

# The challenge is public — it just ensures key uniqueness per machine
openssl rand -hex 20 > ~/.app.conf.challenge
chmod 600 ~/.app.conf.challenge

Derive the encryption key and load into keyring (YubiKey touch required):

# The YubiKey computes HMAC-SHA1(secret, challenge) — never exposes the secret
APP_KEY=$(ykchalresp -2 -x "$(cat ~/.app.conf.challenge)")

keyctl add user app.conf.key "$APP_KEY" @u
unset APP_KEY

Because the derived key is deterministic (same YubiKey + same challenge = same key), you do not need to wrap and store the key anywhere — just re-derive it on each login. This also means the ~/.app.conf.enc file is safe to back up anywhere.

Security note: HMAC-SHA1 challenge-response does not require the PIN, only physical touch. If you need PIN-gated access, use Option A (PIV) instead.

---

YubiKey — Option C: GPG smartcard (OpenPGP applet)

The YubiKey's OpenPGP applet exposes a standard GPG smartcard interface. GnuPG talks to it via scdaemon and pcscd. The encryption subkey lives on the card; GPG holds only a stub. This option integrates tightly with the GPG ecosystem and — uniquely — works transparently over a forwarded gpg-agent socket, meaning you can decrypt dotfile secrets on a remote server using the YubiKey plugged into your local workstation.

Prerequisites:

# Debian/Ubuntu
sudo apt install gnupg scdaemon pcscd gpg-agent

# Fedora/RHEL
sudo dnf install gnupg2 pcsc-lite opensc

# Verify the card is visible
gpg --card-status

Part 1 — One-time GPG key setup

Generate a key pair (off-card, so you can keep a backup):

# Use ECC (cv25519 encrypt + ed25519 sign) for modern hardware
gpg --expert --full-gen-key
# Choose: (11) ECC (set your own capabilities)
# Toggle: disable Sign → only Certify remains → confirm
# Choose: Curve 25519 → set expiry → fill in identity

This creates a certify-only master key. Add the subkeys you need:

gpg --expert --edit-key <fingerprint>

gpg> addkey
# (12) ECC (encrypt only) → Curve 25519 → set expiry
gpg> addkey
# (10) ECC (sign only) → ed25519 → set expiry

gpg> save

Back up the private key material before moving to the card:

gpg --export-secret-keys --armor <fingerprint> > ~/gpg-master-backup.asc
gpg --export-secret-subkeys --armor <fingerprint> > ~/gpg-subkeys-backup.asc
chmod 600 ~/gpg-master-backup.asc ~/gpg-subkeys-backup.asc
# Store these somewhere offline (e.g. encrypted USB)

Move subkeys onto the YubiKey:

gpg --edit-key <fingerprint>

# Select the encryption subkey (usually key index 1)
gpg> key 1
gpg> keytocard
# Prompt: (2) Encryption key → confirm

# If you added a signing subkey, move it too
gpg> key 1    # deselect
gpg> key 2
gpg> keytocard
# Prompt: (1) Signature key → confirm

gpg> save

After save, GPG replaces the local private key material with a card stub. gpg --card-status will show the key fingerprints linked to the card.

Set the card PIN and Admin PIN (defaults are 123456 / 12345678):

gpg --card-edit
gpg/card> admin
gpg/card> passwd
# Option 1: change PIN (user PIN, used for decrypt/sign)
# Option 3: change Admin PIN (used for card management)
gpg/card> quit

Part 2 — Wrap and unwrap the dotfile key

Encrypt (wrap) the dotfile key with your GPG encryption subkey:

APP_KEY=$(openssl rand -base64 32)

# --recipient targets the encryption subkey on the card
echo -n "$APP_KEY" \
  | gpg --encrypt --armor \
        --recipient <fingerprint> \
        --output ~/.app.conf.gpgwrap
chmod 600 ~/.app.conf.gpgwrap
unset APP_KEY

Decrypt and load into the kernel keyring (YubiKey touch + PIN required):

APP_KEY=$(gpg --decrypt --batch --quiet ~/.app.conf.gpgwrap)
keyctl add user app.conf.key "$APP_KEY" @u
unset APP_KEY

gpg-agent caches the PIN for the duration of the session (configurable via --default-cache-ttl). Subsequent decryptions in the same session require only touch, not the PIN.

Part 3 — Remote YubiKey over SSH via gpg-agent socket forwarding

GPG agent forwarding lets a remote host use the gpg-agent running on your local machine — and therefore the YubiKey inserted there. Only the encryption/decryption operations cross the socket; the private key never leaves the card.

[Remote server]                     [Local workstation]
  gpg --decrypt                       gpg-agent
       │                                   │
       └── S.gpg-agent (forwarded) ────────┘
                   SSH tunnel                YubiKey (USB)

Local setup

Enable the extra-socket — a restricted agent socket safe to forward (it cannot be used to export private keys or manage the keyring):

# ~/.gnupg/gpg-agent.conf
echo "extra-socket $(gpgconf --list-dirs agent-extra-socket)" \
  >> ~/.gnupg/gpg-agent.conf

gpgconf --reload gpg-agent
gpgconf --list-dirs agent-extra-socket    # note this path

Remote setup

On the remote host, tell GPG not to auto-start a local agent — it should use the forwarded socket instead:

# On the remote host
mkdir -p ~/.gnupg && chmod 700 ~/.gnupg

# Prevent gpg from silently launching a new agent when none is found
echo "no-autostart" >> ~/.gnupg/gpg.conf

# Import your GPG public key so the remote gpg knows the key metadata
# (public key only — no private key material ever touches the remote)
gpg --export --armor <fingerprint> | ssh remote 'gpg --import'

# Set ultimate trust on the remote (once)
ssh remote "echo '<fingerprint>:6:' | gpg --import-ownertrust"

Allow the SSH daemon on the remote to clean up stale forwarded sockets:

# /etc/ssh/sshd_config (remote)
StreamLocalBindUnlink yes

sudo systemctl reload sshd

SSH configuration

Add a RemoteForward directive to ~/.ssh/config on your local machine. The remote path must match where gpg on the remote looks for its agent socket:

# ~/.ssh/config  (local machine)
Host myremote
    HostName remote.example.com
    User alice

    # Forward local extra-socket → remote agent socket
    # Replace the remote path with: ssh myremote gpgconf --list-dirs agent-socket
    RemoteForward /run/user/1001/gnupg/S.gpg-agent /home/alice/.gnupg/S.gpg-agent.extra

Because UIDs differ between machines, use this one-liner to generate the correct RemoteForward line dynamically:

# Run from local — queries the remote for its socket path
remote_socket=$(ssh myremote gpgconf --list-dirs agent-socket)
local_extra=$(gpgconf --list-dirs agent-extra-socket)
echo "RemoteForward ${remote_socket} ${local_extra}"
# Paste the output into ~/.ssh/config

Or use a wrapper function that avoids hardcoded paths entirely:

# ~/.bashrc  (local machine)
ssh-gpg() {
  local host="$1"; shift
  local remote_sock local_extra
  remote_sock=$(ssh "$host" gpgconf --list-dirs agent-socket 2>/dev/null)
  local_extra=$(gpgconf --list-dirs agent-extra-socket)
  ssh -R "${remote_sock}:${local_extra}" "$host" "$@"
}

ssh-gpg myremote            # interactive session with forwarded agent
ssh-gpg myremote appconf load   # non-interactive: load key on remote

Verifying the forwarded agent

After connecting, confirm the remote GPG sees your YubiKey via the forwarded socket:

# On the remote, after connecting via ssh-gpg
gpg --card-status          # should show your YubiKey's card info
gpg -K                     # should show the key stub with '>'  (e.g.  ssb>)

The > marker on ssb> means the subkey is stored on a card (accessed via the forwarded agent).

Decrypt and load into the remote kernel keyring:

# On remote — YubiKey touch happens on the local machine
APP_KEY=$(gpg --decrypt --batch --quiet ~/.app.conf.gpgwrap)
keyctl add user app.conf.key "$APP_KEY" @u
unset APP_KEY

Troubleshooting forwarded agent

Symptom	Likely cause	Fix
gpg: error getting the agent socket	no-autostart not set; remote launched its own agent	gpgconf --kill gpg-agent on remote, then reconnect
Socket not forwarded	SSH connected before agent was running locally	Run gpgconf --launch gpg-agent locally, then reconnect
Card error or No card	scdaemon on remote intercepting the socket	Ensure scdaemon is not running on remote (gpgconf --kill scdaemon)
PIN prompt appears on remote	Agent not forwarded; remote fallback agent answered	Verify RemoteForward line and that the socket path is correct

---

Comparison

Method	Key material persists on disk	PIN required	Touch required	Works offline	Works over SSH	Hardware
Passphrase (Step 4A)	No (in head)	—	—	Yes	Yes	None
File (Step 4B)	Yes (plaintext)	—	—	Yes	Yes	None
TPM trusted key	Sealed blob	No	No	Yes	No	TPM 2.0
TPM tpm2-tools	Sealed blob	No	No	Yes	No	TPM 2.0
YubiKey PIV (PKCS#11)	Wrapped blob	Yes	Yes	Yes	No	YubiKey 4+
YubiKey HMAC challenge-response	Challenge only	No	Yes	Yes	No	YubiKey 2.2+
YubiKey GPG smartcard	Wrapped blob	Yes	Yes	Yes	Yes	YubiKey 4+

---

Helper script

Save this as ~/bin/appconf:

#!/usr/bin/env bash
set -euo pipefail

KEY_NAME="app.conf.key"
ENC_FILE="$HOME/.app.conf.enc"

_get_key_id() {
  keyctl request user "$KEY_NAME" 2>/dev/null
}

cmd_load() {
  if _get_key_id > /dev/null; then
    echo "Key already loaded." >&2; return
  fi
  read -rs -p "Passphrase: " pass; echo >&2
  keyctl add user "$KEY_NAME" "$pass" @u > /dev/null
  echo "Key loaded into session keyring." >&2
}

cmd_edit() {
  local kid; kid=$(_get_key_id) || { echo "Run: appconf load" >&2; exit 1; }
  local tmp; tmp=$(mktemp)
  trap "shred -u '$tmp'" EXIT
  keyctl print "$kid" \
    | openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 -pass stdin -in "$ENC_FILE" \
    > "$tmp"
  "${EDITOR:-vi}" "$tmp"
  keyctl print "$kid" \
    | openssl enc -aes-256-cbc -pbkdf2 -iter 600000 -pass stdin \
        -in "$tmp" -out "$ENC_FILE"
}

cmd_print() {
  local kid; kid=$(_get_key_id) || { echo "Run: appconf load" >&2; exit 1; }
  keyctl print "$kid" \
    | openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 -pass stdin -in "$ENC_FILE"
}

cmd_unload() {
  local kid; kid=$(_get_key_id) || { echo "Key not loaded." >&2; return; }
  keyctl unlink "$kid" @u
  echo "Key removed from keyring." >&2
}

case "${1:-}" in
  load)   cmd_load   ;;
  edit)   cmd_edit   ;;
  print)  cmd_print  ;;
  unload) cmd_unload ;;
  *)      echo "Usage: appconf {load|edit|print|unload}" >&2; exit 1 ;;
esac

chmod 700 ~/bin/appconf

Usage:

appconf load          # prompt for passphrase, inject into keyring
appconf print         # dump decrypted config to stdout
appconf edit          # open decrypted config in $EDITOR, re-encrypt on save
appconf unload        # remove key from keyring

Security notes

• The kernel keyring is volatile. Keys vanish on logout, hibernation, or reboot — by design.
• Root can read your keys. keyctl does not protect against root compromise. For that, use a hardware token (TPM, YubiKey) as the key source — see the optional hardware section above.
• Use PBKDF2 / high iteration counts. The -iter 600000 flag in the openssl commands above makes brute-force attacks on the ciphertext expensive.
• Prefer named keys over IDs in scripts. Key IDs change between sessions; the name (app.conf.key) is stable.
• Audit key access with audit subsystem. auditctl -a always,exit -F arch=b64 -S keyctl logs every keyctl syscall.

Quick reference

keyctl list @u                           # list all keys in user keyring
keyctl describe <id>                     # show key metadata
keyctl print <id>                        # print key value (careful with terminals)
keyctl add user <name> <val> @u          # add a string key
keyctl add user <name> <val> @s          # add to session keyring instead
keyctl add trusted <name> "new 32" @u   # create a new TPM-sealed 32-byte key
keyctl add trusted <name> "load <hex>" @u  # reload a sealed blob from disk
keyctl pipe <id> | xxd -p -c 0          # export sealed blob as hex
keyctl unlink <id> @u                    # remove from keyring
keyctl timeout <id> <seconds>            # auto-expire after N seconds
keyctl revoke <id>                       # immediately invalidate
keyctl show                              # display the full keyring tree