Skip to content

Instantly share code, notes, and snippets.

@RichardSlater

RichardSlater/LGTM.md

Created April 30, 2026 11:44
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save RichardSlater/c65bdb7cf00d8eb0ea818ded493eed77 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save RichardSlater/c65bdb7cf00d8eb0ea818ded493eed77 to your computer and use it in GitHub Desktop.
Download ZIP
NixOS Windows Subsystem for Linux 25.11 Loki-Grafana-Tempo-Mirmir (LGTM) Stack
Raw
configuration.nix
{
config,
lib,
pkgs,
...
}:
{
imports = [
# include NixOS-WSL modules
<nixos-wsl/modules>
(fetchTarball "https://github.com/nix-community/nixos-vscode-server/tarball/master")
./lgtm.nix
];
nixpkgs.config.allowUnfree = true;
environment.systemPackages = with pkgs; [
neovim
nixfmt
direnv
];
users.users.rslater = {
isNormalUser = true;
packages = with pkgs; [
];
extraGroups = [
"wheel"
"docker"
];
};
wsl.enable = true;
wsl.defaultUser = "nixos";
services.openssh = {
enable = true;
settings = {
PasswordAuthentication = false;
AllowUsers = [ "rslater" ];
UseDns = true;
X11Forwarding = false;
PermitRootLogin = "no";
};
};
services.vscode-server.enable = true;
services.lgtm.enable = true;
security.sudo = {
enable = true;
wheelNeedsPassword = false;
};
# This value determines the NixOS release from which the default
# settings for stateful data, like file locations and database versions
# on your system were taken. It's perfectly fine and recommended to leave
# this value at the release version of the first install of this system.
# Before changing this value read the documentation for this option
# (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
system.stateVersion = "25.11"; # Did you read the comment?
}
Raw
LGTM.md

LGTM Observability Stack

This document describes the local LGTM stack configured for this NixOS machine. The stack is managed declaratively by NixOS, runs as Podman containers under systemd, stores state on the host, and exposes service ports on localhost only.

LGTM means:

  • Loki for logs.
  • Grafana for dashboards, exploration, and datasource access.
  • Tempo for traces.
  • Mimir for metrics through the Prometheus-compatible API.

The implementation lives in ../lgtm.nix. It is imported and enabled from ../configuration.nix.

Current Design

The stack is intended for local development and workstation observability, not production multi-node observability. It uses filesystem-backed storage, single-node service modes, no external object storage, and localhost-only port bindings.

NixOS manages the lifecycle through these systemd units:

Unit Purpose
podman-lgtm-network.service Creates the dedicated Podman bridge network.
podman-lgtm-grafana.service Runs Grafana.
podman-lgtm-loki.service Runs Loki.
podman-lgtm-tempo.service Runs Tempo.
podman-lgtm-mimir.service Runs Mimir.

Podman runs the containers on a dedicated network named lgtm. Grafana connects to the backend services by container DNS names on that network:

Grafana Datasource Internal URL
Loki http://lgtm-loki:3100
Tempo http://lgtm-tempo:3200
Mimir http://lgtm-mimir:9009/prometheus

Components

Grafana

Grafana is the UI for exploring metrics, logs, and traces. It is configured by the lgtm-grafana container and persists its database/plugins under /var/lib/lgtm/grafana.

The module provisions datasources from:

/etc/lgtm/grafana/provisioning/datasources/datasources.yml

The datasource file is bind-mounted directly into the Grafana container. This is intentional: NixOS materializes environment.etc entries through /etc/static symlinks, and bind-mounting the whole provisioning directory can leave symlink targets unavailable inside the container.

Loki

Loki stores logs using local filesystem storage under /var/lib/lgtm/loki. The current configuration disables authentication, uses an in-memory ring, enables TSDB schema v13, and stores chunks/rules below /loki inside the container.

Loki is ready when this endpoint returns ready:

curl -fsS http://127.0.0.1:3100/ready

Tempo

Tempo stores traces on the local filesystem under /var/lib/lgtm/tempo. It enables OTLP receivers for applications and tools that can export traces using OpenTelemetry.

Published local ports:

Port Protocol Purpose
3200 HTTP Tempo API and readiness endpoint.
4317 OTLP gRPC Trace ingestion.
4318 OTLP HTTP Trace ingestion.

Tempo is ready when this endpoint returns ready:

curl -fsS http://127.0.0.1:3200/ready

Mimir

Mimir stores metrics using filesystem-backed storage under /var/lib/lgtm/mimir. It runs in single-process target: all mode and exposes Prometheus-compatible APIs through /prometheus.

Grafana treats Mimir as a Prometheus datasource at:

http://lgtm-mimir:9009/prometheus

Mimir is ready when this endpoint returns ready:

curl -fsS http://127.0.0.1:9009/ready

Container Images

The module pins explicit image tags for repeatability:

Component Image
Grafana grafana/grafana-oss:11.2.0
Loki grafana/loki:3.2.0
Tempo grafana/tempo:2.6.1
Mimir grafana/mimir:2.14.0

Update these through services.lgtm.images in the NixOS configuration, then rebuild and switch the system.

Access

All published ports bind to 127.0.0.1. They are accessible from this NixOS environment and, depending on WSL networking behavior, may also be reachable from the Windows host through localhost forwarding.

Service URL Notes
Grafana http://127.0.0.1:3000 Main UI.
Grafana health http://127.0.0.1:3000/api/health Returns database and version status.
Loki readiness http://127.0.0.1:3100/ready Returns ready after startup.
Tempo readiness http://127.0.0.1:3200/ready Returns ready after startup.
Mimir readiness http://127.0.0.1:9009/ready Returns ready after startup.
Tempo OTLP gRPC 127.0.0.1:4317 Use for OpenTelemetry gRPC exporters.
Tempo OTLP HTTP http://127.0.0.1:4318 Use for OpenTelemetry HTTP exporters.

Grafana sign-up is disabled, anonymous auth is disabled, and the module does not store real credentials. On a fresh Grafana data directory, use Grafana's default first-login behavior, then change the password immediately.

Configuration

Enable the stack from ../configuration.nix:

imports = [
  ./lgtm.nix
];

services.lgtm.enable = true;

The module exposes these options:

Option Default Purpose
services.lgtm.enable false Enables the stack.
services.lgtm.dataDir /var/lib/lgtm Host directory for persistent data.
services.lgtm.listenAddress 127.0.0.1 Address used for published container ports.
services.lgtm.networkName lgtm Podman bridge network name.
services.lgtm.images.grafana grafana/grafana-oss:11.2.0 Grafana image.
services.lgtm.images.loki grafana/loki:3.2.0 Loki image.
services.lgtm.images.tempo grafana/tempo:2.6.1 Tempo image.
services.lgtm.images.mimir grafana/mimir:2.14.0 Mimir image.
services.lgtm.ports.grafana 3000 Host Grafana port.
services.lgtm.ports.loki 3100 Host Loki port.
services.lgtm.ports.tempoHttp 3200 Host Tempo HTTP port.
services.lgtm.ports.tempoOtlpGrpc 4317 Host OTLP gRPC port.
services.lgtm.ports.tempoOtlpHttp 4318 Host OTLP HTTP port.
services.lgtm.ports.mimir 9009 Host Mimir HTTP port.

Example override:

services.lgtm = {
  enable = true;
  dataDir = /var/lib/lgtm;
  listenAddress = "127.0.0.1";
  ports.grafana = 3000;
  images.grafana = "grafana/grafana-oss:11.2.0";
};

After changing configuration, format and rebuild:

nixfmt configuration.nix lgtm.nix
sudo nixos-rebuild switch -I nixos-config=/home/rslater/nixos/configuration.nix

Use test instead of switch when validating a change temporarily:

sudo nixos-rebuild test -I nixos-config=/home/rslater/nixos/configuration.nix

Persistence

Persistent data is stored below /var/lib/lgtm:

Path Owner Purpose
/var/lib/lgtm/grafana UID/GID 472:472 Grafana database, plugins, and local state.
/var/lib/lgtm/loki UID/GID 10001:10001 Loki chunks, indexes, WAL, and rules.
/var/lib/lgtm/tempo UID/GID 10001:10001 Tempo WAL and local trace blocks.
/var/lib/lgtm/mimir UID/GID 10001:10001 Mimir TSDB, blocks, rules, alertmanager, and activity data.

These directories are created by systemd.tmpfiles.rules during activation.

Configuration files are generated below /etc/lgtm:

/etc/lgtm/grafana/provisioning/datasources/datasources.yml
/etc/lgtm/loki/loki.yml
/etc/lgtm/tempo/tempo.yml
/etc/lgtm/mimir/mimir.yml

Do not edit these generated files directly. Change ../lgtm.nix, then rebuild.

Operations

Check Service State

systemctl --no-pager --plain --full status \
  podman-lgtm-network \
  podman-lgtm-grafana \
  podman-lgtm-loki \
  podman-lgtm-tempo \
  podman-lgtm-mimir

For a terse active-state check:

systemctl is-active \
  podman-lgtm-grafana \
  podman-lgtm-loki \
  podman-lgtm-tempo \
  podman-lgtm-mimir \
  podman-lgtm-network

Check Containers

sudo podman ps --filter name=lgtm --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'

Inspect the network:

sudo podman network inspect lgtm

Health Checks

curl -fsS http://127.0.0.1:3000/api/health
curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3200/ready
curl -fsS http://127.0.0.1:9009/ready

View Logs

Use systemd for service-level logs:

journalctl --no-pager -u podman-lgtm-grafana -n 100
journalctl --no-pager -u podman-lgtm-loki -n 100
journalctl --no-pager -u podman-lgtm-tempo -n 100
journalctl --no-pager -u podman-lgtm-mimir -n 100

Use Podman for container logs:

sudo podman logs --tail 100 lgtm-grafana
sudo podman logs --tail 100 lgtm-loki
sudo podman logs --tail 100 lgtm-tempo
sudo podman logs --tail 100 lgtm-mimir

Note that the containers are started with --rm; if a container exits, Podman may remove it. In that case, use journalctl for the unit because the container logs may no longer be available.

Restart Services

Restart the whole stack:

sudo systemctl restart \
  podman-lgtm-loki \
  podman-lgtm-tempo \
  podman-lgtm-mimir \
  podman-lgtm-grafana

Restart only Grafana after changing datasource provisioning:

sudo systemctl restart podman-lgtm-grafana

Stop Services

sudo systemctl stop \
  podman-lgtm-grafana \
  podman-lgtm-loki \
  podman-lgtm-tempo \
  podman-lgtm-mimir

The persistent data in /var/lib/lgtm is not removed when services stop.

Sending Data To The Stack

The stack provisions storage and query backends, but it does not currently install a host collector such as Grafana Alloy. Applications can still send data directly to the local endpoints.

Traces

Configure OpenTelemetry exporters to send traces to Tempo:

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

For gRPC exporters, use:

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Logs

Loki is reachable at:

http://127.0.0.1:3100

Push clients should use Loki's push API path:

http://127.0.0.1:3100/loki/api/v1/push

Metrics

Mimir exposes Prometheus-compatible APIs at:

http://127.0.0.1:9009/prometheus

Remote write clients should target:

http://127.0.0.1:9009/api/v1/push

or the relevant Mimir/Prometheus remote write endpoint expected by the client version in use. Validate client configuration against the Mimir documentation before wiring production-like workloads.

Maintenance

Updating Images

  1. Change the relevant services.lgtm.images.* value in ../configuration.nix or the defaults in ../lgtm.nix.

  2. Review the upstream release notes for breaking changes and configuration migrations.

  3. Run:

    nixfmt configuration.nix lgtm.nix
sudo nixos-rebuild test -I nixos-config=/home/rslater/nixos/configuration.nix

  4. Verify health endpoints and Grafana datasources.

  5. Persist the change:

    sudo nixos-rebuild switch -I nixos-config=/home/rslater/nixos/configuration.nix

Backing Up Data

Stop the stack or accept an application-consistent backup risk, then archive /var/lib/lgtm:

sudo tar -C /var/lib -czf lgtm-backup.tgz lgtm

Restore by stopping services, replacing /var/lib/lgtm, preserving ownerships, and restarting services.

Clearing Local Data

This deletes stored dashboards, logs, traces, and metrics. Use carefully.

sudo systemctl stop \
  podman-lgtm-grafana \
  podman-lgtm-loki \
  podman-lgtm-tempo \
  podman-lgtm-mimir

sudo rm -rf /var/lib/lgtm
sudo nixos-rebuild switch -I nixos-config=/home/rslater/nixos/configuration.nix

The rebuild recreates the data directories with the expected ownerships.

Checking Disk Usage

sudo du -h -d 2 /var/lib/lgtm | sort -h
sudo podman system df

Mimir has compactor_blocks_retention_period: 7d in the current config. Tempo has block_retention: 48h. Loki retention is not explicitly configured in the current module, so Loki data can grow until manually cleaned or retention is added.

Podman Image Cleanup

After image upgrades, remove unused images and layers:

sudo podman image prune
sudo podman system prune

Review what Podman plans to delete before confirming.

Troubleshooting

Grafana Is Not Reachable

Check the unit and logs:

systemctl status podman-lgtm-grafana
journalctl --no-pager -u podman-lgtm-grafana -n 150

Confirm the datasource file is visible inside the container:

sudo podman exec lgtm-grafana test -f /etc/grafana/provisioning/datasources/datasources.yml

If Grafana fails with a datasource provisioning error, verify that ../lgtm.nix bind-mounts the datasource file directly rather than bind-mounting the entire /etc/lgtm/grafana/provisioning directory.

Readiness Endpoints Return 503

Loki, Tempo, and Mimir can briefly return 503 during startup while their rings and internal modules settle. Wait a few seconds and retry:

curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3200/ready
curl -fsS http://127.0.0.1:9009/ready

If readiness does not recover, check the service logs and filesystem permissions under /var/lib/lgtm.

Container Name Resolution Fails

All services must be attached to the lgtm Podman network. Check:

sudo podman network inspect lgtm
sudo podman inspect lgtm-grafana --format '{{json .NetworkSettings.Networks}}'

Restart the network unit and containers if needed:

sudo systemctl restart podman-lgtm-network
sudo systemctl restart podman-lgtm-loki podman-lgtm-tempo podman-lgtm-mimir podman-lgtm-grafana

Port Already In Use

Check listeners:

sudo ss -ltnp | grep -E ':3000|:3100|:3200|:4317|:4318|:9009'

Change the relevant services.lgtm.ports.* option and rebuild.

Security Notes

  • Ports bind to 127.0.0.1 by default. Do not change services.lgtm.listenAddress to 0.0.0.0 unless you also add authentication, firewall rules, and a clear access model.
  • The stack disables anonymous Grafana auth and sign-up, but it does not manage Grafana secrets.
  • The Loki, Tempo, and Mimir single-node configs are local-development oriented and are not hardened for shared or production use.
  • Avoid storing personal, health, payment, or client-sensitive data in this local stack unless the relevant compliance requirements and retention controls are defined.
  • Filesystem-backed Mimir is explicitly suitable for development/testing style deployments, not production-scale observability.

References

The module and this document are based on the behavior of the running NixOS configuration plus upstream documentation for the individual tools:

Raw
lgtm.nix
{
config,
lib,
...
}:
let
cfg = config.services.lgtm;
containerNames = [
"lgtm-grafana"
"lgtm-loki"
"lgtm-tempo"
"lgtm-mimir"
];
containerServiceNames = map (name: "podman-${name}") containerNames;
in
{
options.services.lgtm = {
enable = lib.mkEnableOption "a local Grafana LGTM stack managed by Podman";
dataDir = lib.mkOption {
type = lib.types.path;
default = /var/lib/lgtm;
description = "Persistent data directory for the local LGTM stack.";
};
listenAddress = lib.mkOption {
type = lib.types.str;
default = "127.0.0.1";
description = "Address used for published container ports.";
};
networkName = lib.mkOption {
type = lib.types.str;
default = "lgtm";
description = "Podman network used by the LGTM containers.";
};
images = {
grafana = lib.mkOption {
type = lib.types.str;
default = "grafana/grafana-oss:11.2.0";
description = "Grafana container image.";
};
loki = lib.mkOption {
type = lib.types.str;
default = "grafana/loki:3.2.0";
description = "Loki container image.";
};
tempo = lib.mkOption {
type = lib.types.str;
default = "grafana/tempo:2.6.1";
description = "Tempo container image.";
};
mimir = lib.mkOption {
type = lib.types.str;
default = "grafana/mimir:2.14.0";
description = "Mimir container image.";
};
};
ports = {
grafana = lib.mkOption {
type = lib.types.port;
default = 3000;
description = "Grafana HTTP port.";
};
loki = lib.mkOption {
type = lib.types.port;
default = 3100;
description = "Loki HTTP port.";
};
tempoHttp = lib.mkOption {
type = lib.types.port;
default = 3200;
description = "Tempo HTTP port.";
};
tempoOtlpGrpc = lib.mkOption {
type = lib.types.port;
default = 4317;
description = "Tempo OTLP gRPC port.";
};
tempoOtlpHttp = lib.mkOption {
type = lib.types.port;
default = 4318;
description = "Tempo OTLP HTTP port.";
};
mimir = lib.mkOption {
type = lib.types.port;
default = 9009;
description = "Mimir HTTP port.";
};
};
};
config = lib.mkIf cfg.enable {
virtualisation.oci-containers = {
backend = "podman";
containers = {
lgtm-grafana = {
image = cfg.images.grafana;
dependsOn = [
"lgtm-loki"
"lgtm-tempo"
"lgtm-mimir"
];
environment = {
GF_AUTH_ANONYMOUS_ENABLED = "false";
GF_PATHS_PROVISIONING = "/etc/grafana/provisioning";
GF_SERVER_HTTP_ADDR = "0.0.0.0";
GF_USERS_ALLOW_SIGN_UP = "false";
};
ports = [ "${cfg.listenAddress}:${toString cfg.ports.grafana}:3000" ];
volumes = [
"${toString cfg.dataDir}/grafana:/var/lib/grafana"
"/etc/lgtm/grafana/provisioning/datasources/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml:ro"
];
networks = [ cfg.networkName ];
};
lgtm-loki = {
image = cfg.images.loki;
cmd = [ "-config.file=/etc/loki/loki.yml" ];
ports = [ "${cfg.listenAddress}:${toString cfg.ports.loki}:3100" ];
volumes = [
"${toString cfg.dataDir}/loki:/loki"
"/etc/lgtm/loki/loki.yml:/etc/loki/loki.yml:ro"
];
networks = [ cfg.networkName ];
};
lgtm-tempo = {
image = cfg.images.tempo;
cmd = [ "-config.file=/etc/tempo/tempo.yml" ];
ports = [
"${cfg.listenAddress}:${toString cfg.ports.tempoHttp}:3200"
"${cfg.listenAddress}:${toString cfg.ports.tempoOtlpGrpc}:4317"
"${cfg.listenAddress}:${toString cfg.ports.tempoOtlpHttp}:4318"
];
volumes = [
"${toString cfg.dataDir}/tempo:/var/tempo"
"/etc/lgtm/tempo/tempo.yml:/etc/tempo/tempo.yml:ro"
];
networks = [ cfg.networkName ];
};
lgtm-mimir = {
image = cfg.images.mimir;
cmd = [ "-config.file=/etc/mimir/mimir.yml" ];
ports = [ "${cfg.listenAddress}:${toString cfg.ports.mimir}:9009" ];
volumes = [
"${toString cfg.dataDir}/mimir:/data/mimir"
"/etc/lgtm/mimir/mimir.yml:/etc/mimir/mimir.yml:ro"
];
networks = [ cfg.networkName ];
};
};
};
systemd.tmpfiles.rules = [
"d ${toString cfg.dataDir} 0755 root root - -"
"d ${toString cfg.dataDir}/grafana 0750 472 472 - -"
"d ${toString cfg.dataDir}/loki 0750 10001 10001 - -"
"d ${toString cfg.dataDir}/tempo 0750 10001 10001 - -"
"d ${toString cfg.dataDir}/mimir 0750 10001 10001 - -"
];
environment.etc = {
"lgtm/grafana/provisioning/datasources/datasources.yml".text = ''
apiVersion: 1
datasources:
- name: Loki
uid: loki
type: loki
access: proxy
url: http://lgtm-loki:3100
editable: true
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://lgtm-tempo:3200
editable: true
jsonData:
tracesToLogsV2:
datasourceUid: loki
serviceMap:
datasourceUid: mimir
- name: Mimir
uid: mimir
type: prometheus
access: proxy
url: http://lgtm-mimir:9009/prometheus
isDefault: true
editable: true
'';
"lgtm/loki/loki.yml".text = ''
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
replication_factor: 1
ring:
kvstore:
store: inmemory
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
schema_config:
configs:
- from: 2024-04-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
allow_structured_metadata: true
volume_enabled: true
analytics:
reporting_enabled: false
'';
"lgtm/tempo/tempo.yml".text = ''
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
ingester:
trace_idle_period: 10s
max_block_bytes: 1000000
max_block_duration: 5m
compactor:
compaction:
block_retention: 48h
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks
'';
"lgtm/mimir/mimir.yml".text = ''
target: all
multitenancy_enabled: false
server:
http_listen_port: 9009
grpc_listen_port: 9095
activity_tracker:
filepath: /data/mimir/activity.log
common:
storage:
backend: filesystem
filesystem:
dir: /data/mimir/common
blocks_storage:
backend: filesystem
filesystem:
dir: /data/mimir/blocks
tsdb:
dir: /data/mimir/tsdb
compactor:
data_dir: /data/mimir/compactor
ingester:
ring:
replication_factor: 1
ruler:
rule_path: /data/mimir/ruler
ruler_storage:
backend: filesystem
filesystem:
dir: /data/mimir/rules
alertmanager:
data_dir: /data/mimir/alertmanager
alertmanager_storage:
backend: filesystem
filesystem:
dir: /data/mimir/alertmanager-storage
store_gateway:
sharding_ring:
replication_factor: 1
limits:
compactor_blocks_retention_period: 7d
usage_stats:
enabled: false
'';
};
systemd.services = lib.mkMerge [
{
podman-lgtm-network = {
description = "Create the Podman network for the LGTM stack";
wantedBy = [ "multi-user.target" ];
wants = [ "network-online.target" ];
after = [ "network-online.target" ];
path = [ config.virtualisation.podman.package ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
};
script = ''
podman network exists ${lib.escapeShellArg cfg.networkName} || podman network create ${lib.escapeShellArg cfg.networkName}
'';
};
}
(lib.genAttrs containerServiceNames (_: {
requires = [ "podman-lgtm-network.service" ];
after = [ "podman-lgtm-network.service" ];
}))
];
};
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment