pmlogger / pcp cheatsheet

pmlogger_pcp.md

Performance Co-Pilot (PCP) + pmlogger Cheatsheet

(Updated 2025–2026 – most frequently used commands, configs & patterns)

1. Service Management (systemd)

systemctl status pmcd pmlogger pmproxy # core trio systemctl restart pmcd pmlogger pmproxy systemctl enable --now pmcd pmlogger pmproxy

journalctl -u pmcd -u pmlogger -u pmproxy -xe # quick error check journalctl -u pmlogger -f # tail logging activity

2. Quick "Is PCP alive?" Tests

pmprobe -v kernel.all.load # → should show 1 instance pmprobe -v mem.physmem kernel.all.cpu.nice pmprobe -v disk.dev.total # disk metrics present?

pminfo -f kernel.all.load # current value(s) pminfo -f -n PMNS_DEFAULT kernel.all.pressure.* # pressure metrics (cgroup v2)

pmnsdump # dump full namespace (huge!)

3. pmlogger – Control & Status

pmlc # interactive → show config status pmlc <<EOF show status show mandatory on default EOF

Where archives usually live

ls -lh /var/lib/pcp/pmlogger/$(hostname -s)/

Force rotation / compression now (debug / small test)

pmlogger_daily -X -r -v

Manual foreground run (great for testing new config)

pmlogger -r -T 5s -c /etc/pcp/pmlogger/config.d/custom
-l /tmp/pmlogger.test.log testarchive

4. Common pmlogger Config Snippets

Put in /etc/pcp/pmlogger/config.d/99-custom or similar

log mandatory on default { # ─────────────────────────────────────── # Always include – system health baseline # ─────────────────────────────────────── mem.physmem mem.util.used [ every 5 sec ] mem.util.free mem.util.cached kernel.all.load kernel.all.cpu.{user,system,wait,idle,steal} kernel.all.pressure.{cpu,mem,io}.{some,full}.avg10

# ───────────────────────────────────────
# Disk – choose what matches your system
# ───────────────────────────────────────
disk.dev.read           [ every 5 sec ]
disk.dev.write
disk.dev.total
disk.dev.avactive
disk.dev.await

# ───────────────────────────────────────
# Network – common & useful
# ───────────────────────────────────────
network.interface.in.bytes      [ every 10 sec ]
network.interface.out.bytes
network.interface.total.bytes   [ every 10 sec ]
network.tcpconn.established

# ───────────────────────────────────────
# Container / cgroup / proc (if pmda-cgroup / pmda-proc active)
# ───────────────────────────────────────
proc.hog.cpu
cgroup.cpuacct.usage
cgroup.memory.usage
cgroup.pressure.memory.some.avg10

}

[optional] log advisory on { kernel.all.load > 25 mem.util.used > 0.95 * mem.physmem kernel.all.pressure.cpu.some.avg10 > 0.4 }

5. Reading Archives – pmlogsummary & friends

One day summary

pmlogsummary -K /var/lib/pcp/pmlogger//202603.0.*

Specific metrics across many days

pmlogsummary -K -l mem.util.used kernel.all.load /var/lib/pcp/pmlogger//20260

Highest load average periods last week

pmlogsummary -K -l -5 kernel.all.load /var/lib/pcp/pmlogger//202603

When was swap used > 1 GiB?

pmlogsummary -K -l mem.util.swapused /var/lib/pcp/pmlogger//2026
| awk '$NF > 1024'

6. pmrep – Modern sar/vmstat/iostat replacement

pmrep :vmstat -t 5s pmrep :iostat -t 5s pmrep :cpu -t 5s

Memory + pressure focused

pmrep -t 5s mem.util.{used,free,cached,swapused}
kernel.all.pressure.mem.{some,full}.avg10

Fast network interfaces

pmrep -t 10s -i 'eth.|en.|ens.|bond.'
network.interface.total.bytes

Last 30 minutes of load

pmrep -t 60s -T '@30 minutes ago' kernel.all.load

7. pmproxy + Grafana / HTTP API

systemctl status pmproxy

Quick metric fetch via REST

curl 'http://localhost:44322/metrics?names=kernel.all.load,mem.util.used'

Grafana datasource settings

Type: Prometheus (yes, really – pmproxy speaks Prometheus format) URL: http://localhost:44322 Access: Server (Proxy)

8. Useful One-liners & Patterns

Load > 10 in last 3 days

pmlogsummary -K -l kernel.all.load /var/lib/pcp/pmlogger//202603
| grep -E '[1-9][0-9].'

Compare two hosts side-by-side

pmrep -h server1 -h server2 :cpu -t 10s

List all active PMDAs

pmda -l

Which metrics changed most last hour (interesting!)

pmdiff -t 60s -S @1hour -T @now
/var/lib/pcp/pmlogger/$(hostname)/202603*.0.*
| sort -k3 -nr | head -20

Find archives that grew > 100 MB today

find /var/lib/pcp/pmlogger/ -name '.0.' -mtime -1 -size +100M -ls

9. Troubleshooting Checklist

[ ] systemctl is-active pmcd pmlogger pmproxy [ ] pmprobe -v kernel.all.load → 1 instance? [ ] pminfo -f kernel.all.load → values appear? [ ] ls -l /var/lib/pcp/pmlogger/*/ → archives growing? [ ] pmlc show status → logging mandatory? [ ] journalctl -u pmlogger -u pmcd → errors / permission issues? [ ] pmproxy listening? ss -ltn | grep 44322 [ ] Enough disk space? df -h /var/lib/pcp