[blockprint] TopoLVM v15.5.5 Debug Output (lvmdEmbedded=false) v15.0.0

topolvm-debug-lvmdEmbedded-false.txt

[OK] kubectl is available

=== Kubernetes Pod Status ===
Checking pods in namespace: topolvm-system

Pod: topolvm-controller-6fc6bcd4b5-kjrlh
Status: Pending | Containers: 5 | Ready: false/false/false/false/false
[WARN] Pod is in Pending state
Recent events:
LAST SEEN   TYPE      REASON        OBJECT                                    MESSAGE
30s         Warning   FailedMount   pod/topolvm-controller-6fc6bcd4b5-kjrlh   MountVolume.SetUp failed for volume "certs" : secret "topolvm-mutatingwebhook" not found

Pod: topolvm-controller-6fc6bcd4b5-xvhmk
Status: Pending | Containers: 5 | Ready: false/false/false/false/false
[WARN] Pod is in Pending state
Recent events:
LAST SEEN   TYPE      REASON        OBJECT                                    MESSAGE
29s         Warning   FailedMount   pod/topolvm-controller-6fc6bcd4b5-xvhmk   MountVolume.SetUp failed for volume "certs" : secret "topolvm-mutatingwebhook" not found

Pod: topolvm-lvmd-0-fwfgb
Status: Running | Containers: 1 | Ready: false
[ERROR] Pod has containers in CrashLoopBackOff
Checking logs for container: lvmd
{"level":"info","ts":"2025-05-21T04:13:06Z","msg":"configuration file loaded","device_classes":[{"name":"ssd","volume-group":"myvg1","default":true,"spare-gb":10,"stripe":null,"stripe-size":"","lvcreate-options":null,"type":"","thin-pool":null}],"socket_name":"/run/topolvm/lvmd.sock","file_name":"/etc/topolvm/lvmd.yaml"}
not found
{"level":"error","ts":"2025-05-21T04:13:06Z","msg":"volume group not found","volume_group":"myvg1","error":"not found","stacktrace":"github.com/topolvm/topolvm/cmd/lvmd/app.subMain\n\t/workdir/cmd/lvmd/app/root.go:85\ngithub.com/topolvm/topolvm/cmd/lvmd/app.init.func2\n\t/workdir/cmd/lvmd/app/root.go:58\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/topolvm/topolvm/cmd/lvmd/app.Execute\n\t/workdir/cmd/lvmd/app/root.go:188\nmain.main\n\t/workdir/cmd/hypertopolvm/main.go:38\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}
Error: not found

Pod: topolvm-lvmd-0-gctr7
Status: Running | Containers: 1 | Ready: false
[ERROR] Pod has containers in CrashLoopBackOff
Checking logs for container: lvmd
{"level":"info","ts":"2025-05-21T04:12:59Z","msg":"configuration file loaded","device_classes":[{"name":"ssd","volume-group":"myvg1","default":true,"spare-gb":10,"stripe":null,"stripe-size":"","lvcreate-options":null,"type":"","thin-pool":null}],"socket_name":"/run/topolvm/lvmd.sock","file_name":"/etc/topolvm/lvmd.yaml"}
{"level":"error","ts":"2025-05-21T04:12:59Z","msg":"volume group not found","volume_group":"myvg1","error":"not found","stacktrace":"github.com/topolvm/topolvm/cmd/lvmd/app.subMain\n\t/workdir/cmd/lvmd/app/root.go:85\ngithub.com/topolvm/topolvm/cmd/lvmd/app.init.func2\n\t/workdir/cmd/lvmd/app/root.go:58\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/topolvm/topolvm/cmd/lvmd/app.Execute\n\t/workdir/cmd/lvmd/app/root.go:188\nmain.main\n\t/workdir/cmd/hypertopolvm/main.go:38\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}
Error: not found
not found

Pod: topolvm-lvmd-0-nfcwc
Status: Running | Containers: 1 | Ready: false

Pod: topolvm-node-8ps9n
Status: Running | Containers: 3 | Ready: false/true/false
[ERROR] Pod has containers in CrashLoopBackOff
Checking logs for container: csi-registrar
I0521 04:16:45.674628       1 main.go:150] "Version" version=""
I0521 04:16:45.674682       1 main.go:151] "Running node-driver-registrar" mode=""
I0521 04:16:55.674858       1 connection.go:253] "Still connecting" address="unix:///var/snap/microk8s/common/var/lib/kubelet/plugins/topolvm.io/node/csi-topolvm.sock"
I0521 04:17:05.675375       1 connection.go:253] "Still connecting" address="unix:///var/snap/microk8s/common/var/lib/kubelet/plugins/topolvm.io/node/csi-topolvm.sock"
I0521 04:17:15.675367       1 connection.go:253] "Still connecting" address="unix:///var/snap/microk8s/common/var/lib/kubelet/plugins/topolvm.io/node/csi-topolvm.sock"
E0521 04:17:15.675820       1 main.go:176] "Error connecting to CSI driver" err="context deadline exceeded"
Checking logs for container: topolvm-node
{"level":"info","ts":"2025-05-21T04:13:48Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2025-05-21T04:13:48Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-05-21T04:13:48Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Starting EventSource","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume","source":"kind source: *v1.LogicalVolume"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Starting Controller","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Starting workers","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume","worker count":1}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"All workers finished","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Stopping and waiting for caches"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Stopping and waiting for HTTP servers"}
{"level":"info","ts":"2025-05-21T04:13:48Z","logger":"controller-runtime.metrics","msg":"Shutting down metrics server with timeout of 1 minute"}
{"level":"info","ts":"2025-05-21T04:13:48Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2025-05-21T04:13:48Z","logger":"setup","msg":"problem running manager","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/topolvm/lvmd.sock: connect: no such file or directory\"","stacktrace":"github.com/topolvm/topolvm/cmd/topolvm-node/app.subMain\n\t/workdir/cmd/topolvm-node/app/run.go:159\ngithub.com/topolvm/topolvm/cmd/topolvm-node/app.init.func1\n\t/workdir/cmd/topolvm-node/app/root.go:41\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/topolvm/topolvm/cmd/topolvm-node/app.Execute\n\t/workdir/cmd/topolvm-node/app/root.go:48\nmain.main\n\t/workdir/cmd/hypertopolvm/main.go:42\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/topolvm/lvmd.sock: connect: no such file or directory"
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/topolvm/lvmd.sock: connect: no such file or directory"

Pod: topolvm-node-l2jr9
Status: Running | Containers: 3 | Ready: true/true/true

Pod: topolvm-node-mfg4n
Status: Running | Containers: 3 | Ready: false/true/false
[ERROR] Pod has containers in CrashLoopBackOff
Checking logs for container: csi-registrar
I0521 04:16:03.782511       1 main.go:150] "Version" version=""
I0521 04:16:03.782550       1 main.go:151] "Running node-driver-registrar" mode=""
I0521 04:16:13.783267       1 connection.go:253] "Still connecting" address="unix:///var/snap/microk8s/common/var/lib/kubelet/plugins/topolvm.io/node/csi-topolvm.sock"
I0521 04:16:23.782776       1 connection.go:253] "Still connecting" address="unix:///var/snap/microk8s/common/var/lib/kubelet/plugins/topolvm.io/node/csi-topolvm.sock"
I0521 04:16:33.782855       1 connection.go:253] "Still connecting" address="unix:///var/snap/microk8s/common/var/lib/kubelet/plugins/topolvm.io/node/csi-topolvm.sock"
E0521 04:16:33.783311       1 main.go:176] "Error connecting to CSI driver" err="context deadline exceeded"
Checking logs for container: topolvm-node
{"level":"info","ts":"2025-05-21T04:12:50Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2025-05-21T04:12:50Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-05-21T04:12:50Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Starting EventSource","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume","source":"kind source: *v1.LogicalVolume"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Starting Controller","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Starting workers","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume","worker count":1}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"All workers finished","controller":"logicalvolume","controllerGroup":"topolvm.io","controllerKind":"LogicalVolume"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Stopping and waiting for caches"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Stopping and waiting for HTTP servers"}
{"level":"info","ts":"2025-05-21T04:12:50Z","logger":"controller-runtime.metrics","msg":"Shutting down metrics server with timeout of 1 minute"}
{"level":"info","ts":"2025-05-21T04:12:50Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2025-05-21T04:12:50Z","logger":"setup","msg":"problem running manager","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/topolvm/lvmd.sock: connect: no such file or directory\"","stacktrace":"github.com/topolvm/topolvm/cmd/topolvm-node/app.subMain\n\t/workdir/cmd/topolvm-node/app/run.go:159\ngithub.com/topolvm/topolvm/cmd/topolvm-node/app.init.func1\n\t/workdir/cmd/topolvm-node/app/root.go:41\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/topolvm/topolvm/cmd/topolvm-node/app.Execute\n\t/workdir/cmd/topolvm-node/app/root.go:48\nmain.main\n\t/workdir/cmd/hypertopolvm/main.go:42\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/topolvm/lvmd.sock: connect: no such file or directory"
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/topolvm/lvmd.sock: connect: no such file or directory"

Note: For detailed debugging, use 'kubectl describe pod <name> -n topolvm-system'