OpenShift 4 is an operator-focused NoOps Platform. Almost all services are managed by cluster operators with the exception of few services like crio and kubelet. crio and kubelet run as systemd services.
Systemd services are appropriate for services that you need to always come up on that particular system shortly after it starts.
- The CRI-O container engine (crio), which runs and manages the containers
- Kubelet (kubelet), which accepts requests for managing containers on the machine from master services
With everything else running as pods, you can monitor and debug them using traditional kubernetes debugging steps.
If you want to check logs of systemd services you normally run journalctl. That means you need to hop on to the node where it is running using oc debug. But there is an easier way to check logs of systemd services using oc cli.
# Find nodes in cluster
oc get nodes
# Get last 50 lines of kubelet log
oc adm node-logs <node_name> -u kubelet --tail 50
Note: There is a default 30 seconds context deadline set when streaming journal logs out of the nodes to protect API servers. So gather logs in a limited scope with --tail or --since or --until flags.
Similarly, you can read log files from /var/log also with same approach.
# List log files in /var/log
oc adm node-logs <node_name> --path=/
# Get /var/log/messages
oc adm node-logs <node_name> --path=/messages
# Get /var/log/audit/audit.log
oc adm node-logs <node_name> --path=/audit/audit.log