Kata agent shutdown testing / debugging

kata-agent-shutdown-testing.md

• Use osbuilder to create a custom rootfs.

• Ensure you add extra packages for bash, strace, gdb (and maybe rust-gdb).

• Install an updated kata-agent.service file into the image:

  $ cat kata-agent.service.in
  #
  # Copyright (c) 2018-2019 Intel Corporation
  #
  # SPDX-License-Identifier: Apache-2.0
  #
  
  [Unit]
  Description=Kata Containers Agent
  Documentation=https://github.com/kata-containers/kata-containers
  Wants=kata-containers.target
  
  [Service]
  # Send agent output to tty to allow capture debug logs
  # from a VM vsock port
  # XXX: allow debug console to work
  #StandardOutput=journal
  # ------------------------------
  # XXX: Important!
  StandardInput=null
  #StandardOutput=journal
  #StandardError=journal
  StandardOutput=file:/run/kata-agent.log
  StandardError=file:/run/kata-agent.log
  # ------------------------------
  Type=simple
  ExecStart=@BINDIR@/@AGENT_NAME@
  LimitNOFILE=infinity
  # ExecStop is required for static agent tracing; in all other scenarios
  # the runtime handles shutting down the VM.
  ExecStop=/bin/sync ; /usr/bin/systemctl --force poweroff
  FailureAction=poweroff
  # Discourage OOM-killer from touching the agent
  OOMScoreAdjust=-997

• Install an new kata-debug.service file into the image:

  [Unit]
  Description=Kata Containers debug console
  
  [Service]
  Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  StandardInput=tty
  StandardOutput=tty
  # Must be disabled to allow the job to access the real console
  PrivateDevices=no
  Type=simple
  ExecStart=/bin/bash
  Restart=always
  
  # FIXME: testing to avoid having to add "Requires=kata-debug.service"
  # to kata-containers.target
  [Install]
  WantedBy=kata-containers.target

• Build the agent and install the agent and debug services copy into the rootfs:

  $ export ROOTFS="..."
  $ sudo install -o root -g root -m 0550 -t "${ROOTFS}/bin" "./target/x86_64-unknown-linux-musl/release/kata-agent"
  $ export debug_unit="kata-debug.service"
  $ sudo install -o root -g root -m 0440 ~/"${debug_unit}" "${ROOTFS}/usr/lib/systemd/system/"
  $ sudo chroot ${ROOTFS} systemctl enable "${debug_unit}"

• Build an image file using osbuilder.

• Enable runtime + agent debug in configuration.toml, but DO NOT enable hypervisor.qemu debug!!

• Start a Kata container:

  $ sudo ctr run --runtime "io.containerd.kata.v2" --rm -t "docker.io/library/busybox:latest" foo sh

• Connect to the debug console (change $id to your container id):

  $ console="/var/run/vc/vm/${id}/console.sock"
  $ sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}"

  WARNING: This will take ~10 seconds to start if using a clearlinux rootfs. Press return to see the prompt after that time!

• In the debug console:

  tail -f /run/kata-agent.log

• In the ctr console:

  # exit

• Notice that the kata-agent process is still running.