Idea for a "full fat" Kata Containers guest kernel

kata-full-fat-kernel.md

Full fat kernel

Background

• Originally, the guest kernel was as minimal as possible.
• Booted extremely fast, but didn't support all uses-cases.
• Over time, we have added lots of extra kernel CONFIG_* options...
  • For new kernel options
  • For new hardware features

Problems

• "Death by 1000 cuts": we are slowly bloating the kernel over time.
• Kernel now much bigger/slower than the original.
• Every time we do this, we need to adjust the metrics tests to take account of the changes.

Idea

Why not just enable all kernel options for all architectures?

Rationale

For the common case:

• The guest environment should be "transparent" to the workload.

• It should be atleast as functional as the host environment.

  (to be as equivalent as possible to a "traditional Linux container").

Impacts (page 1)

• "boot speed" and "memory density" will probably be impacted.

• But is that a problem?

  • The default Kata Containers configuration is NOT optimised.
  • Building "full fat" emphasises that, which is a good thing.

Impacts (page 2)

• Difficult / impossible to test "exotic" features in a public CI:

  • CI's don't provide all hardware features

    (often environment already virtualised).

  • CI's don't always support all architectures.

  • However, we are already facing these challenges.

    A full fat kernel provides an opportunity to overcome this though, without custom kernels.

Impacts (page 3)

• Enabling all config option provides a larger attack surface.
  • Is this a concern?
  • Maybe, but current default kernel may not be ideal for production use anyway.

Benefits (page 1)

• Enabling all options is the distro approach for kernels.

• Ensures all use cases "just work" (TM)

  Helpful to newcomers (Principle of least astonishment).

  Examples of currently problematic use-cases, all of which require rebuilding guest kernels: QAT and GPU passthrough (1, 2).

Benefits (page 2)

• In some ways, a slow(er) boot or larger memory footprint is helpful: it emphasises that: Kata is NOT optimised "out of the box"!. It can't be!

• Simplifies packaging

  Rather than packaging (and maintaining!) multiple set of assets for features like QAT or GPU passthrough, the default kernels would work.

Considerations (page 1)

• This change would (have to) alter the meaning of the current kernel "fragments".

  • Rather than defining a "small-ish kernel", they could become the true minimal configurations.

  • This implies the CI would need some extra jobs:

    kernel type	test workloads
    "full fat"	any
    minimal	metrics

Considerations (page 2)

• Need ability to package "custom configurations" (kernel, rootfs, config files).

• Configurations could be use-case or architecture specific.

• For example, we might want to offer example configurations for:

  • "fastest boot"
  • "smallest (memory resident) kernel" / maximum memory density.

Questions

• If we do this, should we build the kernel monolithically or use modules?

• How do we package custom configurations? (GitHub, Snap, OBS, other, ...?)

• Where would such packages be hosted?

• How would such packages be tested?

Summary

Status Quo:

• We currently have a slightly "chubby kernel" which offers most features.

• Difficult for users to explore newer hardware features.

• A full fat kernel would:

  • Help usability
  • Allow us to focus more aggressively on other "profiles"

References

• Issue 3146
• Guest kernel configuration

Waiting to be presented to the Kata Containers Architecture Committee meeting.

To view as HTML:

$ infile="kata-full-fat-kernel.md"
$ outfile="/tmp/full-fat-kernel.html"
$ pandoc -s -f markdown -t revealjs -V revealjs-url="https://unpkg.com/reveal.js" --metadata title="Kata Containers"--metadata subtitle="A \"full fat\" guest kernel?" --metadata author="James O. D. Hunt" --metadata date="14 December 2021" --incremental -o "$outfile" "$infile"
$ "$BROWSER" "$outfile"

See also:

• kata-containers/kata-containers#3399
• kata-containers/kata-containers#3146

Presented at the AC meeting today.