wlonkly

I wrote this down after I responded to a page today (a holiday) because it would've been a decent pairing opportunity for a couple of new people on my team. Second best is that people can read what I did afterwards and ask me any questions. And then I realized that there's nothing PagerDuty-specific or confidential in here, so I may as well share it wider. It's hardly an epic incident, but it's a good example of "doing the work", I think. I borrowed the "write down what you learned" approach from Julia "b0rk" Evans. It's a fantastic practice.

The PagerDuty incident: "Disk will be full in 12 hours. device:/dev/nvme0n1p1, host:stg-nomadusw2-client-..."

(Note for non-PD readers: We run Nomad where others might run Kubernetes.)

Here's the process I went through.

• Noticed that the usual docker system prune -a -f didn't resolve it
• Tried docker system prune -a -f and it cleared up 0B

dims

Folks, Leave me a comment / URL if something you like is missing!

Beginner Resources

Resource	Description
Kube Academy	https://kube.academy/
kuernetes-101	https://kube.academy/courses/kubernetes-101/
Docs Home	https://kubernetes.io/docs/home/
CKS CKA CKAD Simulator	https://killer.sh/

ehazlett

func withCheckpointMounts(c *container.Container) containerd.CheckpointOpts {
	return func(ctx context.Context, client *containerd.Client, _ *containers.Container, index *imagespec.Index, _ *options.CheckpointOptions) error {
		for _, m := range c.MountPoints {
			// tarstream of content
			// a tmpdir is created to use as the blank comparison directory for the diff
			tmpdir, err := ioutil.TempDir("", "container-data-")
			if err != nil {
				return err
			}
			defer os.RemoveAll(tmpdir)

innovia

# This script is built on kops bootstrap
# https://github.com/dcwangmit01/kops/blob/ef958a7f870eb3dc20981617859b8ad69057bb2a/hooks/nvidia-bootstrap/image/run.sh

mkdir nvidia
cd nvidia

# Get the NVIDIA driver
wget http://us.download.nvidia.com/tesla/390.46/NVIDIA-Linux-x86_64-390.46.run
chmod a+x NVIDIA-Linux-x86_64-390.46.run