Note: this doc credit to Fupan Li

cri

Note: The standalone cri-containerd binary is end-of-life. cri-containerd is transitioning from a standalone binary that talks to containerd to a plugin within containerd. This github branch is for the cri plugin. See standalone-cri-containerd branch for information about the standalone version of cri-containerd.

Note: You need to drain your node before upgrading from standalone cri-containerd to containerd with cri plugin.

cri is a containerd plugin implementation of Kubernetes container runtime interface (CRI).

With it, you could run Kubernetes using containerd as the container runtime.

Current Status

cri is a native plugin of containerd 1.1 and above. It is built into containerd and enabled by default.

cri is in GA:

It is feature complete.
It (the GA version) works with Kubernetes 1.10 and above.
It has passed all CRI validation tests.
It has passed all node e2e tests.
It has passed all e2e tests.

See test dashboard

Support Metrics

CRI-Containerd Version	Containerd Version	Kubernetes Version	CRI Version
v1.0.0-alpha.x		1.7, 1.8	v1alpha1
v1.0.0-beta.x		1.9	v1alpha1
End-Of-Life	v1.1	1.10+	v1alpha2
	HEAD	1.10+	v1alpha2

Production Quality Cluster on GCE

For a production quality cluster on GCE brought up with kube-up.sh refer here.

Installing with Ansible and Kubeadm

For a multi node cluster installer and bring up steps using ansible and kubeadm refer here.

Custom Installation

For non ansible users, you can download the cri-containerd release tarball and deploy kubernetes cluster using kubeadm as described here.

Getting Started for Developers

Binary Dependencies and Specifications

The current release of the cri plugin has the following dependencies:

See versions of these dependencies cri is tested with.

As containerd and runc move to their respective general availability releases, we will do our best to rebase/retest cri with these releases on a weekly/monthly basis. Similarly, given that cri uses the Open Container Initiative (OCI) image and runtime specifications, we will also do our best to update cri to the latest releases of these specifications as appropriate.

Install Dependencies

Install development libraries:

libseccomp development library. Required by cri and runc seccomp support. libseccomp-dev (Ubuntu, Debian) / libseccomp-devel (Fedora, CentOS, RHEL). On releases of Ubuntu <=Trusty and Debian <=jessie a backport version of libseccomp-dev is required. See travis.yml for an example on trusty.
btrfs development library. Required by containerd btrfs support. btrfs-tools(Ubuntu, Debian) / btrfs-progs-devel(Fedora, CentOS, RHEL)

Install socat (required by portforward).
Install and setup a go 1.10 development environment.
Make a local clone of this repository.
Install binary dependencies by running the following command from your cloned cri/ project directory:

# Note: install.deps installs the above mentioned runc, containerd, and CNI
# binary dependencies. install.deps is only provided for general use and ease of
# testing. To customize `runc` and `containerd` build tags and/or to configure
# `cni`, please follow instructions in their documents.
make install.deps

Build and Install `cri`

To build and install a version of containerd with the cri plugin, enter the following commands from your cri project directory:

make
sudo make install

NOTE: The version of containerd built and installed from the Makefile is only for testing purposes. The version tag carries the suffix "-TEST".

Build Tags

cri supports optional build tags for compiling support of various features. To add build tags to the make option the BUILD_TAGS variable must be set.

make BUILD_TAGS='seccomp apparmor'

Build Tag	Feature	Dependency
seccomp	syscall filtering	libseccomp development library
selinux	selinux process and mount labeling
apparmor	apparmor profile support

To use Kata as the runtime for containerd + cri

Build and Install `kata-runtime`

$ go get -d -u github.com/hyperhq/kata-runtime
$ mv $GOPATH/src/github.com/hyperhq $GOPATH/src/github.com/kata-containers
$ mv $GOPATH/src/github.com/kata-containers/kata-runtime $GOPATH/src/github.com/kata-containers/runtime 
$ cd $GOPATH/src/github.com/kata-containers/runtime
$ git checkout shimv2
$ make && sudo -E PATH=$PATH make install

The build will create the following:

runtime binary shim: /usr/local/bin/containerd-shim-kata-v2
configuration file: /usr/share/defaults/kata-containers/configuration.toml

Enable the factory template feature by editing the configure file

$ sudo sed -i 's/^#*enable_template/enable_template/' /usr/share/defaults/kata-containers/configuration.toml

Uncoment the image setting, let kata use the default initrd

$ sudo sed -i 's/^image =/#image =/'  /usr/share/defaults/kata-containers/configuration.toml

Create and install kata initrd image

Build a custom Kata agent

$ go get -d -u github.com/kata-containers/agent
$ cd $GOPATH/src/github.com/kata-containers/agent && make

Get the osbuilder

$ go get -d -u github.com/kata-containers/osbuilder

Create a local rootfs for initrd image

$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs"
$ sudo rm -rf ${ROOTFS_DIR}
$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
$ sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true ./rootfs.sh alpine'

AGENT_INIT controls if the guest image uses kata agent as the guest init process. When you create an initrd image, always set AGENT_INIT to yes.

Optionally, add your custom agent binary to the rootfs with the following:

$ sudo install -o root -g root -m 0550 -T ../../agent/kata-agent ${ROOTFS_DIR}/sbin/init

Build an initrd image

$ cd $GOPATH/src/github.com/kata-containers/osbuilder/initrd-builder
$ sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh ${ROOTFS_DIR}'

Install the initrd image

$ commit=$(git log --format=%h -1 HEAD)
$ date=$(date +%Y-%m-%d-%T.%N%z)
$ image="kata-containers-initrd-${date}-${commit}"
$ sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "/usr/share/kata-containers/${image}"
$ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img)

Install guest kernel images for kata

As a prerequisite, you need to install libelf-dev and bc. Otherwise, you will not be able to build the kernel from sources.

$ go get github.com/kata-containers/tests
$ cd $GOPATH/src/github.com/kata-containers/tests/.ci
$ kernel_arch="$(./kata-arch.sh)"
$ kernel_dir="$(./kata-arch.sh --kernel)"
$ tmpdir="$(mktemp -d)"
$ pushd "$tmpdir"
$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/configs/${kernel_dir}_kata_kvm_4.14.x -o .config
$ kernel_version=$(grep "Linux/[${kernel_arch}]*" .config | cut -d' ' -f3 | tail -1)
$ kernel_tar_file="linux-${kernel_version}.tar.xz"
$ kernel_url="https://cdn.kernel.org/pub/linux/kernel/v$(echo $kernel_version | cut -f1 -d.).x/${kernel_tar_file}"
$ curl -LOk ${kernel_url}
$ tar -xf ${kernel_tar_file}
$ mv .config "linux-${kernel_version}"
$ pushd "linux-${kernel_version}"
$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/patches/0001-NO-UPSTREAM-9P-always-use-cached-inode-to-fill-in-v9.patch | patch -p1
$ make ARCH=${kernel_dir} -j$(nproc)
$ kata_kernel_dir="/usr/share/kata-containers"
$ kata_vmlinuz="${kata_kernel_dir}/kata-vmlinuz-${kernel_version}.container"
$ [ $kernel_arch = ppc64le ] && kernel_file="$(realpath ./vmlinux)" || kernel_file="$(realpath arch/${kernel_arch}/boot/bzImage)"
$ sudo install -o root -g root -m 0755 -D "${kernel_file}" "${kata_vmlinuz}"
$ sudo ln -sf "${kata_vmlinuz}" "${kata_kernel_dir}/vmlinuz.container"
$ kata_vmlinux="${kata_kernel_dir}/kata-vmlinux-${kernel_version}"
$ sudo install -o root -g root -m 0755 -D "$(realpath vmlinux)" "${kata_vmlinux}"
$ sudo ln -sf "${kata_vmlinux}" "${kata_kernel_dir}/vmlinux.container"
$ popd
$ popd
$ rm -rf "${tmpdir}"

To use kata-runtime as the containerd runtime for CRI

please set the following in the containerd's config.toml file

[plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.kata.v2"
        runtime_engine = "io.containerd.runtime.kata.v2"

Validate Your `cri` Setup

A Kubernetes incubator project called cri-tools includes programs for exercising CRI implementations such as the cri plugin. More importantly, cri-tools includes the program critest which is used for running CRI Validation Testing.

Run the CRI Validation test to validate your installation of containerd with cri built in:

make test-cri

Running a Kubernetes local cluster

If you already have a working development environment for supported Kubernetes version, you can try cri in a local cluster:

Start the version of containerd with cri plugin that you built and installed above as root in a first terminal:

sudo containerd

From the Kubernetes project directory startup a local cluster using containerd:

CONTAINER_RUNTIME=remote CONTAINER_RUNTIME_ENDPOINT='unix:///run/containerd/containerd.sock' ./hack/local-up-cluster.sh

Test

See here for information about test.

Using crictl

See here for information about using crictl to debug pods, containers, and images.

Configurations

See here for information about how to configure cri plugins and here for information about how to configure containerd

Documentation

See here for additional documentation.

Contributing

Interested in contributing? Check out the documentation.

Communication

This project was originally established in April of 2017 in the Kubernetes Incubator program. After reaching the Beta stage, In January of 2018, the project was merged into containerd.

For async communication and long running discussions please use issues and pull requests on this github repo. This will be the best place to discuss design and implementation.

For sync communication we have a community slack with a #containerd channel that everyone is welcome to join and chat about development.

Slack: https://dockr.ly/community

Other Communications

As this project is tightly coupled to CRI and CRI-Tools and they are Kubernetes projects, some of our project communications take place in the Kubernetes' SIG: sig-node.

For more information about sig-node, CRI, and the CRI-Tools projects:

sig-node community site
Slack: #sig-node channel in Kubernetes (kubernetes.slack.com)
Mailing List: https://groups.google.com/forum/#!forum/kubernetes-sig-node

Reporting Security Issues

If you are reporting a security issue, please reach out discreetly at [email protected].

Licenses

The containerd codebase is released under the Apache 2.0 license. The README.md file, and files in the "docs" folder are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file "LICENSE.docs". You may obtain a duplicate copy of the same license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.

Code of Conduct

This project follows the CNCF Code of Conduct.

gnawux/Containerd-Kata-shim-v2.md

cri