1. Open Container Initiative

The Open Container Initiative (OCI) is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

Docker is donating its container format and runtime, runC, to the OCI to serve as the cornerstone of this new effort.

2. runC

unix principles: several simple components are better than a single, complicated one.

runC is a lightweight, portable container runtime for creating and running containers according to the OCI container runtime specification.

  • runC only supports Linux.

  • runc is a CLI tool for spawning and running containers on Linux according to the OCI specification.

runhcs is a fork of runc. Like runc, runhcs is a command line client for running applications packaged on Windows according to the OCI format and is a compliant implementation of the OCI specification.

$ runc help
NAME:
   runc - Open Container Initiative runtime

runc is a command line client for running applications packaged according to
the Open Container Initiative (OCI) format and is a compliant implementation of the
Open Container Initiative specification.

runc integrates well with existing process supervisors to provide a production
container runtime environment for applications. It can be used with your
existing process monitoring tools and the container will be spawned as a
direct child of the process supervisor.

Containers are configured using bundles. A bundle for a container is a directory
that includes a specification file named "config.json" and a root filesystem.
The root filesystem contains the contents of the container.

To start a new instance of a container:

    # runc run [ -b bundle ] <container-id>

Where "<container-id>" is your name for the instance of the container that you
are starting. The name you provide for the container instance must be unique on
your host. Providing the bundle directory using "-b" is optional. The default
value for "bundle" is the current directory.

USAGE:
   runc [global options] command [command options] [arguments...]

VERSION:
   1.1.8
commit: v1.1.8-0-g82f18fe
. . .
Lists containers started by runc using Docker with the given root (default "/var/run/docker")
$ dockerd --help | grep exec-root
      --exec-root string                        Root directory for execution state files (default "/var/run/docker")
$ # --root value        root directory for storage of container state (this should be located in tmpfs) (default: "/run/runc")
$ sudo runc --root /var/run/docker/runtime-runc/moby list
ID                                                                 PID         STATUS      BUNDLE                                                                                                                CREATED                          OWNER
011d6fd316e09fc8a5ff3b226d35b9394cd93c57604c91aa52573559368a822c   940971      running     /run/containerd/io.containerd.runtime.v2.task/moby/011d6fd316e09fc8a5ff3b226d35b9394cd93c57604c91aa52573559368a822c   2021-11-25T04:10:25.216394136Z   root
. . .
Lists containers started by runc using Kubernetes with the given root (default "/run/containerd/runc/k8s.io")
$ containerd config dump | grep '^state'
state = "/run/containerd"
$ sudo runc --root /run/containerd/runc/k8s.io list
ID                                                                 PID         STATUS      BUNDLE                                                                                                                  CREATED                          OWNER
14542e5b60446f87af20e200e019a15d1ad1509eb506a2068266b9f98694c704   1191        running     /run/containerd/io.containerd.runtime.v2.task/k8s.io/14542e5b60446f87af20e200e019a15d1ad1509eb506a2068266b9f98694c704   2021-12-08T12:55:21.616268255Z   root
. . .
$ sudo runc --root /run/containerd/runc/k8s.io state 14542e5b60446f87af20e200e019a15d1ad1509eb506a2068266b9f98694c704
{
  "ociVersion": "1.0.2-dev",
  "id": "14542e5b60446f87af20e200e019a15d1ad1509eb506a2068266b9f98694c704",
  "pid": 1191,
  "status": "running",
  "bundle": "/run/containerd/io.containerd.runtime.v2.task/k8s.io/14542e5b60446f87af20e200e019a15d1ad1509eb506a2068266b9f98694c704",
  "rootfs": "/run/containerd/io.containerd.runtime.v2.task/k8s.io/14542e5b60446f87af20e200e019a15d1ad1509eb506a2068266b9f98694c704/rootfs",
  "created": "2021-12-08T12:55:21.616268255Z",
  "annotations": {
    "io.kubernetes.cri.container-name": "kube-flannel",
    "io.kubernetes.cri.container-type": "container",
    "io.kubernetes.cri.image-name": "quay.io/coreos/flannel:v0.15.0",
    "io.kubernetes.cri.sandbox-id": "f0c7bf11fac17f29a8df40f1d937ec35df81e202e42ea8a604e37806aebbb662",
    "io.kubernetes.cri.sandbox-name": "kube-flannel-ds-6xpbj",
    "io.kubernetes.cri.sandbox-namespace": "kube-system"
  },
  "owner": ""
}
$ sudo runc --root /run/containerd/runc/k8s.io ps 14542e5b60446f87af20e200e019a15d1ad1509eb506a2068266b9f98694c704
UID         PID   PPID  C STIME TTY          TIME CMD
root       1191    806  0 Dec08 ?        00:00:25 /opt/bin/flanneld --ip-masq --kube-subnet-mgr

3. containerd

containerd is available as a daemon for Linux and Windows. It manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond.

architecture containerd platform

containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users.

There are many different ways to use containerd:

  • If you are a developer working on containerd you can use the ctr tool to quickly test features and functionality without writing extra code

  • If you want to integrate containerd into your project, you can use a simple client package.

$ ctr help
NAME:
   ctr -
        __
  _____/ /______
 / ___/ __/ ___/
/ /__/ /_/ /
\___/\__/_/

containerd CLI


USAGE:
   ctr [global options] command [command options] [arguments...]

VERSION:
   1.4.11

DESCRIPTION:

ctr is an unsupported debug and administrative client for interacting
with the containerd daemon. Because it is unsupported, the commands,
options, and operations are not guaranteed to be backward compatible or
stable from release to release of the containerd project.

COMMANDS:
   plugins, plugin            provides information about containerd plugins
   version                    print the client and server versions
   containers, c, container   manage containers
   content                    manage content
   events, event              display containerd events
   images, image, i           manage images
   leases                     manage leases
   namespaces, namespace, ns  manage namespaces
   pprof                      provide golang pprof outputs for containerd
   run                        run a container
   snapshots, snapshot        manage snapshots
   tasks, t, task             manage tasks
   install                    install a new package
   oci                        OCI tools
   shim                       interact with a shim directly
   help, h                    Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --debug                      enable debug output in logs
   --address value, -a value    address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS]
   --timeout value              total timeout for ctr commands (default: 0s)
   --connect-timeout value      timeout for connecting to containerd (default: 0s)
   --namespace value, -n value  namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE]
   --help, -h                   show help
   --version, -v                print the version
Save image from Docker and import to containerd
$ docker save nginx:1.25 | xz -zv -T0 > nginx.1.25.tar.xz
  100 %        41.9 MiB / 182.0 MiB = 0.230   8.0 MiB/s       0:22
$ xz -dk nginx.1.25.tar.xz
$ ls
nginx.1.25.tar  nginx.1.25.tar.xz
$ sudo ctr i import nginx.1.25.tar # import to the default namespace
unpacking docker.io/library/nginx:1.25 (sha256:7477fb7aa691ad976bdd0f12afd00c094e8bef473051e5125591f532efd21022)...done
$ sudo ctr ns ls
NAME    LABELS
default
k8s.io
moby
$ sudo ctr i ls # same as `sudo ctr -n default i ls`
REF                          TYPE                                                 DIGEST                                                                  SIZE      PLATFORMS   LABELS
docker.io/library/nginx:1.25 application/vnd.docker.distribution.manifest.v2+json sha256:7477fb7aa691ad976bdd0f12afd00c094e8bef473051e5125591f532efd21022 182.0 MiB linux/amd64 -
Show the information about containerd plugins
$ sudo ctr plugin ls
TYPE                                  ID                       PLATFORMS      STATUS
io.containerd.content.v1              content                  -              ok
. . .
io.containerd.snapshotter.v1          overlayfs                linux/amd64    ok
io.containerd.snapshotter.v1          zfs                      linux/amd64    skip
io.containerd.metadata.v1             bolt                     -              ok
. . .
io.containerd.grpc.v1                 cri                      linux/amd64    ok

3.1. containerd config

containerd is meant to be a simple daemon to run on any system. It provides a minimal config with knobs to configure the daemon and what plugins are used when necessary.

$ containerd help
NAME:
   containerd -

. . .

USAGE:
   containerd [global options] command [command options] [arguments...]

VERSION:
   1.4.11

DESCRIPTION:

containerd is a high performance container runtime whose daemon can be started
by using this command. If none of the *config*, *publish*, or *help* commands
are specified, the default action of the **containerd** command is to start the
containerd daemon in the foreground.


A default configuration is used if no TOML configuration is specified or located
at the default file location. The *containerd config* command can be used to
generate the default configuration for containerd. The output of that command
can be used and modified as necessary as a custom configuration.

COMMANDS:
   config    information on the containerd config
   publish   binary to publish events to containerd
   oci-hook  provides a base for OCI runtime hooks to allow arguments to be injected.
   help, h   Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --config value, -c value     path to the configuration file (default: "/etc/containerd/config.toml")
   --log-level value, -l value  set the logging level [trace, debug, info, warn, error, fatal, panic]
   --address value, -a value    address for containerd's GRPC server
   --root value                 containerd root directory
   --state value                containerd state directory
   --help, -h                   show help
   --version, -v                print the version

While a few daemon level options can be set from CLI flags, the majority of containerd’s configuration is kept in the configuration file. The default path for the config file is located at /etc/containerd/config.toml. You can change this path via the --config,-c flags when booting the daemon.

In the containerd config file you will find settings for persistent and runtime storage locations as well as grpc, debug, and metrics addresses for the various APIs.

$ sudo containerd config dump # See the output of the final main config
. . .
root = "/var/lib/containerd"
state = "/run/containerd"
. . .
  • root will be used to store any type of persistent data for containerd. Snapshots, content, metadata for containers and image, as well as any plugin data will be kept in this location.

    The root is also namespaced for plugins that containerd loads. Each plugin will have its own directory where it stores data. containerd itself does not actually have any persistent data that it needs to store, its functionality comes from the plugins that are loaded.

    /var/lib/containerd/
    ├── io.containerd.content.v1.content
    │   └── ingest
    ├── io.containerd.metadata.v1.bolt
    │   └── meta.db
    ├── io.containerd.runtime.v1.linux
    ├── io.containerd.runtime.v2.task
    ├── io.containerd.snapshotter.v1.btrfs
    ├── io.containerd.snapshotter.v1.native
    │   └── snapshots
    ├── io.containerd.snapshotter.v1.overlayfs
    │   └── snapshots
    └── tmpmounts
  • state will be used to store any type of ephemeral data. Sockets, pids, runtime state, mount points, and other plugin data that must not persist between reboots are stored in this location.

    run/containerd/
    ├── containerd.sock
    ├── containerd.sock.ttrpc
    ├── io.containerd.runtime.v1.linux
    └── io.containerd.runtime.v2.task

Both the root and state directories are namespaced for plugins.

By the way, you can also type the command: containerd config default to print the output of the default config. The follow sample is used by Docker CE as default.

disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"

3.2. containerd plugins

At the end of the day, containerd’s core is very small. The real functionality comes from plugins. Everything from snapshotters, runtimes, and content are all plugins that are registered at runtime. Because these various plugins are so different we need a way to provide type safe configuration to the plugins. The only way we can do this is via the config file and not CLI flags.

3.2.1. Built-in Plugins

containerd uses plugins internally to ensure that internal implementations are decoupled, stable, and treated equally with external plugins. To see all the plugins containerd has, use ctr plugins ls.

$ sudo ctr plugin ls
TYPE                            ID                       PLATFORMS      STATUS
io.containerd.content.v1        content                  -              ok
io.containerd.snapshotter.v1    aufs                     linux/amd64    error
io.containerd.snapshotter.v1    btrfs                    linux/amd64    error
io.containerd.snapshotter.v1    devmapper                linux/amd64    error
io.containerd.snapshotter.v1    native                   linux/amd64    ok
io.containerd.snapshotter.v1    overlayfs                linux/amd64    ok
io.containerd.snapshotter.v1    zfs                      linux/amd64    error
io.containerd.metadata.v1       bolt                     -              ok
io.containerd.differ.v1         walking                  linux/amd64    ok
io.containerd.gc.v1             scheduler                -              ok
...

From the output all the plugins can be seen as well those which did not successfully load. In this case aufs and zfs are expected not to load since they are not support on the machine. The logs will show why it failed, but you can also get more details using the -d option.

$ sudo ctr plugin ls -d id==aufs id==zfs
Type:          io.containerd.snapshotter.v1
ID:            aufs
Platforms:     linux/amd64
Exports:
               root      /var/lib/containerd/io.containerd.snapshotter.v1.aufs
Error:
               Code:        Unknown
               Message:     aufs is not supported (modprobe aufs failed: exit status 1 "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.10.0-9-amd64\n"): skip plugin

Type:          io.containerd.snapshotter.v1
ID:            zfs
Platforms:     linux/amd64
Exports:
               root      /var/lib/containerd/io.containerd.snapshotter.v1.zfs
Error:
               Code:        Unknown
               Message:     path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin

3.2.2. Configuration

Plugins are configured using the [plugins] section of containerd’s config. Every plugin can have its own section using the pattern [plugins.<plugin id>].

[plugins]
  # indentation (tabs and/or spaces) is allowed but not required
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "k8s.gcr.io/pause:3.5"
    # <other paramters>
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      # <other paramters>
    [plugins."io.containerd.grpc.v1.cri".containerd]
        # <other paramters>
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          # <other paramters>
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            # <other paramters>
            SystemdCgroup = true

3.3. containerd/cri

cri is a containerd built-in plugin implementation of Kubernetes container runtime interface (CRI).

While OCI specs defines a single container, CRI (container runtime interface) describes containers as workload(s) in a shared sandbox environment called a pod. Pods can contain one or more container workloads.

With it, you could run Kubernetes using containerd as the container runtime.

cri

crictl is a command-line interface for CRI-compatible container runtimes.

$ sudo crictl pods
POD ID              CREATED             STATE               NAME                             NAMESPACE           ATTEMPT             RUNTIME
f69d876947d10       About an hour ago   Ready               coredns-5dd5756b68-6zhnn         kube-system         0                   (default)
5b182c180224e       About an hour ago   Ready               coredns-5dd5756b68-979lp         kube-system         0                   (default)
a61a8b645eca4       About an hour ago   Ready               kube-flannel-ds-fstxv            kube-flannel        0                   (default)
b9a8397f4d1ec       About an hour ago   Ready               kube-proxy-tlrsr                 kube-system         0                   (default)
f5c97e2b1dca3       About an hour ago   Ready               kube-controller-manager-node-1   kube-system         0                   (default)
05b17a7b61b01       About an hour ago   Ready               kube-apiserver-node-1            kube-system         0                   (default)
56a45a81d9c3a       About an hour ago   Ready               kube-scheduler-node-1            kube-system         0                   (default)
5da873ac2515e       About an hour ago   Ready               etcd-node-1                      kube-system         0                   (default)
$ sudo crictl inspectp f69d876947d10 | head
{
  "status": {
    "id": "f69d876947d103f23b41ca677e498468aaef6a9d35e287c6dcd999cf62e40dbd",
    "metadata": {
      "attempt": 0,
      "name": "coredns-5dd5756b68-6zhnn",
      "namespace": "kube-system",
      "uid": "f364d6dd-ba20-4ab6-8ebb-0053ac1b43e0"
    },
    "state": "SANDBOX_READY",

3.4. containerd/namespaces

containerd offers a fully namespaced API so multiple consumers can all use a single containerd instance without conflicting with one another.

Namespaces allow multi-tenancy within a single daemon.

Consumers are able to have containers with the same names but with settings and/or configurations that vary drastically. For example, system or infrastructure level containers can be hidden in one namespace while user level containers are kept in another. Underlying image content is still shared via content addresses but image names and metadata are separate per namespace.

Namespaces allow various features, most notably, the ability for one client to create, edit, and delete resources without affecting another client. A resource can be anything from an: image, container, task, or snapshot.

When a client queries for a resource, they only see the resources that are part of their namespace.

$ sudo ctr ns ls # list namespaces
NAME    LABELS
default
k8s.io
moby
$ # pull image to namespace alice (create it if not existed)
$ sudo ctr -n alice image pull docker.io/library/nginx:1.25
docker.io/library/nginx:1.25:                                                     resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:48a84a0728cab8ac558f48796f901f6d31d287101bc8b317683678125e0d2d35: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:eea7b3dcba7ee47c0d16a60cc85d2b977d166be3960541991f3e6294d795ed24:   done           |++++++++++++++++++++++++++++++++++++++|
. . .
elapsed: 65.9s                                                                    total:  66.8 M (1.0 MiB/s)
unpacking linux/amd64 sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c...
done: 2.224968944s
$ # pull image to namespace bob (create it if not existed)
$ sudo ctr -n bob image pull docker.io/library/nginx:1.25
docker.io/library/nginx:1.25:                                                     resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:48a84a0728cab8ac558f48796f901f6d31d287101bc8b317683678125e0d2d35: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:eea7b3dcba7ee47c0d16a60cc85d2b977d166be3960541991f3e6294d795ed24:   done           |++++++++++++++++++++++++++++++++++++++|
. . .
elapsed: 2.2 s                                                                    total:   0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c...
done: 2.453252148s
$ sudo ctr ns ls
NAME    LABELS
alice
bob
default
k8s.io
moby

As we see, there are namespaces alice and bob, but what are moby and k8s.io ?

  • moby is default namespace for dockerd and k8s.io is default namespace for kubelet, i.e. Kubernetes.

    $ dockerd --help | grep containerd-namespace
          --containerd-namespace string             Containerd namespace to use (default "moby")
    $ kubelet --help | grep containerd-namespace
          --containerd-namespace string                              containerd namespace (default "k8s.io") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)

Manage container

$ # run a container named `nginx-a`
$ sudo ctr -n alice run --null-io -d docker.io/library/nginx:1.25 nginx-a
$ # list containers
$ sudo ctr -n alice c ls
CONTAINER    IMAGE                           RUNTIME
nginx-a      docker.io/library/nginx:1.25    io.containerd.runc.v2
$ # list tasks
$ sudo ctr -n alice t ls
TASK       PID      STATUS
nginx-a    43776    RUNNING

$ sudo ctr -n bob container ls
CONTAINER    IMAGE    RUNTIME
$ sudo ctr -n bob task ls
TASK    PID    STATUS
$ sudo ctr -n bob run --null-io -d docker.io/library/nginx:latest nginx-b
$ sudo ctr -n bob t ls
TASK       PID       STATUS
nginx-b    967330    RUNNING

$ sudo ctr -n alice t ls
TASK       PID      STATUS
nginx-a    43776    RUNNING
$ sudo nsenter -t 43776 -a lsns
        NS TYPE   NPROCS PID USER COMMAND
4026531835 cgroup      3   1 root nginx: master process nginx -g daemon off;
4026531837 user        3   1 root nginx: master process nginx -g daemon off;
4026532706 mnt         3   1 root nginx: master process nginx -g daemon off;
4026532707 uts         3   1 root nginx: master process nginx -g daemon off;
4026532708 ipc         3   1 root nginx: master process nginx -g daemon off;
4026532709 pid         3   1 root nginx: master process nginx -g daemon off;
4026532711 net         3   1 root nginx: master process nginx -g daemon off;
$ sudo nsenter -t 43776 -a curl -iI localhost
HTTP/1.1 200 OK
Server: nginx/1.25.2
Date: Tue, 22 Aug 2023 09:44:58 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 15 Aug 2023 17:03:04 GMT
Connection: keep-alive
ETag: "64dbafc8-267"
Accept-Ranges: bytes

3.5. contianerd/proxy

The contianerd daemon uses the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environmental variables in its start-up environment to configure HTTP or HTTPS proxy behavior.

  1. Create a systemd drop-in directory for the containerd service:

    $ sudo mkdir -p /etc/systemd/system/containerd.service.d
  2. Create a file called http-proxy.conf at the above directory that adds the HTTP_PROXY environment variable:

    [Service]
    Environment="HTTP_PROXY=http://proxy.example.com:80/"

    Or, if you are behind an HTTPS proxy server, adds the HTTPS_PROXY environment variable:

    [Service]
    Environment="HTTP_PROXY=http://proxy.example.com:80/"
    Environment="HTTPS_PROXY=https://proxy.example.com:443/"

    If you have internal Docker registries that you need to contact without proxying you can specify them via the NO_PROXY environment variable:

    [Service]
    Environment="HTTP_PROXY=http://proxy.example.com:80/"
    Environment="HTTPS_PROXY=https://proxy.example.com:443/"
    Environment="NO_PROXY=localhost,127.0.0.1,docker-registry.somecorporation.com"

    The NO_PROXY environment variable specifies URLs that should be excluded from proxying (on servers that should be contacted directly). This should be a comma-separated list of hostnames, domain names, or a mixture of both. Asterisks can be used as wildcards, but other clients may not support that. Domain names may be indicated by a leading dot. For example:

    NO_PROXY="*.aventail.com,home.com,.seanet.com"

    says to contact all machines in the ‘aventail.com’ and ‘seanet.com’ domains directly, as well as the machine named ‘home.com’. If NO_PROXY isn’t defined, no_PROXY and no_proxy are also tried, in that order.

    You can also use the systemctl edit containerd to edit override.conf at /etc/systemd/system/containrd.service.d for the containerd service.
  3. Flush changes:

    $ sudo systemctl daemon-reload
  4. Restart containerd:

    $ sudo systemctl restart containerd
  5. Verify that the configuration has been loaded:

    $ systemctl show --property=Environment containerd --full --no-pager
    Environment=HTTP_PROXY=http://127.0.0.1:8118 HTTPS_PROXY=http://127.0.0.1:8118 NO_PROXY=localhost,127.0.0.1,docker.io,docker.com,docker-cn.com,aliyuncs.com,mcr.microsoft.com,mcrea0.blob.core.windows.net,.azurecr.io,.elastic.co,.cloudfront.net,quay.io,.amazonaws.com,.amazonaws.com.cn,mscr.io

4. crictl

crictl is a command-line interface for CRI-compatible container runtimes. You can use it to inspect and debug container runtimes and applications on a Kubernetes node. crictl and its source are hosted in the cri-tools repository.

$ sudo crictl image ls
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] unable to determine image API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: connection refused"
IMAGE               TAG                 IMAGE ID            SIZE

To solve the above problem, please specify the runtime-endpoint option:

$ sudo crictl --runtime-endpoint=unix:///run/containerd/containerd.sock image ls
IMAGE               TAG                 IMAGE ID            SIZE

or

set the the runtime-endpoint in configuration file /etc/crictl.yaml:

$ sudo crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock

$ sudo crictl image ls
IMAGE               TAG                 IMAGE ID            SIZE

crictl image list = ctr -n=k8s.io image list

$ sudo ctr -n k8s.io i ls
REF                                                                                               TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                                                                                          LABELS
docker.io/library/busybox:latest                                                                  application/vnd.docker.distribution.manifest.list.v2+json sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353 758.9 KiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/riscv64,linux/s390x io.cri-containerd.image=managed
docker.io/library/busybox@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353 application/vnd.docker.distribution.manifest.list.v2+json sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353 758.9 KiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/riscv64,linux/s390x io.cri-containerd.image=managed
k8s.gcr.io/pause:3.2                                                                              application/vnd.docker.distribution.manifest.v2+json      sha256:2a7b365f500c323286ac47e9e32af9bd50ee65de7fe2a27355eb5987c8df9ad8 669.7 KiB linux/amd64                                                                                                                        io.cri-containerd.image=managed
sha256:7138284460ffa3bb6ee087344f5b051468b3f8697e2d1427bac1a20c8d168b14                           application/vnd.docker.distribution.manifest.list.v2+json sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353 758.9 KiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/riscv64,linux/s390x io.cri-containerd.image=managed
sha256:80d28bedfe5dec59da9ebf8e6260224ac9008ab5c11dbbe16ee3ba3e4439ac2c                           application/vnd.docker.distribution.manifest.v2+json      sha256:61e45779fc594fcc1062bb9ed2cf5745b19c7ba70f0c93eceae04ffb5e402269 669.7 KiB linux/amd64                                                                                                                        io.cri-containerd.image=managed

$ sudo crictl image ls
IMAGE                       TAG                 IMAGE ID            SIZE
docker.io/library/busybox   latest              7138284460ffa       1.46MB
k8s.gcr.io/pause            3.2                 80d28bedfe5de       686kB

create a pod sandbox and run a container

container-config.json
{
  "metadata": {
    "name": "busybox"
  },
  "image":{
    "image": "busybox"
  },
  "command": [
    "top"
  ],
  "log_path":"busybox.0.log",
  "linux": {
  }
}
pod-config.json
{
  "metadata": {
    "name": "nginx-sandbox",
    "namespace": "default",
    "attempt": 1,
    "uid": "hdishd83djaidwnduwk28bcsb"
  },
  "log_directory": "/tmp",
  "linux": {
  }
}
$ sudo crictl run container-config.json pod-config.json
b08ad7b8517d0e37853f3a7211fbc7ba283a7b34cff5bd0ae108e9d956034a24

$ sudo crictl pods
POD ID              CREATED             STATE               NAME                NAMESPACE           ATTEMPT             RUNTIME
91ff0a7d5e81a       15 seconds ago      Ready               nginx-sandbox       default             1                   (default)
$ sudo crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
b08ad7b8517d0       busybox             15 seconds ago      Running             busybox             0                   91ff0a7d5e81a
$ sudo crictl stopp 91ff0a7d5e81a
Stopped sandbox 91ff0a7d5e81a
$ sudo crictl rmp 91ff0a7d5e81a
Removed sandbox 91ff0a7d5e81a

5. docker and dockershim

dockershim is a Docker CRI implementation for kubelet to interact with dockerd to manage containers.

cri containerd
dockershim deprecation was announced as a part of the Kubernetes v1.20 release.

Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community.

Introduce experimental support for containerd as the content store (replacing the existing storage drivers) of the Docker 24.0.

Switching to containerd snapshotters causes you to temporarily lose images and containers created using the classic storage drivers. Those resources still exist on your filesystem, and you can retrieve them by turning off the containerd snapshotters feature.

The following steps explain how to enable the containerd snapshotters feature.

  1. Add the following configuration to your /etc/docker/daemon.json configuration file:

    {
      "features": {
        "containerd-snapshotter": true
      }
    }
  2. Restart the daemon for the changes to take effect.

    sudo systemctl restart docker
  3. After restarting the daemon, running docker info shows that you’re using containerd snapshotter storage drivers.

    docker info -f '{{ .DriverStatus }}'

Docker Engine uses the overlayfs containerd snapshotter by default.

$ sudo ctr plugin ls
TYPE                                  ID                       PLATFORMS      STATUS
. . .
io.containerd.snapshotter.v1          overlayfs                linux/amd64    ok
. . .
moby is default namespace for dockerd and k8s.io is default namespace for kubelet, i.e. Kubernetes.
$ sudo docker info -f '{{.DriverStatus}}'
[[driver-type io.containerd.snapshotter.v1]]
$ sudo docker image ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
$ sudo ctr ns ls
NAME LABELS
moby
$ sudo ctr -n moby content ls
DIGEST	SIZE	AGE	LABELS
$ sudo ctr -n moby image pull docker.io/library/nginx:1.25.2
docker.io/library/nginx:1.25.2:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:48a84a0728cab8ac558f48796f901f6d31d287101bc8b317683678125e0d2d35: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:eea7b3dcba7ee47c0d16a60cc85d2b977d166be3960541991f3e6294d795ed24:   done           |++++++++++++++++++++++++++++++++++++++|
. . .
layer-sha256:e3b6889c89547ec9ba653ab44ed32a99370940d51df956968c0d578dd61ab665:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 65.9s                                                                    total:  66.8 M (1.0 MiB/s)
unpacking linux/amd64 sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c...
done: 1.641277314s
$ docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
nginx        1.25.2    104c7c5c54f2   7 seconds ago   272MB
$ docker pull nginx:1.25.2
104c7c5c54f2: Already exists
48a84a0728ca: Already exists
eea7b3dcba7e: Already exists
docker.io/library/nginx:1.25.2
$ docker image ls
REPOSITORY   TAG       IMAGE ID       CREATED              SIZE
nginx        1.25.2    104c7c5c54f2   About a minute ago   272MB
$ sudo ctr -n k8s.io i pull docker.io/library/nginx:1.25.2
docker.io/library/nginx:1.25.2:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:48a84a0728cab8ac558f48796f901f6d31d287101bc8b317683678125e0d2d35: done           |++++++++++++++++++++++++++++++++++++++|
. . .
layer-sha256:a7c4092be9044bd4eef78f27c95785ef3a9f345d01fd4512bc94ddaaefc359f4:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:eea7b3dcba7ee47c0d16a60cc85d2b977d166be3960541991f3e6294d795ed24:   done           |++++++++++++++++++++++++++++++++++++++|
. . .
layer-sha256:52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.3 s                                                                    total:   0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c...
done: 3.058695129s
$ kubectl get nodes -owide
NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
node-0   Ready    control-plane   50m   v1.28.0   192.168.91.128   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-9-amd64    containerd://1.6.22

Developers can still use the Docker platform to build, share, and run containers on Kubernetes!

If you’re using Docker, you’re already using containerd.

$ dockerd --help | grep containerd
      --containerd string                       containerd grpc address
      --containerd-namespace string             Containerd namespace to use (default "moby")
      --containerd-plugins-namespace string     Containerd namespace to use for plugins (default "plugins.moby")
      --cri-containerd                          start containerd with cri

The images Docker builds are compliant with OCI (Open Container Initiative), are fully supported on containerd, and will continue to run great on Kubernetes.

Docker’s runtime is built upon containerd while providing a great developer experience around it. For production environments that benefit from a minimal container runtime, such as Kubernetes, and may have no need for Docker’s great developer experience, it’s reasonable to directly use lightweight runtimes like containerd.

If you’re using Docker, you’ll find that the cri plugin was disabled at /etc/containerd/config.toml.

$ containerd config dump | grep 'disabled_plugins'
disabled_plugins = ["cri"]
$ # OR
$ grep cri /etc/containerd/config.toml
disabled_plugins = ["cri"]
$ # OR
$ sudo ctr plugin ls | grep cri

To migrate runtime from Docker to containerd, please enable the cri plugin, and specify the cri parameters --container-runtime=remote and --container-runtime-endpoint=/run/containerd/containerd.sock for kubelet.

Using the systemd cgroup driver for containerd

To use the systemd cgroup driver in /etc/containerd/config.toml with runc, set

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  # ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

If you apply this change make sure to restart containerd again:

$ sudo systemctl restart containerd

When using kubeadm, manually configure the cgroup driver for kubelet.

Using the pause image with kubeadm for cri plugin
[plugins]
  # ...
  [plugins."io.containerd.grpc.v1.cri"]
    # ...
    sandbox_image = "k8s.gcr.io/pause:3.5" # consider to keep same as the `--pod-infra-container-image` of kubelet
Use kubeadm to init a single node cluster with containerd
$ sudo ctr plugin ls | grep cri
io.containerd.grpc.v1           cri                      linux/amd64    ok

$ sudo kubeadm init --cri-socket /run/containerd/containerd.sock --ignore-preflight-errors NumCPU --kubernetes-version v1.22.3
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks

<other outputs>

Your Kubernetes control-plane has initialized successfully!

$ sudo kubectl get node -owide --kubeconfig /etc/kubernetes/admin.conf
NAME     STATUS   ROLES                  AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION    CONTAINER-RUNTIME
node-1   Ready    control-plane,master   6m55s   v1.22.4   192.168.91.137   <none>        Debian GNU/Linux 10 (buster)   4.19.0-17-amd64   containerd://1.4.8

$ sudo kubectl get no node-1 -ogo-template='{{.status.nodeInfo.containerRuntimeVersion}}' --kubeconfig /etc/kubernetes/admin.conf
containerd://1.4.8

$ systemctl status kubelet.service --no-page --full
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Thu 2021-11-25 17:29:02 CST; 21min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 38090 (kubelet)
    Tasks: 13 (limit: 2330)
   Memory: 54.7M
   CGroup: /system.slice/kubelet.service
           └─38090 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.5

It is recommended to keep the sandbox_image of containerd consistent with the pod-infra-container-image (also known as the pause container image) of the kubelet.

Both images are used to create the pause container, which serves as the "parent container" for all other containers in a Kubernetes pod. Ensuring that these images match helps maintain consistency and avoid potential issues within your Kubernetes environment.

The pause container holds the network namespace and other shared resources for all containers within a pod.

Having a consistent pause container image ensures that all components of your Kubernetes cluster use the same image, reducing the likelihood of conflicts and maintaining a unified environment.

To make sure both configurations are using the same image, follow these steps:

  1. Configure the sandbox_image in containerd’s configuration file, usually located at /etc/containerd/config.toml. For example:

    [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
      ...
      sandbox_image = "k8s.gcr.io/pause:3.6"
      ...
  2. Configure the pod-infra-container-image in the kubelet’s configuration file or command-line flags. For example, add the following flag to the kubelet’s command-line options:

    --pod-infra-container-image=k8s.gcr.io/pause:3.6
    $ sudo cat /var/lib/kubelet/kubeadm-flags.env
    KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"

    or set the pod-infra-container-image in the kubelet’s configuration file (usually /var/lib/kubelet/config.yaml):

    pod-infra-container-image: "k8s.gcr.io/pause:3.6"

After making these changes, restart the containerd and kubelet services to apply the new configurations.

By keeping the sandbox_image and pod-infra-container-image consistent, you can ensure that your Kubernetes cluster operates smoothly and avoids potential issues related to using different pause container images.