Containerd客户端工具（CLI）介绍ctr,nerdctl,crictl，podman以及docker

下面设计到私有镜像库私有镜像库 http和https两种方式

 

浅谈 containerd 架构及高级技巧

containerd是一个高级容器运行时，又名容器管理器。简单来说，它是一个守护进程，在单个主机上管理完整的容器生命周期：创建、启动、停止容器、拉取和存储镜像、配置挂载、网络等。

Containerd被设计成可以很容易地嵌入到更大的系统中。

Docker使用containerd来运行容器。

Kubernetes可以通过CRI使用containerd来管理单个节点上的容器。

但是较小的项目也可以从与containerd集成的便利中获益——例如，faasd使用containerd在独立的服务器上运行一个成熟的功能即服务解决方案。

但是，以编程方式使用 containerd 并不是唯一的选择。它还可以通过可用客户端之一从命令行使用。由此产生的容器 UX 可能不像docker客户端提供的那样全面和用户友好，但它仍然是有用的，例如，用于调试或学习目的。

如何在 ctr 中使用 containerd

ctr是作为 containerd 项目的一部分提供的命令行客户端。如果您在一台机器上运行了 containerd，那么ctr二进制文件很可能也在那里。

该ctr界面 [显然] 与 Docker CLI不兼容，乍一看，可能看起来不太用户友好。显然，它的主要受众是测试守护进程的容器开发人员。但是，由于它最接近实际的 containerd API，因此它可以作为一种很好的探索手段——通过检查可用命令，您可以大致了解 containerd可以做什么和不可以做什么。

ctr也非常适合学习的能力低级别[OCI]容器的运行时间，因为ctr + containerd是更接近实际的容器比docker + dockerd。

令人惊讶的是，containerd不提供开箱即用的镜像构建支持。然而，containerd 本身经常被更高级别的工具用来构建镜像。

不使用ctr构建镜像，您可以导入用docker build或其他oci兼容软件构建的现有镜像:

docker build -t my-app .
docker save -o my-app.tar my-app

ctr images import my-app.tar

有了ctr，你也可以挂载镜像

$ mkdir /tmp/nginx
$ ctr images mount docker.io/nginx:latest /tmp/nginx

$ ls -l /tmp/httpbin/
total 80
drwxr-xr-x 2 root root 4096 Oct 18  2018 bin
drwxr-xr-x 2 root root 4096 Apr 24  2018 boot
drwxr-xr-x 4 root root 4096 Oct 18  2018 dev
drwxr-xr-x 1 root root 4096 Oct 24  2018 etc
drwxr-xr-x 2 root root 4096 Apr 24  2018 home
drwxr-xr-x 3 root root 4096 Oct 24  2018 nginx
...

$ ctr images unmount /tmp/nginx

有了一个本地镜像，你可以通过ctr运行<image-ref> <container-id>来运行一个容器。例如: 

$ ctr run --rm -t docker.io/library/debian:latest cont1

 注意，与友好的docker运行生成唯一的容器ID不同，使用ctr，你必须自己提供唯一的容器ID。ctr运行命令也只支持一些常见的docker运行标志:——env， -t，——tty， -d，——detach，——rm，等等。但是没有端口发布或使用——restart=总是开箱即用的自动容器重新启动。

ctrl运行命令实际上是ctrl容器创建+ ctrl任务启动的方式:

$ ctr container create -t docker.io/library/nginx:latest nginx
$ ctr container ls
CONTAINER    IMAGE                              RUNTIME
nginx      docker.io/library/nginx:latest     io.containerd.runc.v2

$ ctr task ls
TASK    PID    STATUS        # Empty!

$ ctr task start -d nginx  # -d for --detach
$ ctr task list
TASK     PID      STATUS
nginx  10074    RUNNING

我喜欢这种容器和任务子命令的分离，因为它反映了经常被遗忘的OCI容器的本质。尽管人们普遍认为容器不是进程——对于进程来说，容器是隔离的和受限制的执行环境。

使用ctr任务连接，你可以重新连接到一个在容器中运行的现有任务的stdio流:

 ctr task attach nginx

 ctr task exec -t --exec-id $RANDOM nginx bash

在移除一个容器之前，它的所有任务必须停止:

 ctr task rm -f nginx

最后，要删除容器，运行:

 ctr container rm nginx

如何使用容器与nerdctl

Nerdctl是一个相对较新的containerd命令行客户端。与ctr不同，nerdctl的目标是用户友好和docker兼容。在某种程度上，nerdctl + containerd可以无缝地替代docker + dockerd。然而，这似乎不是项目的目标:

nerdctl的目标是促进试验Docker中没有的最前沿的容器特性。这些特性包括但不限于lazy-pulling(stargz)和镜像加密(ocicrypt)。这些功能预计最终也会在Docker中实现，然而，这可能需要几个月，甚至几年的时间，因为Docker目前只设计使用了容器子系统的一小部分。重构Docker以使用整个容器是可能的，但并不简单。所以我们[NTT]决定创建一个完全使用container的CLI，但我们不打算用Docker来完成。我们一直在为Docker/Moby以及容器做出贡献，并将继续这样做。

从基本用法的角度来看，与ctr相比，nerdctl支持:

• 使用nerdctl构建镜像
• 容器网络管理
• Docker Compose与nerdctl Compose up
• 最酷的部分是nerdctl试图提供docker(和podman)命令行UX相同的功能。所以，如果你熟悉docker(或podman) CLI，你就已经熟悉nerdctl了。

如何使用容器与crictl

crictl是一个命令行客户端，用于[Kubernetes] cri兼容的容器运行时。

引入 Kubernetes 容器运行时接口 (CRI)以使 Kubernetes 容器运行时不可知。Kubernetes节点代理kubelet实现了 CRI客户端 API，可以使用任何实现 CRI 服务器 API的容器运行时来管理其节点上的容器和 Pod。

从1.1版开始，containerd就自带了一个内置的CRI插件。因此，containerd是一个与cri兼容的容器运行时。因此，它可以与critl一起使用。

创建crictl是为了检查和调试Kubernetes节点上的容器运行时和应用程序。支持以下操作:

attach: Attach to a running container
create: Create a new container
exec: Run a command in a running container
version: Display runtime version information
images, image, img: List images
inspect: Display the status of one or more containers
inspecti: Return the status of one or more images
imagefsinfo: Return image filesystem info
inspectp: Display the status of one or more pods
logs: Fetch the logs of a container
port-forward: Forward local port to a pod
ps: List containers
pull: Pull an image from a registry
run: Run a new container inside a sandbox
runp: Run a new pod
rm: Remove one or more containers
rmi: Remove one or more images
rmp: Remove one or more pods
pods: List pods
start: Start one or more created containers
info: Display information of the container runtime
stop: Stop one or more running containers
stopp: Stop one or more running pods
update: Update one or more running containers
config: Get and set crictl client configuration options
stats: List container(s) resource usage statistics

这里有趣的部分是，通过crictl + containerdbundle，人们可以了解pod是如何实际实现的。

有关如何crictl与 containerd一起使用的更多信息，请查看此文档（containerd 项目的一部分）。

Why and How to Use containerd From Command Line

在 containerd v1.5 中为ctr 客户端（管理员/开发人员的容器工具）、容器镜像服务客户端和 CRI 客户端（例如kubectl和crictl).

ctr,crictl,nerdctl：都是containerd客户端，但containerd的客户端工具ctr及crictl使用不方便，containerd 全新的一个客户端工具nerdctl  Releases · containerd/nerdctl · GitHub  #文章 Title | #网站标题 

crictl,nerdctl：可以使用它来检查和调试 k8s 节点上的容器运行时和应用程序。

接下来就是crictl的的常见命令，其中能完全替代docker命令的参照下列表格

操作	crictl	docker
查看运行容器	crictl ps	docker ps
查看镜像	crictl images	docker images
查看容器日志	crictl logs	docker logs
登陆容器控制台	crictl exec	docker exec
pull镜像	crictl pull	docker pull
容器启动/停止	crictl start/stop	docker start/stop
容器资源情况	crictl stats	docker stats

可以看到crictl对容器生命周期的管理基本已经覆盖，不过在crictl我们不能完成操作也比较多，比如对镜像的管理就不属于它的管理范围。这部分还得依靠ctr来实现，操作方式同样可以参照下表

操作	ctr	docker
查看镜像	ctr images ls	docker images
镜像导入/导出	ctr images import/exporter	docker load/save
镜像拉取/推送	ctr images pull/push	docker pull/push
镜像tag	ctr images tag	docker tag

私有仓库的使用

1. crictl使用：k8s/k3s集群在node通过kubelet节点下载镜像也是用的这个而不是ctr

1.1 第一种情况私有仓库https的且自签名

1.1.1)生成配置文件，然后修改文件

[root@jettoloader containerd]# containerd config default > /etc/containerd/config.toml

[root@jettoloader ~]# cat /etc/containerd/config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "k8s.gcr.io/pause:3.5"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5043"]
          endpoint = ["https://nginx.cclinux.cn:5043"]
        #[plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5000"]
        #  endpoint = ["http://nginx.cclinux.cn:5000"]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
        [plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
 	  #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"
        #[plugins."io.containerd.grpc.v1.cri".registry.auths."http://nginx.cclinux.cn:5000"]
 	#  #auth = "XXXXXXXXXXX"
        #  username = "admin"
        #  password = "123456aA"
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".tls]
          ca_file = "/opt/auth/ca.crt"
  	  #cert_file = "/opt/ssl/nginx.cclinux.cn.crt"
  	  #key_file = "/opt/ssl/nginx.cclinux.cn.key"
          #insecure_skip_verify = true
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
        #  username = "admin"
        #  password = "123456aA"
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5000".auth]
        #  username = "admin"
        #  password = "123456aA"
   

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

1.1.2 需改的地方：

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5043"]
          endpoint = ["https://nginx.cclinux.cn:5043"]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
                [plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
 	      #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".tls]
          ca_file = "/opt/auth/ca.crt"   #根证书 用来生成cert_file 和key_file  
          #cert_file = "/opt/ssl/harbor.jettech.com.crt"  #harbor的crt
          #key_file = "/opt/ssl/harbor.jettech.com.key"  #harbor的key
          #insecure_skip_verify = true  #两者选其一
        [plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
          username = "admin"
          password = "123456aA"

经过测试：         

         #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"

这段认证配置有两个地方可以配置：

1） 在[plugins."io.containerd.grpc.v1.cri".registry.auths]下面

      [plugins."io.containerd.grpc.v1.cri".registry.auths]
          [plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
            username = "admin"
            password = "123456aA"

 2） 在[plugins."io.containerd.grpc.v1.cri".registry.configs]下面

     [plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
          username = "admin"
          password = "123456aA"

insecure_skip_verify = true 意为跳过证书认证。 

然后测试：

[root@jettoloader containerd]# crictl pull  nginx.cclinux.cn:5043/library/registry:latest
Image is up to date for sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2

注意：ctr是不可以用这个配置文件的，ctr 不使用 CRI，因此它不读取plugins."io.containerd.grpc.v1.cri"配置issues 6285  issues 5407  这个很重要，困惑我很久。

[root@jettoloader containerd]# ctr images pull nginx.cclinux.cn:5043/library/registry:latest
INFO[0000] trying next host                              error="failed to do request: Head \"https://nginx.cclinux.cn:5043/v2/library/registry/manifests/latest\": x509: certificate signed by unknown authority" host="nginx.cclinux.cn:5043"
ctr: failed to resolve reference "nginx.cclinux.cn:5043/library/registry:latest": failed to do request: Head "https://nginx.cclinux.cn:5043/v2/library/registry/manifests/latest": x509: certificate signed by unknown authorit

1.1.3 如果想用ctr下载上面的镜像方法有两个：

1）参数化：

[root@jettoloader containerd]#  ctr  images pull --user admin:123456aA  --tlscacert=/opt/auth/ca.crt   nginx.cclinux.cn:5043/library/registry:latest
nginx.cclinux.cn:5043/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.3 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 33.847854ms

我应该在其他地方指定这个配置吗？我不想-u user:password --tlscacert每次必须使用 ctr pull 

但是cri没有配置可以配置ca证书？，也没有配置可以设置skip-verify，官方档里说可以配置: 参考 2）

2） hosts方式：          hosts.toml   详细参考章节2        

1.2 第二种情况 私有仓库http方式的

1.2.1)生成配置文件，然后修改文件

[root@jettoloader ~]# cat /etc/containerd/config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "k8s.gcr.io/pause:3.5"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        #[plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5043"]
        #  endpoint = ["https://nginx.cclinux.cn:5043"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5000"]
          endpoint = ["http://nginx.cclinux.cn:5000"]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
        #[plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
 	#  #auth = "XXXXXXXXXXX"
        #  username = "admin"
        #  password = "123456aA"
        [plugins."io.containerd.grpc.v1.cri".registry.auths."http://nginx.cclinux.cn:5000"]
 	  #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"
           
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".tls]
        #  ca_file = "/opt/auth/ca.crt"
      	#  #cert_file = "/opt/auth/nginx.cclinux.cn.crt"
       	#  #key_file = "/opt/auth/nginx.cclinux.cn.key"
        #  #insecure_skip_verify = true
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
        #  username = "admin"
        #  password = "123456aA"
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5000".auth]
        #  username = "admin"
        #  password = "123456aA"
   

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

1.2.2 需改的地方：

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        #[plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5043"]
        #  endpoint = ["https://nginx.cclinux.cn:5043"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."nginx.cclinux.cn:5000"]
          endpoint = ["http://nginx.cclinux.cn:5000"]
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
        #[plugins."io.containerd.grpc.v1.cri".registry.auths."https://nginx.cclinux.cn:5043"]
 	#  #auth = "XXXXXXXXXXX"
        #  username = "admin"
        #  password = "123456aA"
        [plugins."io.containerd.grpc.v1.cri".registry.auths."http://nginx.cclinux.cn:5000"]
 	  #auth = "XXXXXXXXXXX"
          username = "admin"
          password = "123456aA"
           
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".tls]
        #  ca_file = "/opt/auth/ca.crt"
      	#  #cert_file = "/opt/auth/nginx.cclinux.cn.crt"
       	#  #key_file = "/opt/auth/nginx.cclinux.cn.key"
        #  #insecure_skip_verify = true
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5043".auth]
        #  username = "admin"
        #  password = "123456aA"
        #[plugins."io.containerd.grpc.v1.cri".registry.configs."nginx.cclinux.cn:5000".auth]
        #  username = "admin"
        #  password = "123456aA"

然后测试：

[root@jettoloader ~]# crictl pull  nginx.cclinux.cn:5000/library/registry:latest
Image is up to date for sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2

注意：ctr是不可以用这个配置文件的，ctr 不使用 CRI，因此它不读取plugins."io.containerd.grpc.v1.cri"配置issues 6285  issues 5407  这个很重要，困惑我很久。

[root@jettoloader ~]# ctr images pull nginx.cclinux.cn:5000/library/registry:latest
INFO[0000] trying next host                              error="failed to do request: Head \"https://nginx.cclinux.cn:5000/v2/library/registry/manifests/latest\": http: server gave HTTP response to HTTPS client" host="nginx.cclinux.cn:5000"
ctr: failed to resolve reference "nginx.cclinux.cn:5000/library/registry:latest": failed to do request: Head "https://nginx.cclinux.cn:5000/v2/library/registry/manifests/latest": http: server gave HTTP response to HTTPS client

1.2.3 如果想用ctr下载上面的镜像方法有两个：

1）参数化：

[root@jettoloader ~]#  ctr --address=/run/containerd/containerd.sock images pull --skip-verify  --user admin:123456aA  --plain-http  nginx.cclinux.cn:5000/library/registry:latest
nginx.cclinux.cn:5000/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.3 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 74.140227ms

我应该在其他地方指定这个配置吗？我不想-u user:password 每次必须使用 ctr pull 

--user user:password

--plain-http:allow connections using plain HTTP

2） hosts方式：          hosts.toml   详细参考 2. ctr 工具介绍

2. ctr 工具介绍

      github hosts.toml

     https://github.com/containerd/containerd/blob/main/docs/hosts.md

指定配置目录

将主机命名空间配置与 CTR 结合使用

通过ctr使用--hosts-dir选项来拉取容器映像时，告诉ctr 查找并使用位于指定路径中的主机配置文件：

ctr images pull --hosts-dir "/etc/containerd/certs.d" myregistry.io:5000/image_name:tag

CRI

用于指定 registry.mirrors 和 registry.configs 的旧 CRI 配置模式已弃用。您现在应该将注册表指向文件config_path所在的路径 hosts.toml。

修改您的config.toml（默认位置/etc/containerd/config.toml：）如下：

Modify your config.toml (default location: /etc/containerd/config.toml) as follows:

[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

在你的 config.toml 中为 containerd设置 config_path = "/etc/containerd/certs.d"。在配置路径中创建一个目录树，该目录树包含host-namespace/hosts.toml一个表示要配置的主机命名空间的目录。然后在 中添加一个hosts.toml文件主机来配置主机命名空间。它应该如下所示：

完成路径：/etc/containerd/certs.d/<host-namespace>/hosts.toml

支持 Docker 的证书文件模式

如果主机目录中不存在 hosts.toml 配置，它将回退到根据Docker 的证书文件模式检查证书文件 （“.crt”文件用于 CA 证书，“.cert”/“.key”文件用于客户端证书）。

 2.1)第一种情况https方式

2.1.1.1) 忽略证书skip_verify = true

[root@jettoloader ~]#  mkdir -p /etc/containerd/certs.d/nginx.cclinux.cn:5043


[root@jettoloader nginx.cclinux.cn:5043]# cat hosts.toml 
#server = "https://docker.io"
#server = "http://nginx.cclinux.cn:5000"
#server = "https://nginx.cclinux.cn:5043"

#[host."http://nginx.cclinux.cn:5000"]
[host."https://nginx.cclinux.cn:5043"]
  capabilities = ["pull", "resolve","push"]
  skip_verify = true

2.1.1.2)不需重启测试注意不用重启systemctl restart containerd。pull和push

[root@jettoloader ~]#  ctr images pull   --user admin:123456aA  --hosts-dir "/etc/containerd/certs.d"  nginx.cclinux.cn:5043/library/registry:latest
nginx.cclinux.cn:5043/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 21.297093ms 

ctr 不使用 CRI 配置。因此，您在 config.toml 的 CRI 部分中指定的身份验证不会被 ctr 客户端读取。

如果需要auth[用户名和密码] 需要加参数：-u user:password 

如果需要证书需要指定： --hosts-dir "/etc/containerd/certs.d"

2.1.2.1) 不忽略证书skip_verify = true，此时需要制定ca证书文件

[root@jettoloader nginx.cclinux.cn:5043]# cat hosts.toml 
#server = "https://docker.io"
#server = "http://nginx.cclinux.cn:5000"

[host."https://nginx.cclinux.cn:5043"]
  capabilities = ["pull", "resolve","push"]
  #skip_verify = true
  #ca = "ca.crt"   #相对路径
  #ca = "/opt/auth/ca.crt"  #绝对路径
  #ca = ["/opt/auth/ca.crt"]
  ca = ["ca.crt"]
  client = [["/opt/auth/nginx.cclinux.cn.crt", "/opt/auth/nginx.cclinux.cn.key"]]
  #client = [["/opt/auth/nginx.cclinux.cn.crt", "/opt/auth/nginx.cclinux.cn.key"],["",""]]

注意：

此处的相对路径是：

[root@jettoloader nginx.cclinux.cn:5043]# pwd
/etc/containerd/certs.d/nginx.cclinux.cn:5043
[root@jettoloader nginx.cclinux.cn:5043]# ls
ca.crt  hosts.toml

此处的绝对路径是：

[root@jettoloader nginx.cclinux.cn:5043]# ls /opt/auth/
ca.crt  nginx.cclinux.cn.crt  nginx.cclinux.cn.key  nginx.conf  nginx.htpasswd

2.1.2.2)不需重启测试注意不用重启systemctl restart containerd。pull和push  

[root@jettoloader ~]#  ctr images pull   --user admin:123456aA  --hosts-dir "/etc/containerd/certs.d"  nginx.cclinux.cn:5043/library/registry:latest
nginx.cclinux.cn:5043/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 36.400849ms

 2.2)第一种情况http方式

2.2.1)创建镜像仓库域名或ip目录,然后添加hosts.toml文件，

[root@jettoloader ~]# mkdir -p /etc/containerd/certs.d/nginx.cclinux.cn:5000
[root@jettoloader nginx.cclinux.cn:5000]# pwd
/etc/containerd/certs.d/nginx.cclinux.cn:5000
[root@jettoloader nginx.cclinux.cn:5000]# cat hosts.toml 
#server = "https://docker.io"
server = "http://nginx.cclinux.cn:5000"

[host."http://nginx.cclinux.cn:5000"]
  capabilities = ["pull", "resolve","push"]
  skip_verify = true

2.2.2) 测试pull、push，【无认证就是无用户名和密码】 注意不用重启systemctl restart containerd

[root@jettoloader ~]#  ctr images pull --hosts-dir "/etc/containerd/certs.d"  nginx.cclinux.cn:5000/library/registry:latest
nginx.cclinux.cn:5000/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 20.402958ms 

 2.2.3) 测试pull、push，【有认证就是用户名和密码】 注意不用重启systemctl restart containerd

[root@jettoloader ~]#  ctr images pull   --user admin:123456aA  --hosts-dir "/etc/containerd/certs.d"  nginx.cclinux.cn:5000/library/registry:latest
nginx.cclinux.cn:5000/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.5 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4...
done: 52.012024ms

                                                                        Q/A 环节

Q:我应该在其他地方指定这个配置吗？我不想-u user:password每次必须使用 ctr pull

A:目前，我们没有其他方法可以为该ctr工具指定身份验证

Crictl can pull images but ctr gives unauthorized, private registry with basic auth · Issue #5586 · containerd/containerd · GitHub​

作为一个团队，尚未就主机身份验证配置改进的路径做出决定..注意：

https://github.com/containerd/containerd/blob/main/docs/hosts.md

containerd/config.md at main · containerd/containerd · GitHub

该 文档涵盖了最近为支持除主机身份验证之外的所有主机配置所做的更改。

nerdctl (another containerd CLI) supports .docker/config.json for authentication

GitHub - containerd/nerdctl: contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...

​看 第三章节nerdctl工具

3. nerdctl 工具介绍

3.1 nerdctl，介绍

            2021年8月31日，Docker公司更改了收费策略，简而言之就是对个人和小公司不收费，对大公司收费,是时候开始试一下 containerd + lima[ 这对mac系统的]这个开源、免费的Docker替代品

           这可能已经是Docker公司第二次提醒我们 Docker 不等于容器了

          就像Kubernetes需要一个中立的容器标准，所以OCI出现了；容器客户端也需要一个中立的标准和参考实现，目前一下是一个比较合适的方案，值得关注这两个项目的长期发展。

           mac:containerd+nerdctl+lima

           linux:containerd+nerdctl

            Kubernetes 虽然制定了容器运行时接口（CRI）标准，但早期能用的容器运行时只有 Docker，而 Docker 又不适配这个标准，于是给 Docker 开了后门，花了大量的精力去适配它。后来有了更多的容器运行时可以选择后，Kubernetes 就不得不重新考量要不要继续适配 Docker 了，因为每次更新 Kubelet 都要考虑与 Docker 的适配问题。
          标准这个东西就是这样，我定好标准，你兼容了就一起玩，不兼容就拜拜，它就像两个人在一起的底线，你可以重，你可以丑，你也可以不完善，但是你不兼容标准就真的不能一起玩了，于是 Kubernetes 就把 Docker 踢出了群聊。

          最终 Kubernetes 选择了 Containerd，时至今日 Containerd 已经变成一个工业级的容器运行时了，它足够简单、健壮，可移植性也很强。

          现有 CLI 的不足

          但是在containerd的目标聚焦在和其他系统集成，所以它的默认命令行工具（crictl）也不是很好用，和docker也不兼容。

          虽然 Docker 能干的事情，现在 Containerd 都能干，但 Containerd 还有一个非常明显的缺陷：CLI 不够友好。它无法像 Docker 和 Podman 一样通过一条简单的命令启动一个容器，它的两个 CLI 工具 ctr 和 crictl 都无法实现这么一件非常简单的需求，而这个需求是大多数人都需要的，我总不能为了在本地测试容器而专门部署一个 Kubernetes 集群吧。可以认为ctr 和 crictl两个工具是针对k8s/k3s集群而生的。

nerdctl不仅与docker兼容，而且还支持了更多的功能：

• 支持containerd的命名空间查看，nerdctl不仅可以管理Docker容器，也可以直接管理本地的的Kubernetes pod
• 支持将Docker Image Manifest镜像转换为OCI镜像、estargz镜像
• 支持OCIcrypt（镜像加密）

什么是lima？

containerd要在macOS上使用，需要安装虚拟机、然后在虚拟机配置containerd，这个过程很浪费时间。

所以，lima项目简化了这个过程。

lima会启动一个虚拟机，然后在虚拟机中安装containerd，并自动配置虚拟机的文件共享，网络等。

lima相比Docker for Desktop Mac，也有一些优势：

• 内置lazy pulling支持（基于stargz）
• 额外支持ARM on Intel，Intel on ARM（基于QEMU）
• 可以自定义虚拟机发行版（默认是Ubuntu）

  3.1.1)nerdctl  (containerd CLI)，支持 .docker/config.json 进行身份验证   nerdctl

nerdctl 是containerd 的cli客户端，与docker cli大部分兼容。
nerdctl是containerd的非核心子项目

3.1.2)ctr 的设计对人类不太友好，例如缺少以下这些和 Docker 类似的功能：

• docker run -p <PORT>
• docker run --restart=always
• 通过凭证文件 ~/.docker/config.json 来拉取镜像
• docker logs

3.1.3)除此之外还有一个 CLI 工具叫 crictl，和 ctr 一样不太友好。

3.1.4)为了解决这个痛点，Containerd 官方推出了一个新的 CLI 叫 nerdctl。nerdctl 的使用体验和 docker 一样顺滑，例如：

erdctl run -d -p 8080:80 --name=nginx --restart=always nginx

3.1.5)nerdctl 只是 docker 的复制品？

       nerdctl 的目标并不是单纯地复制 docker 的功能，它还实现了很多 docker 不具备的功能，例如延迟拉取镜像（lazy-pulling）、镜像加密（imgcrypt）等。具体看nerdctl

延迟拉取镜像功能可以参考这篇文章：Containerd 使用 Stargz Snapshotter 延迟拉取镜像

虽然这些功能预计最终也会在 Docker 中实现，可能需要几个月甚至几年的时间，因为 Docker 目前的设计只使用一小部分 Containerd 子系统。将来 Docker 有可能重构代码以使用完整的 Containerd，但目前还没看到什么实质性进展。所以 Containerd 社区决定创建一个新的 CLI 来更友好地使用 Containerd。

3.2 安装

下载地址：Releases · containerd/nerdctl,中下载最新的可执行文件，每一个版本都有两种可用的发行版

• 精简 (nerdctl-<version>-linux-amd64.tar.gz): 只包含nerdctl
• 完整 (nerdctl-full-<version>-linux-amd64.tar.gz): 包含 containerd, runc, and CNI等依赖

如果你已经安装了 Containerd，只需要选择前一个发行版，否则就选择完整版。

精简Minimal

将归档文件解压到 /usr/local/bin 或 ~/bin

[root@jettoloader nerdctl]# tar Cxzvvf /usr/local/bin nerdctl-0.16.0-linux-amd64.tar.gz
[root@jettoloader nerdctl]# ls -l /usr/local/bin/
-rwxr-xr-x 2 root root        52 1月  31 12:26 nerdctl
-rwxr-xr-x 2 root root        52 1月  31 12:26 containerd-rootless-setuptool.sh
-rwxr-xr-x 2 root root        52 1月  31 12:26 containerd-rootless.sh

完整Full

解压文件到/usr/local/

[root@jettoloader nerdctl]# tar -xf nerdctl-full-0.16.0-linux-amd64.tar.gz -C /usr/local/ 

启动服务

[root@jettoloader nerdctl]# ls /usr/local/lib//systemd/system/
buildkit.service  containerd.service  stargz-snapshotter.service

[root@jettoloader nerdctl]# cp /usr/local/lib/systemd/system/*.service /etc/systemd/system/


[root@jettoloader nerdctl]# systemctl enable  buildkit containerd --now
[root@jettoloader nerdctl]# systemctl status buildkit containerd
● buildkit.service
   Loaded: loaded (/etc/systemd/system/buildkit.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2022-01-31 12:34:12 CST; 12min ago
 Main PID: 22978 (buildkitd)
   CGroup: /system.slice/buildkit.service
           └─22978 /usr/local/bin/buildkitd

1月 31 12:34:12 jettoloader systemd[1]: Starting buildkit.service...
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="auto snapshotter: using overlayfs"
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=warning msg="using host network as the default"
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="found worker \"89o6fwn2m85nergdyc3vxh9ri\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproje...md64 linux/386]"
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=warning msg="using host network as the default"
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="found worker \"xsw05ykrexuxsb51r9ky8ozd1\", labels=map[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproj...
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="found 2 workers, default=\"89o6fwn2m85nergdyc3vxh9ri\""
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=warning msg="currently, only the default worker can be used."
1月 31 12:34:12 jettoloader systemd[1]: Started buildkit.service.
1月 31 12:34:12 jettoloader buildkitd[22978]: time="2022-01-31T12:34:12+08:00" level=info msg="running server on /run/buildkit/buildkitd.sock"

● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2022-01-31 12:36:52 CST; 10min ago
     Docs: https://containerd.io
 Main PID: 29686 (containerd)
   CGroup: /system.slice/containerd.service
           └─29686 /usr/local/bin/containerd

1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055182710+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055262035+08:00" level=info msg="Start subscribing containerd event"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055414180+08:00" level=info msg="Start recovering state"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055452341+08:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055493510+08:00" level=info msg=serving... address=/run/containerd/containerd.sock
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.055557796+08:00" level=info msg="containerd successfully booted in 0.044853s"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.080950138+08:00" level=info msg="Start event monitor"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.081005740+08:00" level=info msg="Start snapshots syncer"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.081021647+08:00" level=info msg="Start cni network conf syncer"
1月 31 12:36:52 jettoloader containerd[29686]: time="2022-01-31T12:36:52.081030575+08:00" level=info msg="Start streaming server"
Hint: Some lines were ellipsized, use -l to show in full.
[root@jettoloader nerdctl]# 

对于非root用户以上命令需要加上sudo，而且还要额外运行

1	containerd-rootless-setuptool.sh installz

注意，这个rootless-setuptool目前不支持centOS 7及其以下。

3.3 设置nerdctl子命令可以使用tab键

编辑文件
[root@jettoloader nerdctl]# vim /etc/profile
source <(nerdctl completion bash)

#时期生效
[root@jettoloader nerdctl]#  source /etc/profile

3.4 修改containerd配置文件并配置加速器

[root@jettoloader nerdctl]# cat /etc/containerd/config.toml 
disabled_plugins = ["restart"]
[plugins]
   [plugins.cri.registry.mirrors."docker.io"]
     endpoint = ["https://frz7i079.mirror.aliyuncs.com"]
   #[plugins.cri.registry.mirrors."nginx.cclinux.cn:5000"]
   #  endpoint = ["http://nginx.cclinux.cn:5000"]



重启containerd

 [root@jettoloader nerdctl]# systemctl restart containerd

3.5  基本使用使用默认CNI网络（10.4.0.0/24）启动一个容器

测试运行

[root@jettoloader nerdctl]# nerdctl run -it --rm alpine
/ # ls
bin    dev    etc    home   lib    media  mnt    opt    proc   root   run    sbin   srv    sys    tmp    usr    var


/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if108: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether c2:5e:b8:43:b4:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.4.0.3/24 brd 10.4.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::c05e:b8ff:fe43:b4f4/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever

3.6 构建镜像,nerdctl build：从 Dockerfile 构建镜像

镜像构建是平时我们非常重要的一个需求，我们知道 ctr 并没有构建镜像的命令，而现在我们又不使用 Docker 了，那么如何进行镜像构建了，幸运的是 nerdctl 就提供了 nerdctl build 这样的镜像构建命令。

3.6.1) 比如现在我们定制一个 nginx 镜像，新建一个如下所示的 Dockerfile 文件：

[root@jettoloader ~]# mkdir demo 
[root@jettoloader ~]# cat demo/Dockerfile 
FROM nginx:alpine
RUN echo 'Hello Nerdctl From Containerd' > /usr/share/nginx/html/index.html

然后在文件所在目录执行镜像构建命令：

[root@jettoloader ~]# nerdctl build -t nginx:nerctl -f demo/Dockerfile demo
ERRO[0000] error: failed to list workers: Unavailable: connection error: desc = "transport: error while dialing: dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory" 
FATA[0000] `buildctl` needs to be installed and `buildkitd` needs to be running, see https://github.com/moby/buildkit: exit status 1

FATA[0000] `buildctl` needs to be installed and `buildkitd` needs to be running, see https://github.com/moby/buildkit: exec: "buildctl": executable file not found in $PATH

 注意：也可以加上这个–no-cache选项

可以看到有一个错误提示，需要我们安装 buildctl 并运行 buildkitd，这是因为 nerdctl build 需要依赖 buildkit 工具。

buildkit 项目也是 Docker 公司开源的一个构建工具包，支持 OCI 标准的镜像构建。它主要包含以下部分:

服务端 buildkitd：当前支持 runc 和 containerd 作为 worker，默认是 runc，我们这里使用 containerd
客户端 buildctl：负责解析 Dockerfile，并向服务端 buildkitd 发出构建请求
buildkit 是典型的 C/S 架构，客户端和服务端是可以不在一台服务器上，而 nerdctl 在构建镜像的.

https://github.com/moby/buildkit

nerdctl 也可以和 buildkit 结合使用来构建容器镜像，需要先下载 buildkit 的可执行文件：

下载
[root@jettoloader nerdctl]#  wget https://github.com/moby/buildkit/releases/download/v0.9.3/buildkit-v0.9.3.linux-amd64.tar.gz

[root@containerd ~]#tar tf buildkit-v0.9.1.linux-amd64.tar.gz
bin/
bin/buildctl              #客户端
bin/buildkit-qemu-aarch64
bin/buildkit-qemu-arm
bin/buildkit-qemu-i386
bin/buildkit-qemu-mips64
bin/buildkit-qemu-mips64el
bin/buildkit-qemu-ppc64le
bin/buildkit-qemu-riscv64
bin/buildkit-qemu-s390x
bin/buildkit-runc
bin/buildkitd              #服务端


解压到PATH
[root@jettoloader nerdctl]#  tar -C /usr/local/ -zxvf buildkit-v0.9.3.linux-amd64.tar.gz


服务端:服务文件
[root@jettoloader nerdctl]#  /etc/systemd/system/buildkit.service
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit

[Service]
ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true

[Install]
WantedBy=multi-user.target


#启动服务
[root@jettoloader nerdctl]#  systemctl enable --now buildkit.service

#看下日志
[root@jettoloader nerdctl]#  journalctl -u buildkit

#案例下载：
[root@jettoloader nerdctl]#  git clone --depth=1 https://github.com.cnpmjs.org/kubesphere/kubesphere.git

#进入仓库目录，编译二进制文件：
[root@jettoloader nerdctl]# cd kubesphere
[root@jettoloader nerdctl]# make ks-apiserver


#将二进制文件拷贝到 Dockerfile 目录：
[root@jettoloader nerdctl]#  cp bin/cmd/ks-apiserver build/ks-apiserver


#进入 Dockerfile 目录，修改 Dockerfile：

# Copyright 2020 The KubeSphere Authors. All rights reserved.
# Use of this source code is governed by an Apache license
# that can be found in the LICENSE file.
FROM alpine:3.11

ARG HELM_VERSION=v3.5.2

RUN apk add --no-cache ca-certificates
# install helm
RUN wget https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz && \
    tar xvf helm-${HELM_VERSION}-linux-amd64.tar.gz && \
    rm helm-${HELM_VERSION}-linux-amd64.tar.gz && \
    mv linux-amd64/helm /usr/bin/ && \
    rm -rf linux-amd64
# To speed up building process, we copy binary directly from make
# result instead of building it again, so make sure you run the
# following command first before building docker image
#   make ks-apiserver
#
COPY  ks-apiserver /usr/local/bin/

EXPOSE 9090
CMD ["sh"]







#构建镜像：
[root@jettoloader nerdctl]# cd build/ks-apiserver
[root@jettoloader nerdctl]# nerdctl build -t ks-apiserver .

现在我们再来重新构建镜像：

[root@jettoloader ~]# nerdctl build -t harbor.jettech.com/nginx:nerctl -f demo/Dockerfile demo
[+] Building 4.4s (6/6) FINISHED                                                                                                                                                                                                                                              
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                     0.1s
 => => transferring dockerfile: 131B                                                                                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                        0.1s
 => => transferring context: 2B                                                                                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/nginx:alpine                                                                                                                                                                                                          3.2s
 => [1/2] FROM docker.io/library/nginx:alpine@sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3                                                                                                                                                    0.1s
 => => resolve docker.io/library/nginx:alpine@sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3                                                                                                                                                    0.1s
 => CACHED [2/2] RUN echo 'Hello Nerdctl From Containerd' > /usr/share/nginx/html/index.html                                                                                                                                                                             0.0s
 => exporting to oci image format                                                                                                                                                                                                                                        0.8s
 => => exporting layers                                                                                                                                                                                                                                                  0.0s
 => => exporting manifest sha256:0f36e8f9982c9f3157256f8db4272bef1895476be588216a9a5dd512abdbbd4d                                                                                                                                                                        0.0s
 => => exporting config sha256:94dacf915be33a342b7b40607bc476b07a39e608605b6182c835b54a2747ad58                                                                                                                                                                          0.0s
 => => sending tarball                                                                                                                                                                                                                                                   0.6s
unpacking harbor.jettech.com/nginx:nerctl (sha256:0f36e8f9982c9f3157256f8db4272bef1895476be588216a9a5dd512abdbbd4d)...done
[root@jettoloader ~]# nerdctl images
REPOSITORY                                TAG       IMAGE ID        CREATED           PLATFORM       SIZE
nginx                                     latest    2834dc507516    2 days ago        linux/amd64    146.2 MiB
harbor.jettech.com/nginx                  nerctl    0f36e8f9982c    6 seconds ago

这样我们就使用 nerdctl + buildkitd 轻松完成了容器镜像的构建

C/S架构

nerdctl或buildctl  【客户端】  +buildkitd【服务端】

docker cli【客户端】   + dockerd【服务端】

关于 nerdctl 的更多用法，可以参考官方仓库的 README

3.7 containerd网络，它对接的是cni

3.8 nerdctl 私有仓库

第一种情况：http方式

3.8.1）配置文件

一下两个哪个都可以
[root@jettoloader containerd]# mkdir -p /etc/docker/certs.d/nginx.cclinux.cn:5000
[root@jettoloader containerd]# mkdir -p /etc/containerd/certs.d/nginx.cclinux.cn:5000


[root@jettoloader containerd]# cat certs.d/nginx.cclinux.cn\:5000/hosts.toml 
server = "https://docker.io"

[host."http://nginx.cclinux.cn:5000"]
  capabilities = ["pull", "resolve","push"]
  #skip_verify = true
  #ca = "ca.crt"   #相对路径
  #ca = "/opt/auth/ca.crt"  #绝对路径
  #ca = ["/opt/auth/ca.crt"]
  #ca = ["ca.crt"]
  #client = [["/opt/auth/nginx.cclinux.cn.crt", "/opt/auth/nginx.cclinux.cn.key"]]

测试

[root@jettoloader containerd]# nerdctl pull  --all-platforms  nginx.cclinux.cn:5000/library/registry:latest
nginx.cclinux.cn:5000/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.2 s                                                                    total:   0.0 B (0.0 B/s) 

第二种情况：https方式

3.8.2）配置文件

以下两个哪个都可以
[root@jettoloader nginx.cclinux.cn:5043]# mkdir -p  /etc/containerd/certs.d/nginx.cclinux.cn:5043
[root@jettoloader nginx.cclinux.cn:5043]# mkdir -p  /etc/docker/certs.d/nginx.cclinux.cn:5043



[root@jettoloader nginx.cclinux.cn:5043]# cat hosts.toml 
server = "https://docker.io"

[host."https://nginx.cclinux.cn:5043"]
  capabilities = ["pull", "resolve","push"]
  #skip_verify = true
  #ca = "ca.crt"   #相对路径
  #ca = "/opt/auth/ca.crt"  #绝对路径
  #ca = ["/opt/auth/ca.crt"]
  ca = ["ca.crt"]
  #client = [["/opt/auth/nginx.cclinux.cn.crt", "/opt/auth/nginx.cclinux.cn.key"]]
[root@jettoloader nginx.cclinux.cn:5043]# ls
ca.crt  hosts.toml

测试：

登录
[root@jettoloader nginx.cclinux.cn:5043]# echo 123456aA | nerdctl login --username "admin" --password-stdin  nginx.cclinux.cn:5043
ERRO[0000] failed to call tryLoginWithRegHost            error="failed to call rh.Client.Do: Get \"https://nginx.cclinux.cn/v2/\": dial tcp 172.16.10.1:443: connect: connection refused" i=0
ERRO[0001] failed to call tryLoginWithRegHost            error="failed to call rh.Client.Do: Get \"https://nginx.cclinux.cn/v2/\": dial tcp 172.16.10.1:443: connect: connection refused" i=0
Login Succeeded

下载
[root@jettoloader nginx.cclinux.cn:5043]# nerdctl pull  --all-platforms  nginx.cclinux.cn:5043/library/registry:latest
nginx.cclinux.cn:5043/library/registry:latest:                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:36cb5b157911061fb610d8884dc09e0b0300a767a350563cbfd88b4b85324ce4: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3790aef225b922bc97aaba099fe762f7b115aec55a0083824b548a6a1e610719:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0d96da54f60b86a4d869d44b44cfca69d71c4776b81d361bc057d6666ec0d878:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b8604a3fe8543c9e6afc29550de05b36cd162a97aa9b2833864ea8a5be11f3e2:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b27040df4a23c90c3837d926f633fb327fb3af9ac4fa5d5bc3520ad578acb10:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e2ead8259a04d39492c25c9548078200c5ec429f628dcf7b7535137954cc2df0:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:79e9f2f55bf5465a02ee6a6170e66005b20c7aa6b115af6fcd04fad706ea651a:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.3 s                                                                    total:   0.0 B (0.0 B/s) 

总结
从行业趋势来看，Docker 已经和 Kubernetes 社区渐行渐远，以 Containerd 为代表的实现了 CRI 接口的容器运行时将会受到 Kubernetes 的青睐。但纯粹使用 Containerd 还是有诸多困扰，比如不方便通过 CLI 来创建管理容器，有了 nerdctl 这个 CLI 工具，就就可以填补 Containerd 易用性的空缺，让你在单机上也能愉快地使用 Containerd。Containerd高级命令行工具