kubelet与runtime的整合

Kubernetes 1.10版本Kubelet调用docker创建容器的调用链如图所示

解析该流程能够深入的了解kubernetes是怎么与docker进行交互的．

CRI

首先介绍CRI, CRI全称为container runtime interface接口，是kubernetes在1.5版本之后引入的概念，为了使得kubernetes更好的扩展以及整合容器运行时（docker,rkt,containerd,cri-o）

CRI的框架如图所示，采用gRPC远程调用的方式，调用容器运行时提供的ｇRPC远程服务，当前现状:

• docker: kubelet中调用CRI shim提供的imageService,ContainerService接口，作为gRPC client，dockershim实现了CRI gRPC Server服务端的服务实现，但是dockershim仍然整合到了kubelet中，作为kubelet默认的CRI　shim实现．
• rkt: 当前仍然不支持CRI,社区存在项目在跟进rkt支持CRI工作．
• containerd,cri-o：支持CRI接口，通过kubelet中remote的方式与kubernetes进行整合．

dockershim

这里主要分析当前的默认方式:采用dockershim的方式与docker进行整合

首先，介绍Kubernetes中关于CRI的接口定义文件见src/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto
主要需要实现两类接口service RuntimeService以及service ImageService．

// Runtime service defines the public APIs for remote container runtimes
service RuntimeService {
    // Version returns the runtime name, runtime version, and runtime API version.
    rpc Version(VersionRequest) returns (VersionResponse) {}

    // RunPodSandbox creates and starts a pod-level sandbox. Runtimes must ensure
    // the sandbox is in the ready state on success.
    rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandboxResponse) {}
    // StopPodSandbox stops any running process that is part of the sandbox and
    // reclaims network resources (e.g., IP addresses) allocated to the sandbox.
    // If there are any running containers in the sandbox, they must be forcibly
    // terminated.
    // This call is idempotent, and must not return an error if all relevant
    // resources have already been reclaimed. kubelet will call StopPodSandbox
    // at least once before calling RemovePodSandbox. It will also attempt to
    // reclaim resources eagerly, as soon as a sandbox is not needed. Hence,
    // multiple StopPodSandbox calls are expected.
    rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSandboxResponse) {}
    // RemovePodSandbox removes the sandbox. If there are any running containers
    // in the sandbox, they must be forcibly terminated and removed.
    // This call is idempotent, and must not return an error if the sandbox has
    // already been removed.
    rpc RemovePodSandbox(RemovePodSandboxRequest) returns (RemovePodSandboxResponse) {}
    // PodSandboxStatus returns the status of the PodSandbox. If the PodSandbox is not
    // present, returns an error.
    rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSandboxStatusResponse) {}
    // ListPodSandbox returns a list of PodSandboxes.
    rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSandboxResponse) {}

    // CreateContainer creates a new container in specified PodSandbox
    rpc CreateContainer(CreateContainerRequest) returns (CreateContainerResponse) {}
    // StartContainer starts the container.
    rpc StartContainer(StartContainerRequest) returns (StartContainerResponse) {}
    // StopContainer stops a running container with a grace period (i.e., timeout).
    // This call is idempotent, and must not return an error if the container has
    // already been stopped.
    // TODO: what must the runtime do after the grace period is reached?
    rpc StopContainer(StopContainerRequest) returns (StopContainerResponse) {}
    // RemoveContainer removes the container. If the container is running, the
    // container must be forcibly removed.
    // This call is idempotent, and must not return an error if the container has
    // already been removed.
    rpc RemoveContainer(RemoveContainerRequest) returns (RemoveContainerResponse) {}
    // ListContainers lists all containers by filters.
    rpc ListContainers(ListContainersRequest) returns (ListContainersResponse) {}
    // ContainerStatus returns status of the container. If the container is not
    // present, returns an error.
    rpc ContainerStatus(ContainerStatusRequest) returns (ContainerStatusResponse) {}
    // UpdateContainerResources updates ContainerConfig of the container.
    rpc UpdateContainerResources(UpdateContainerResourcesRequest) returns (UpdateContainerResourcesResponse) {}
    // ReopenContainerLog asks runtime to reopen the stdout/stderr log file
    // for the container. This is often called after the log file has been
    // rotated. If the container is not running, container runtime can choose
    // to either create a new log file and return nil, or return an error.
    // Once it returns error, new container log file MUST NOT be created.
    rpc ReopenContainerLog(ReopenContainerLogRequest) returns (ReopenContainerLogResponse) {}

    // ExecSync runs a command in a container synchronously.
    rpc ExecSync(ExecSyncRequest) returns (ExecSyncResponse) {}
    // Exec prepares a streaming endpoint to execute a command in the container.
    rpc Exec(ExecRequest) returns (ExecResponse) {}
    // Attach prepares a streaming endpoint to attach to a running container.
    rpc Attach(AttachRequest) returns (AttachResponse) {}
    // PortForward prepares a streaming endpoint to forward ports from a PodSandbox.
    rpc PortForward(PortForwardRequest) returns (PortForwardResponse) {}

    // ContainerStats returns stats of the container. If the container does not
    // exist, the call returns an error.
    rpc ContainerStats(ContainerStatsRequest) returns (ContainerStatsResponse) {}
    // ListContainerStats returns stats of all running containers.
    rpc ListContainerStats(ListContainerStatsRequest) returns (ListContainerStatsResponse) {}

    // UpdateRuntimeConfig updates the runtime configuration based on the given request.
    rpc UpdateRuntimeConfig(UpdateRuntimeConfigRequest) returns (UpdateRuntimeConfigResponse) {}

    // Status returns the status of the runtime.
    rpc Status(StatusRequest) returns (StatusResponse) {}
}

// ImageService defines the public APIs for managing images.
service ImageService {
    // ListImages lists existing images.
    rpc ListImages(ListImagesRequest) returns (ListImagesResponse) {}
    // ImageStatus returns the status of the image. If the image is not
    // present, returns a response with ImageStatusResponse.Image set to
    // nil.
    rpc ImageStatus(ImageStatusRequest) returns (ImageStatusResponse) {}
    // PullImage pulls an image with authentication config.
    rpc PullImage(PullImageRequest) returns (PullImageResponse) {}
    // RemoveImage removes the image.
    // This call is idempotent, and must not return an error if the image has
    // already been removed.
    rpc RemoveImage(RemoveImageRequest) returns (RemoveImageResponse) {}
    // ImageFSInfo returns information of the filesystem that is used to store images.
    rpc ImageFsInfo(ImageFsInfoRequest) returns (ImageFsInfoResponse) {}
}

就是说支持kubernetes-CRI接口的运行时需要实现gRPC service,并提供定义的对外服务调用．当前dockershim支持CRI接口．

/src/k8s.io/kubernetes/pkg/kubelet/dockershim# tree -L 1
.
├── BUILD
├── cm　　
├── convert.go
├── convert_test.go
├── doc.go
├── docker_checkpoint.go
├── docker_checkpoint_test.go
├── docker_container.go　　　实现RuntimeService中关于container相关的接口
├── docker_container_test.go
├── docker_image.go　　　　实现ImageService中关于image相关的接口
├── docker_image_linux.go
├── docker_image_test.go
├── docker_image_unsupported.go
├── docker_image_windows.go
├── docker_legacy_service.go
├── docker_logs.go
├── docker_sandbox.go　　实现RuntimeService中关于sandbox相关的接口
├── docker_sandbox_test.go
├── docker_service.go
├── docker_service_test.go
├── docker_stats_linux.go 实现RuntimeService中关于status相关的接口，例如ListContainerStats,ContainerStatus
├── docker_stats_unsupported.go
├── docker_stats_windows.go
├── docker_streaming.go 实现了RuntimeService中关于stream相关的接口，比如ExecSync,Exec,Attach,PortForward
├── exec.go
├── helpers.go
├── helpers_linux.go
├── helpers_linux_test.go
├── helpers_test.go
├── helpers_unsupported.go
├── helpers_windows.go
├── libdocker
├── metrics
├── naming.go
├── naming_test.go
├── remote
├── security_context.go
├── security_context_test.go
├── selinux_util.go
├── selinux_util_test.go
└── testing

在dockershim中调用对应的gRPC远程函数，在该函数中大部分会继续调用src/k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker/kube_docker_client.go中具体的函数接口，kube_docker_client.go中会使用docker提供的client来调用cli接口，例如kube_docker_client.go中ListImages

func (d *kubeDockerClient) ListImages(opts dockertypes.ImageListOptions) ([]dockertypes.ImageSummary, error) {
    ctx, cancel := d.getTimeoutContext()
    defer cancel()
    images, err := d.client.ImageList(ctx, opts)
    if ctxErr := contextError(ctx); ctxErr != nil {
        return nil, ctxErr
    }
    if err != nil {
        return nil, err
    }
    return images, nil
}

直接调用d.client.ImageList(),　该函数实现src/k8s.io/kubernetes/vendor/github.com/docker/docker/client/image_list.go

func (cli *Client) ImageList(ctx context.Context, options types.ImageListOptions) ([]types.ImageSummary, error) {
    var images []types.ImageSummary
    query := url.Values{}

    optionFilters := options.Filters
    referenceFilters := optionFilters.Get("reference")
    if versions.LessThan(cli.version, "1.25") && len(referenceFilters) > 0 {
        query.Set("filter", referenceFilters[0])
        for _, filterValue := range referenceFilters {
            optionFilters.Del("reference", filterValue)
        }
    }
    if optionFilters.Len() > 0 {
        filterJSON, err := filters.ToParamWithVersion(cli.version, optionFilters)
        if err != nil {
            return images, err
        }
        query.Set("filters", filterJSON)
    }
    if options.All {
        query.Set("all", "1")
    }

    serverResp, err := cli.get(ctx, "/images/json", query, nil)
    if err != nil {
        return images, err
    }

    err = json.NewDecoder(serverResp.body).Decode(&images)
    ensureReaderClosed(serverResp)
    return images, err
}

最终我们从代码上可以看到是调用cli.get(ctx, “/images/json”, query, nil)完成与docker之间的交互，cli.get类似于我们执行docker images命令的时候发送的cli命令．