etcd 启动分析_kube-apiserver 源码分析

最新推荐文章于 2024-04-03 10:44:46 发布

一林黄葉

最新推荐文章于 2024-04-03 10:44:46 发布

阅读量329

点赞数

文章标签： etcd 启动分析

本文链接：https://blog.csdn.net/weixin_42527401/article/details/112192025

版权

kube-apiserver 是 Kubernetes 最重要的核心组件之一，主要提供以下的功能:

提供集群管理的 REST API 接口，包括认证授权、数据校验以及集群状态变更等
提供其他模块之间的数据交互和通信的枢纽（其他模块通过 API Server 查询或修改数据，只有 API Server 能直接操作 etcd）

注：

以下代码分析基于 kubernetes release-1.14 cmd/kube-apiserver
…代表省略的代码，只展现主要逻辑代码。

编译

make WHAT=cmd/kube-apiserver

可以自行编译一下，编译完的 binary 可在 ./_output/bin/ 下找到。

main 函数

k8s 各组件程序的入口位于 cmd 目录下，我们来看看 kube-apiserver 的源码:

cmd/kube-apiserver/apiserver.go

func main() {
    ...

    // 初始化命令
    command := app.NewAPIServerCommand(server.SetupSignalHandler())

    ...

    // Execute
    if err := command.Execute(); err != nil {
        fmt.Fprintf(os.Stderr, "error: %vn", err)
        os.Exit(1)
    }
}

这里主要用到了 Cobra CLI 框架。

另外，注意到，main 包同级目录还有个 app 包，其中包含了创建 cobra CLI 具体实现代码，所以 main 包三俩行就结束了，至于为什么要这样设计呢？

如果你看看 cmd/hyperkube/main.go 的源码，你就会明白。hyperkube 把所有的 Command 都聚合到一起了，这样我们编译一个 hyperkube，就可以拥有 k8s 所有组件的能力。另外，由于各组件创建 Cobra CLI 的逻辑都在 app 包下，那么当组件更新时，hyperkube 重新编译也能获取更新，所以提取 app 包做到了解耦。

构造 CLI

app.NewAPIServerCommand() 返回 *cobra.Command, 执行 command.Execute() 最终会调用 *cobra.Command 的 RunE 字段的函数。

我们来看看 app.NewAPIServerCommand() 是如何构造 *cobra.Command 的：

// NewAPIServerCommand creates a *cobra.Command object with default parameters
func NewAPIServerCommand(stopCh <-chan struct{}) *cobra.Command {
    s := options.NewServerRunOptions()
    // 构造 Command
    cmd := &cobra.Command{
        Use: "kube-apiserver",
        ...
        RunE: func(cmd *cobra.Command, args []string) error {
            ...

            return Run(completedOptions, stopCh)
        },
    }

    // 参数解析
    fs := cmd.Flags()
    namedFlagSets := s.Flags()
    verflag.AddFlags(namedFlagSets.FlagSet("global"))
    globalflag.AddGlobalFlags(namedFlagSets.FlagSet("global"), cmd.Name())
    options.AddCustomGlobalFlags(namedFlagSets.FlagSet("generic"))
    for _, f := range namedFlagSets.FlagSets {
        fs.AddFlagSet(f)
    }

    // 构造 usage 和 help
    usageFmt := "Usage:n  %sn"
    cols, _, _ := term.TerminalSize(cmd.OutOrStdout())
    cmd.SetUsageFunc(func(cmd *cobra.Command) error {
        fmt.Fprintf(cmd.OutOrStderr(), usageFmt, cmd.UseLine())
        cliflag.PrintSections(cmd.OutOrStderr(), namedFlagSets, cols)
        return nil
    })
    cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) {
        fmt.Fprintf(cmd.OutOrStdout(), "%snn"+usageFmt, cmd.Long, cmd.UseLine())
        cliflag.PrintSections(cmd.OutOrStdout(), namedFlagSets, cols)
    })

    return cmd
}

启动 Http 服务

通过 CreateServerChain() 创建 server，经过 PrepareRun() 返回 preparedGenericAPIServer 并最终调用其 Run() 方法。

// Run runs the specified APIServer.  This should never exit.
func Run(completeOptions completedServerRunOptions, stopCh <-chan struct{}) error {
    ...

    // 创建 server
    server, err := CreateServerChain(completeOptions, stopCh)
    if err != nil {
        return err
    }

    return server.PrepareRun().Run(stopCh)
}

进入 server.PrepareRun().Run()：

// Run spawns the secure http server. It only returns if stopCh is closed
// or the secure port cannot be listened on initially.
func (s preparedGenericAPIServer) Run(stopCh <-chan struct{}) error {
    err := s.NonBlockingRun(stopCh)
    if err != nil {
        return err
    }

    <-stopCh

    err = s.RunPreShutdownHooks()
    if err != nil {
        return err
    }

    // Wait for all requests to finish, which are bounded by the RequestTimeout variable.
    s.HandlerChainWaitGroup.Wait()

    return nil
}

Run 函数调用了 s.NonBlockingRun()，看下 NonBlockingRun() 的实现：

// NonBlockingRun spawns the secure http server. An error is
// returned if the secure port cannot be listened on.
func (s preparedGenericAPIServer) NonBlockingRun(stopCh <-chan struct{}) error {
    ...

    // Use an internal stop channel to allow cleanup of the listeners on error.
    internalStopCh := make(chan struct{})
    var stoppedCh <-chan struct{}
    if s.SecureServingInfo != nil && s.Handler != nil {
        var err error
        stoppedCh, err = s.SecureServingInfo.Serve(s.Handler, s.ShutdownTimeout, internalStopCh)
        if err != nil {
            close(internalStopCh)
            return err
        }
    }

    ...

    return nil
}

其中又继续调用了 s.SecureServingInfo.Serve() 来启动 http 服务器，继续深入：

func (s *SecureServingInfo) Serve(handler http.Handler, shutdownTimeout time.Duration, stopCh <-chan struct{}) (<-chan struct{}, error) {
    ...

    // 构造 http.Server
    secureServer := &http.Server{
        Addr:           s.Listener.Addr().String(),
        Handler:        handler,
        ...
    }

    ...

    return RunServer(secureServer, s.Listener, shutdownTimeout, stopCh)
}

其中构造了 http.Server，里面包含处理 http 请求的 handler，并继续调用了 RunServer()，继续深入，

代码大致逻辑：

前一个 goroutine 在 <-stopCh 这里阻塞，如果这个 channel 被关闭，那么就会停止阻塞并执行退出逻辑的代码，实现了优雅退出。
后一个 goroutine 中调用了 server.Serve() 来启动 http 服务。

// RunServer listens on the given port if listener is not given,
// then spawns a go-routine continuously serving until the stopCh is closed.
// It returns a stoppedCh that is closed when all non-hijacked active requests
// have been processed.
// This function does not block
// TODO: make private when insecure serving is gone from the kube-apiserver
func RunServer(
    server *http.Server,
    ln net.Listener,
    shutDownTimeout time.Duration,
    stopCh <-chan struct{},
) (<-chan struct{}, error) {
    ...

    // Shutdown server gracefully.
    stoppedCh := make(chan struct{})
    go func() {
        defer close(stoppedCh)
        <-stopCh
        ctx, cancel := context.WithTimeout(context.Background(), shutDownTimeout)
        server.Shutdown(ctx)
        cancel()
    }()

    go func() {
        ...

        err := server.Serve(listener)

        msg := fmt.Sprintf("Stopped listening on %s", ln.Addr().String())
        select {
        case <-stopCh:
            klog.Info(msg)
        default:
            panic(fmt.Sprintf("%s due to error: %v", msg, err))
        }
    }()

    return stoppedCh, nil
}

构造 kube-apiserver

回到 CreateServerChain() 继续看如何构造 kube-apiserver：

// CreateServerChain creates the apiservers connected via delegation.
func CreateServerChain(completedOptions completedServerRunOptions, stopCh <-chan struct{}) (*genericapiserver.GenericAPIServer, error) {
    ...

    // 创建扩展的 API server
    apiExtensionsServer, err := createAPIExtensionsServer(apiExtensionsConfig, genericapiserver.NewEmptyDelegate())
    if err != nil {
        return nil, err
    }

    // 创建核心的 kubeAPIServer
    kubeAPIServer, err := CreateKubeAPIServer(kubeAPIServerConfig, apiExtensionsServer.GenericAPIServer, admissionPostStartHook)
    if err != nil {
        return nil, err
    }

    ...

    // 聚合 server 的配置和创建
    aggregatorServer, err := createAggregatorServer(aggregatorConfig, kubeAPIServer.GenericAPIServer, apiExtensionsServer.Informers)
    if err != nil {
        // we don't need special handling for innerStopCh because the aggregator server doesn't create any go routines
        return nil, err
    }

    ...

    return aggregatorServer.GenericAPIServer, nil
}

先看下 CreateKubeAPIServer()：

根据传递的 kubeAPIServerConfig 来创建一个 kube-apiserver 对象
创建过程中会涉及到将遗留 API 和标准 restful API 对象注册到 restful.Container 中

// CreateKubeAPIServer creates and wires a workable kube-apiserver
func CreateKubeAPIServer(kubeAPIServerConfig *master.Config, delegateAPIServer genericapiserver.DelegationTarget, admissionPostStartHook genericapiserver.PostStartHookFunc) (*master.Master, error) {
    kubeAPIServer, err := kubeAPIServerConfig.Complete().New(delegateAPIServer)
    if err != nil {
        return nil, err
    }

    ...

    return kubeAPIServer, nil
}

进入 kubeAPIServerConfig.Complete().New()，涉及到 kube-apiserver 的创建及 API 的注册过程，

代码大致逻辑：

创建一个 kube-apiserver 对象，并放到 Master 结构体中
配置 legacy API 接口并注册到 kube-apiserver 的 restful.Container 中去
配置标准的 restful API 接口并注册到 kube-apiserver 的 restful.Container 中去

这些代码遵循 go-restful 的设计模式，即：

将处理方法注册到 Route 中去，同一个根路径下的 Route 注册到 WebService 中去，WebService append 到 Container.webServices 中。
访问的过程为 Container -> WebService -> Route。

func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Master, error) {
    ...

    // 创建 kube-apiserver
    s, err := c.GenericConfig.New("kube-apiserver", delegationTarget)
    if err != nil {
        return nil, err
    }

    m := &Master{
        GenericAPIServer: s,
    }

    // install legacy rest storage
    // 将 /api 开头的路由注册到 restful.Container
    if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
        legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
            StorageFactory:              c.ExtraConfig.StorageFactory,
            ProxyTransport:              c.ExtraConfig.ProxyTransport,
            KubeletClientConfig:         c.ExtraConfig.KubeletClientConfig,
            EventTTL:                    c.ExtraConfig.EventTTL,
            ServiceIPRange:              c.ExtraConfig.ServiceIPRange,
            ServiceNodePortRange:        c.ExtraConfig.ServiceNodePortRange,
            LoopbackClientConfig:        c.GenericConfig.LoopbackClientConfig,
            ServiceAccountIssuer:        c.ExtraConfig.ServiceAccountIssuer,
            ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
            APIAudiences:                c.GenericConfig.Authentication.APIAudiences,
        }
        m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider)
    }

    // The order here is preserved in discovery.
    // If resources with identical names exist in more than one of these groups (e.g. "deployments.apps"" and "deployments.extensions"),
    // the order of this list determines which group an unqualified resource name (e.g. "deployments") should prefer.
    // This priority order is used for local discovery, but it ends up aggregated in `k8s.io/kubernetes/cmd/kube-apiserver/app/aggregator.go
    // with specific priorities.
    // TODO: describe the priority all the way down in the RESTStorageProviders and plumb it back through the various discovery
    // handlers that we have.
    // 将 /apis 开头的路由注册到 restful.Container
    restStorageProviders := []RESTStorageProvider{
        auditregistrationrest.RESTStorageProvider{},
        authenticationrest.RESTStorageProvider{Authenticator: c.GenericConfig.Authentication.Authenticator, APIAudiences: c.GenericConfig.Authentication.APIAudiences},
        authorizationrest.RESTStorageProvider{Authorizer: c.GenericConfig.Authorization.Authorizer, RuleResolver: c.GenericConfig.RuleResolver},
        autoscalingrest.RESTStorageProvider{},
        batchrest.RESTStorageProvider{},
        certificatesrest.RESTStorageProvider{},
        coordinationrest.RESTStorageProvider{},
        extensionsrest.RESTStorageProvider{},
        networkingrest.RESTStorageProvider{},
        policyrest.RESTStorageProvider{},
        rbacrest.RESTStorageProvider{Authorizer: c.GenericConfig.Authorization.Authorizer},
        schedulingrest.RESTStorageProvider{},
        settingsrest.RESTStorageProvider{},
        storagerest.RESTStorageProvider{},
        // keep apps after extensions so legacy clients resolve the extensions versions of shared resource names.
        // See https://github.com/kubernetes/kubernetes/issues/42392
        appsrest.RESTStorageProvider{},
        admissionregistrationrest.RESTStorageProvider{},
        eventsrest.RESTStorageProvider{TTL: c.ExtraConfig.EventTTL},
    }
    m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...)

    ...

    return m, nil
}

restStorageProviders 中，每一个注册的 Route，我们可以进入接口 RESTStorageProvider 中，找到对应的路由信息，如 batchrest.RESTStorageProvider{}，在 pkg/registry/batch/rest/storage_batch.go：

...

// 创建一个接口并返回 API 组相关信息
func (p RESTStorageProvider) NewRESTStorage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) (genericapiserver.APIGroupInfo, bool) {
    apiGroupInfo := genericapiserver.NewDefaultAPIGroupInfo(batch.GroupName, legacyscheme.Scheme, legacyscheme.ParameterCodec, legacyscheme.Codecs)
    // If you add a version here, be sure to add an entry in `k8s.io/kubernetes/cmd/kube-apiserver/app/aggregator.go with specific priorities.
    // TODO refactor the plumbing to provide the information in the APIGroupInfo

    if apiResourceConfigSource.VersionEnabled(batchapiv1.SchemeGroupVersion) {
        apiGroupInfo.VersionedResourcesStorageMap[batchapiv1.SchemeGroupVersion.Version] = p.v1Storage(apiResourceConfigSource, restOptionsGetter)
    }
    if apiResourceConfigSource.VersionEnabled(batchapiv1beta1.SchemeGroupVersion) {
        apiGroupInfo.VersionedResourcesStorageMap[batchapiv1beta1.SchemeGroupVersion.Version] = p.v1beta1Storage(apiResourceConfigSource, restOptionsGetter)
    }
    if apiResourceConfigSource.VersionEnabled(batchapiv2alpha1.SchemeGroupVersion) {
        apiGroupInfo.VersionedResourcesStorageMap[batchapiv2alpha1.SchemeGroupVersion.Version] = p.v2alpha1Storage(apiResourceConfigSource, restOptionsGetter)
    }

    return apiGroupInfo, true
}

// 路由信息，可根据 API Version 区分
func (p RESTStorageProvider) v1Storage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) map[string]rest.Storage {
    storage := map[string]rest.Storage{}
    // jobs
    jobsStorage, jobsStatusStorage := jobstore.NewREST(restOptionsGetter)
    storage["jobs"] = jobsStorage
    storage["jobs/status"] = jobsStatusStorage

    return storage
}

func (p RESTStorageProvider) v1beta1Storage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) map[string]rest.Storage {
    storage := map[string]rest.Storage{}
    // cronjobs
    cronJobsStorage, cronJobsStatusStorage := cronjobstore.NewREST(restOptionsGetter)
    storage["cronjobs"] = cronJobsStorage
    storage["cronjobs/status"] = cronJobsStatusStorage

    return storage
}

...

回到上一步，继续看 m.InstallAPIs() 方法，

代码大致逻辑：

遍历所有的 Porviders，获取 API 组信息
整合所有组的 API 组信息到 apiGroupsInfo
注册安装所有的 API 组信息

// InstallAPIs will install the APIs for the restStorageProviders if they are enabled.
func (m *Master) InstallAPIs(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter, restStorageProviders ...RESTStorageProvider) {
    apiGroupsInfo := []*genericapiserver.APIGroupInfo{}

    // 遍历所有的 Porviders，获取 API 组信息
    for _, restStorageBuilder := range restStorageProviders {
        groupName := restStorageBuilder.GroupName()
        if !apiResourceConfigSource.AnyVersionForGroupEnabled(groupName) {
            klog.V(1).Infof("Skipping disabled API group %q.", groupName)
            continue
        }

        // 整合所有组的 API 组信息到 apiGroupsInfo
        apiGroupInfo, enabled := restStorageBuilder.NewRESTStorage(apiResourceConfigSource, restOptionsGetter)
        if !enabled {
            klog.Warningf("Problem initializing API group %q, skipping.", groupName)
            continue
        }

        ...

        apiGroupsInfo = append(apiGroupsInfo, &apiGroupInfo)
    }

    // 注册安装所有的 API 组信息
    if err := m.GenericAPIServer.InstallAPIGroups(apiGroupsInfo...); err != nil {
        klog.Fatalf("Error in registering group versions: %v", err)
    }
}

进入到 m.GenericAPIServer.InstallAPIGroups():

// Exposes given api groups in the API.
func (s *GenericAPIServer) InstallAPIGroups(apiGroupInfos ...*APIGroupInfo) error {
    ...

    for _, apiGroupInfo := range apiGroupInfos {
        if err := s.installAPIResources(APIGroupPrefix, apiGroupInfo, openAPIModels); err != nil {
            return fmt.Errorf("unable to install api resources: %v", err)
        }

        ...

        s.DiscoveryGroupManager.AddGroup(apiGroup)
        s.Handler.GoRestfulContainer.Add(discovery.NewAPIGroupHandler(s.Serializer, apiGroup).WebService())
    }
    return nil
}

进入到 s.installAPIResources():

// installAPIResources is a private method for installing the REST storage backing each api groupversionresource
func (s *GenericAPIServer) installAPIResources(apiPrefix string, apiGroupInfo *APIGroupInfo, openAPIModels openapiproto.Models) error {
    for _, groupVersion := range apiGroupInfo.PrioritizedVersions {
        ...

        if err := apiGroupVersion.InstallREST(s.Handler.GoRestfulContainer); err != nil {
            return fmt.Errorf("unable to setup API %v: %v", apiGroupInfo, err)
        }
    }

    return nil
}

进入到 apiGroupVersion.InstallREST():

// InstallREST registers the REST handlers (storage, watch, proxy and redirect) into a restful Container.
// It is expected that the provided path root prefix will serve all operations. Root MUST NOT end
// in a slash.
func (g *APIGroupVersion) InstallREST(container *restful.Container) error {
    prefix := path.Join(g.Root, g.GroupVersion.Group, g.GroupVersion.Version)
    // 创建一个 APIInstaller，包含 API 组、路径信息
    installer := &APIInstaller{
        group:                        g,
        prefix:                       prefix,
        minRequestTimeout:            g.MinRequestTimeout,
        enableAPIResponseCompression: g.EnableAPIResponseCompression,
    }

    apiResources, ws, registrationErrors := installer.Install()
    versionDiscoveryHandler := discovery.NewAPIVersionHandler(g.Serializer, g.GroupVersion, staticLister{apiResources})
    versionDiscoveryHandler.AddToWebService(ws)
    //将 webservice 添加到 container 中
    container.Add(ws)
    return utilerrors.NewAggregate(registrationErrors)
}

进入到 installer.Install():

// Install handlers for API resources.
func (a *APIInstaller) Install() ([]metav1.APIResource, *restful.WebService, []error) {
    var apiResources []metav1.APIResource
    var errors []error
    ws := a.newWebService()

    // Register the paths in a deterministic (sorted) order to get a deterministic swagger spec.
    paths := make([]string, len(a.group.Storage))
    var i int = 0
    for path := range a.group.Storage {
        paths[i] = path
        i++
    }
    sort.Strings(paths)
    for _, path := range paths {
        // 将路由注册到 webservice
        apiResource, err := a.registerResourceHandlers(path, a.group.Storage[path], ws)
        if err != nil {
            errors = append(errors, fmt.Errorf("error in registering resource: %s, %v", path, err))
        }
        if apiResource != nil {
            apiResources = append(apiResources, *apiResource)
        }
    }
    return apiResources, ws, errors
}

进入到 a.registerResourceHandlers()，

这段代码较长，可在 vendor/k8s.io/apiserver/pkg/endpoints/installer.go 中找到。

代码大致逻辑：

根据类型断言判断 storage 接口类型
根据接口类型和 scope，创建不同类型的 action, LIST, POST, PUT, WATCH 等等
遍历所有的 action, 根据 action 类型设置不同的 handler, 并将 path 和对应的 handler 封装成 RouteBuilder 结构
将所有 routes 添加到 webservice 中

func (a *APIInstaller) registerResourceHandlers(path string, storage rest.Storage, ws *restful.WebService) (*metav1.APIResource, error) {
    ...

    // 根据类型断言判断接口类型
    // what verbs are supported by the storage, used to know what verbs we support per path
    creater, isCreater := storage.(rest.Creater)
    namedCreater, isNamedCreater := storage.(rest.NamedCreater)
    lister, isLister := storage.(rest.Lister)
    getter, isGetter := storage.(rest.Getter)
    getterWithOptions, isGetterWithOptions := storage.(rest.GetterWithOptions)
    gracefulDeleter, isGracefulDeleter := storage.(rest.GracefulDeleter)
    collectionDeleter, isCollectionDeleter := storage.(rest.CollectionDeleter)
    updater, isUpdater := storage.(rest.Updater)
    patcher, isPatcher := storage.(rest.Patcher)
    watcher, isWatcher := storage.(rest.Watcher)
    connecter, isConnecter := storage.(rest.Connecter)
    storageMeta, isMetadata := storage.(rest.StorageMetadata)

    ...

    var apiResource metav1.APIResource
    // Get the list of actions for the given scope.
    // 根据接口类型和 scope，创建不同类型的 action, `LIST`, `POST`, `PUT`, `WATCH` 等等
    switch {
    case !namespaceScoped:
        ...

        // Handler for standard REST verbs (GET, PUT, POST and DELETE).
        // Add actions at the resource path: /api/apiVersion/resource
        actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
        actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)
        actions = appendIf(actions, action{"DELETECOLLECTION", resourcePath, resourceParams, namer, false}, isCollectionDeleter)
        // DEPRECATED in 1.11
        actions = appendIf(actions, action{"WATCHLIST", "watch/" + resourcePath, resourceParams, namer, false}, allowWatchList)

        // Add actions at the item path: /api/apiVersion/resource/{name}
        actions = appendIf(actions, action{"GET", itemPath, nameParams, namer, false}, isGetter)
        if getSubpath {
            actions = appendIf(actions, action{"GET", itemPath + "/{path:*}", proxyParams, namer, false}, isGetter)
        }
        actions = appendIf(actions, action{"PUT", itemPath, nameParams, namer, false}, isUpdater)
        actions = appendIf(actions, action{"PATCH", itemPath, nameParams, namer, false}, isPatcher)
        actions = appendIf(actions, action{"DELETE", itemPath, nameParams, namer, false}, isGracefulDeleter)
        // DEPRECATED in 1.11
        actions = appendIf(actions, action{"WATCH", "watch/" + itemPath, nameParams, namer, false}, isWatcher)
        actions = appendIf(actions, action{"CONNECT", itemPath, nameParams, namer, false}, isConnecter)
        actions = appendIf(actions, action{"CONNECT", itemPath + "/{path:*}", proxyParams, namer, false}, isConnecter && connectSubpath)
    default:
        namespaceParamName := "namespaces"
        // Handler for standard REST verbs (GET, PUT, POST and DELETE).
        namespaceParam := ws.PathParameter("namespace", "object name and auth scope, such as for teams and projects").DataType("string")
        namespacedPath := namespaceParamName + "/{namespace}/" + resource
        namespaceParams := []*restful.Parameter{namespaceParam}

        resourcePath := namespacedPath
        resourceParams := namespaceParams
        itemPath := namespacedPath + "/{name}"
        nameParams := append(namespaceParams, nameParam)
        proxyParams := append(nameParams, pathParam)
        itemPathSuffix := ""
        if isSubresource {
            itemPathSuffix = "/" + subresource
            itemPath = itemPath + itemPathSuffix
            resourcePath = itemPath
            resourceParams = nameParams
        }
        apiResource.Name = path
        apiResource.Namespaced = true
        apiResource.Kind = resourceKind
        namer := handlers.ContextBasedNaming{
            SelfLinker:         a.group.Linker,
            ClusterScoped:      false,
            SelfLinkPathPrefix: gpath.Join(a.prefix, namespaceParamName) + "/",
            SelfLinkPathSuffix: itemPathSuffix,
        }

        actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
        actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)
        actions = appendIf(actions, action{"DELETECOLLECTION", resourcePath, resourceParams, namer, false}, isCollectionDeleter)
        // DEPRECATED in 1.11
        actions = appendIf(actions, action{"WATCHLIST", "watch/" + resourcePath, resourceParams, namer, false}, allowWatchList)

        actions = appendIf(actions, action{"GET", itemPath, nameParams, namer, false}, isGetter)
        if getSubpath {
            actions = appendIf(actions, action{"GET", itemPath + "/{path:*}", proxyParams, namer, false}, isGetter)
        }
        actions = appendIf(actions, action{"PUT", itemPath, nameParams, namer, false}, isUpdater)
        actions = appendIf(actions, action{"PATCH", itemPath, nameParams, namer, false}, isPatcher)
        actions = appendIf(actions, action{"DELETE", itemPath, nameParams, namer, false}, isGracefulDeleter)
        // DEPRECATED in 1.11
        actions = appendIf(actions, action{"WATCH", "watch/" + itemPath, nameParams, namer, false}, isWatcher)
        actions = appendIf(actions, action{"CONNECT", itemPath, nameParams, namer, false}, isConnecter)
        actions = appendIf(actions, action{"CONNECT", itemPath + "/{path:*}", proxyParams, namer, false}, isConnecter && connectSubpath)

        // list or post across namespace.
        // For ex: LIST all pods in all namespaces by sending a LIST request at /api/apiVersion/pods.
        // TODO: more strongly type whether a resource allows these actions on "all namespaces" (bulk delete)
        if !isSubresource {
            actions = appendIf(actions, action{"LIST", resource, params, namer, true}, isLister)
            // DEPRECATED in 1.11
            actions = appendIf(actions, action{"WATCHLIST", "watch/" + resource, params, namer, true}, allowWatchList)
        }
    }

    ...

    // 遍历所有的 action, 根据 action 类型设置不同的 handler, 并将 path 和对应的 handler 封装成 RouteBuilder 结构
    for _, action := range actions {
        producedObject := storageMeta.ProducesObject(action.Verb)
        if producedObject == nil {
            producedObject = defaultVersionedObject
        }
        reqScope.Namer = action.Namer

        requestScope := "cluster"
        var namespaced string
        var operationSuffix string
        if apiResource.Namespaced {
            requestScope = "namespace"
            namespaced = "Namespaced"
        }
        if strings.HasSuffix(action.Path, "/{path:*}") {
            requestScope = "resource"
            operationSuffix = operationSuffix + "WithPath"
        }
        if action.AllNamespaces {
            requestScope = "cluster"
            operationSuffix = operationSuffix + "ForAllNamespaces"
            namespaced = ""
        }

        if kubeVerb, found := toDiscoveryKubeVerb[action.Verb]; found {
            if len(kubeVerb) != 0 {
                kubeVerbs[kubeVerb] = struct{}{}
            }
        } else {
            return nil, fmt.Errorf("unknown action verb for discovery: %s", action.Verb)
        }

        routes := []*restful.RouteBuilder{}

        // If there is a subresource, kind should be the parent's kind.
        if isSubresource {
            parentStorage, ok := a.group.Storage[resource]
            if !ok {
                return nil, fmt.Errorf("missing parent storage: %q", resource)
            }

            fqParentKind, err := GetResourceKind(a.group.GroupVersion, parentStorage, a.group.Typer)
            if err != nil {
                return nil, err
            }
            kind = fqParentKind.Kind
        }

        verbOverrider, needOverride := storage.(StorageMetricsOverride)

        switch action.Verb {
        case "GET": // Get a resource.
            var handler restful.RouteFunction
            if isGetterWithOptions {
                handler = restfulGetResourceWithOptions(getterWithOptions, reqScope, isSubresource)
            } else {
                handler = restfulGetResource(getter, exporter, reqScope)
            }

            if needOverride {
                // need change the reported verb
                handler = metrics.InstrumentRouteFunc(verbOverrider.OverrideMetricsVerb(action.Verb), group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
            } else {
                handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
            }

            if a.enableAPIResponseCompression {
                handler = genericfilters.RestfulWithCompression(handler)
            }
            doc := "read the specified " + kind
            if isSubresource {
                doc = "read " + subresource + " of the specified " + kind
            }
            route := ws.GET(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("read"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
                Returns(http.StatusOK, "OK", producedObject).
                Writes(producedObject)
            if isGetterWithOptions {
                if err := addObjectParams(ws, route, versionedGetOptions); err != nil {
                    return nil, err
                }
            }
            if isExporter {
                if err := addObjectParams(ws, route, versionedExportOptions); err != nil {
                    return nil, err
                }
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        case "LIST": // List all resources of a kind.
            doc := "list objects of kind " + kind
            if isSubresource {
                doc = "list " + subresource + " of objects of kind " + kind
            }
            handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, restfulListResource(lister, watcher, reqScope, false, a.minRequestTimeout))
            if a.enableAPIResponseCompression {
                handler = genericfilters.RestfulWithCompression(handler)
            }
            route := ws.GET(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("list"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), allMediaTypes...)...).
                Returns(http.StatusOK, "OK", versionedList).
                Writes(versionedList)
            if err := addObjectParams(ws, route, versionedListOptions); err != nil {
                return nil, err
            }
            switch {
            case isLister && isWatcher:
                doc := "list or watch objects of kind " + kind
                if isSubresource {
                    doc = "list or watch " + subresource + " of objects of kind " + kind
                }
                route.Doc(doc)
            case isWatcher:
                doc := "watch objects of kind " + kind
                if isSubresource {
                    doc = "watch " + subresource + "of objects of kind " + kind
                }
                route.Doc(doc)
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        case "PUT": // Update a resource.
            doc := "replace the specified " + kind
            if isSubresource {
                doc = "replace " + subresource + " of the specified " + kind
            }
            handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, restfulUpdateResource(updater, reqScope, admit))
            route := ws.PUT(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("replace"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
                Returns(http.StatusOK, "OK", producedObject).
                // TODO: in some cases, the API may return a v1.Status instead of the versioned object
                // but currently go-restful can't handle multiple different objects being returned.
                Returns(http.StatusCreated, "Created", producedObject).
                Reads(defaultVersionedObject).
                Writes(producedObject)
            if err := addObjectParams(ws, route, versionedUpdateOptions); err != nil {
                return nil, err
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        case "PATCH": // Partially update a resource
            doc := "partially update the specified " + kind
            if isSubresource {
                doc = "partially update " + subresource + " of the specified " + kind
            }
            supportedTypes := []string{
                string(types.JSONPatchType),
                string(types.MergePatchType),
                string(types.StrategicMergePatchType),
            }
            if utilfeature.DefaultFeatureGate.Enabled(features.ServerSideApply) {
                supportedTypes = append(supportedTypes, string(types.ApplyPatchType))
            }
            handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, restfulPatchResource(patcher, reqScope, admit, supportedTypes))
            route := ws.PATCH(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Consumes(supportedTypes...).
                Operation("patch"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
                Returns(http.StatusOK, "OK", producedObject).
                Reads(metav1.Patch{}).
                Writes(producedObject)
            if err := addObjectParams(ws, route, versionedPatchOptions); err != nil {
                return nil, err
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        case "POST": // Create a resource.
            var handler restful.RouteFunction
            if isNamedCreater {
                handler = restfulCreateNamedResource(namedCreater, reqScope, admit)
            } else {
                handler = restfulCreateResource(creater, reqScope, admit)
            }
            handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
            article := getArticleForNoun(kind, " ")
            doc := "create" + article + kind
            if isSubresource {
                doc = "create " + subresource + " of" + article + kind
            }
            route := ws.POST(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("create"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
                Returns(http.StatusOK, "OK", producedObject).
                // TODO: in some cases, the API may return a v1.Status instead of the versioned object
                // but currently go-restful can't handle multiple different objects being returned.
                Returns(http.StatusCreated, "Created", producedObject).
                Returns(http.StatusAccepted, "Accepted", producedObject).
                Reads(defaultVersionedObject).
                Writes(producedObject)
            if err := addObjectParams(ws, route, versionedCreateOptions); err != nil {
                return nil, err
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        case "DELETE": // Delete a resource.
            article := getArticleForNoun(kind, " ")
            doc := "delete" + article + kind
            if isSubresource {
                doc = "delete " + subresource + " of" + article + kind
            }
            handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, restfulDeleteResource(gracefulDeleter, isGracefulDeleter, reqScope, admit))
            route := ws.DELETE(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("delete"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
                Writes(versionedStatus).
                Returns(http.StatusOK, "OK", versionedStatus).
                Returns(http.StatusAccepted, "Accepted", versionedStatus)
            if isGracefulDeleter {
                route.Reads(versionedDeleterObject)
                route.ParameterNamed("body").Required(false)
                if err := addObjectParams(ws, route, versionedDeleteOptions); err != nil {
                    return nil, err
                }
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        case "DELETECOLLECTION":
            doc := "delete collection of " + kind
            if isSubresource {
                doc = "delete collection of " + subresource + " of a " + kind
            }
            handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, restfulDeleteCollection(collectionDeleter, isCollectionDeleter, reqScope, admit))
            route := ws.DELETE(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("deletecollection"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
                Writes(versionedStatus).
                Returns(http.StatusOK, "OK", versionedStatus)
            if err := addObjectParams(ws, route, versionedListOptions); err != nil {
                return nil, err
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        // deprecated in 1.11
        case "WATCH": // Watch a resource.
            doc := "watch changes to an object of kind " + kind
            if isSubresource {
                doc = "watch changes to " + subresource + " of an object of kind " + kind
            }
            doc += ". deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter."
            handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, restfulListResource(lister, watcher, reqScope, true, a.minRequestTimeout))
            route := ws.GET(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("watch"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(allMediaTypes...).
                Returns(http.StatusOK, "OK", versionedWatchEvent).
                Writes(versionedWatchEvent)
            if err := addObjectParams(ws, route, versionedListOptions); err != nil {
                return nil, err
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        // deprecated in 1.11
        case "WATCHLIST": // Watch all resources of a kind.
            doc := "watch individual changes to a list of " + kind
            if isSubresource {
                doc = "watch individual changes to a list of " + subresource + " of " + kind
            }
            doc += ". deprecated: use the 'watch' parameter with a list operation instead."
            handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, restfulListResource(lister, watcher, reqScope, true, a.minRequestTimeout))
            route := ws.GET(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("watch"+namespaced+kind+strings.Title(subresource)+"List"+operationSuffix).
                Produces(allMediaTypes...).
                Returns(http.StatusOK, "OK", versionedWatchEvent).
                Writes(versionedWatchEvent)
            if err := addObjectParams(ws, route, versionedListOptions); err != nil {
                return nil, err
            }
            addParams(route, action.Params)
            routes = append(routes, route)
        case "CONNECT":
            for _, method := range connecter.ConnectMethods() {
                connectProducedObject := storageMeta.ProducesObject(method)
                if connectProducedObject == nil {
                    connectProducedObject = "string"
                }
                doc := "connect " + method + " requests to " + kind
                if isSubresource {
                    doc = "connect " + method + " requests to " + subresource + " of " + kind
                }
                handler := metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, restfulConnectResource(connecter, reqScope, admit, path, isSubresource))
                route := ws.Method(method).Path(action.Path).
                    To(handler).
                    Doc(doc).
                    Operation("connect" + strings.Title(strings.ToLower(method)) + namespaced + kind + strings.Title(subresource) + operationSuffix).
                    Produces("*/*").
                    Consumes("*/*").
                    Writes(connectProducedObject)
                if versionedConnectOptions != nil {
                    if err := addObjectParams(ws, route, versionedConnectOptions); err != nil {
                        return nil, err
                    }
                }
                addParams(route, action.Params)
                routes = append(routes, route)

                // transform ConnectMethods to kube verbs
                if kubeVerb, found := toDiscoveryKubeVerb[method]; found {
                    if len(kubeVerb) != 0 {
                        kubeVerbs[kubeVerb] = struct{}{}
                    }
                }
            }
        default:
            return nil, fmt.Errorf("unrecognized action verb: %s", action.Verb)
        }
        // 将所有 routes 添加到 webservice 中
        for _, route := range routes {
            route.Metadata(ROUTE_META_GVK, metav1.GroupVersionKind{
                Group:   reqScope.Kind.Group,
                Version: reqScope.Kind.Version,
                Kind:    reqScope.Kind.Kind,
            })
            route.Metadata(ROUTE_META_ACTION, strings.ToLower(action.Verb))
            ws.Route(route)
        }
        // Note: update GetAuthorizerAttributes() when adding a custom handler.
    }

    ...

    return &apiResource, nil
}

filter （权限相关）

kube-apiserver 权限相关的功能，AuthN、AuthZ、Admission 等，是通过 filter 的形式实现的。

回到 cmd/kube-apiserver/app/server.go，进去 CreateKubeAPIServerConfig() -> buildGenericConfig() -> genericapiserver.NewConfig() -> DefaultBuildHandlerChain，可以看到如下代码：

func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler {
    handler := genericapifilters.WithAuthorization(apiHandler, c.Authorization.Authorizer, c.Serializer)
    handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc)
    handler = genericapifilters.WithImpersonation(handler, c.Authorization.Authorizer, c.Serializer)
    handler = genericapifilters.WithAudit(handler, c.AuditBackend, c.AuditPolicyChecker, c.LongRunningFunc)
    failedHandler := genericapifilters.Unauthorized(c.Serializer, c.Authentication.SupportsBasicAuth)
    failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, c.AuditBackend, c.AuditPolicyChecker)
    handler = genericapifilters.WithAuthentication(handler, c.Authentication.Authenticator, failedHandler, c.Authentication.APIAudiences)
    handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true")
    handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc, c.RequestTimeout)
    handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup)
    handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver)
    handler = genericfilters.WithPanicRecovery(handler)
    return handler
}

回到 cmd/kube-apiserver/app/server.go，进去 CreateKubeAPIServer() -> kubeAPIServerConfig.Complete().New() -> c.GenericConfig.New()，可以看到如下代码：

// New creates a new server which logically combines the handling chain with the passed server.
// name is used to differentiate for logging. The handler chain in particular can be difficult as it starts delgating.
// delegationTarget may not be nil.
func (c completedConfig) New(name string, delegationTarget DelegationTarget) (*GenericAPIServer, error) {
    ...

    // handlerChainBuilder
    handlerChainBuilder := func(handler http.Handler) http.Handler {
        return c.BuildHandlerChainFunc(handler, c.Config)
    }
    apiServerHandler := NewAPIServerHandler(name, c.Serializer, handlerChainBuilder, delegationTarget.UnprotectedHandler())
    ...

    return s, nil
}

进入 NewAPIServerHandler()，

代码大致逻辑：

将 director 作为参数传给回调函数 handlerChainBuilder，完成 restful.Container 的 handler 注册工作

func NewAPIServerHandler(name string, s runtime.NegotiatedSerializer, handlerChainBuilder HandlerChainBuilderFn, notFoundHandler http.Handler) *APIServerHandler {
    nonGoRestfulMux := mux.NewPathRecorderMux(name)
    if notFoundHandler != nil {
        nonGoRestfulMux.NotFoundHandler(notFoundHandler)
    }

    gorestfulContainer := restful.NewContainer()
    gorestfulContainer.ServeMux = http.NewServeMux()
    gorestfulContainer.Router(restful.CurlyRouter{}) // e.g. for proxy/{kind}/{name}/{*}
    gorestfulContainer.RecoverHandler(func(panicReason interface{}, httpWriter http.ResponseWriter) {
        logStackOnRecover(s, panicReason, httpWriter)
    })
    gorestfulContainer.ServiceErrorHandler(func(serviceErr restful.ServiceError, request *restful.Request, response *restful.Response) {
        serviceErrorHandler(s, serviceErr, request, response)
    })

    director := director{
        name:               name,
        goRestfulContainer: gorestfulContainer,
        nonGoRestfulMux:    nonGoRestfulMux,
    }

    return &APIServerHandler{
        // 回调函数
        FullHandlerChain:   handlerChainBuilder(director),
        GoRestfulContainer: gorestfulContainer,
        NonGoRestfulMux:    nonGoRestfulMux,
        Director:           director,
    }
}

就这样，通过 DefaultBuildHandlerChain 中各种 filter，kube-apiserver 完成了权限控制。