k8s源码分析——kube-apiserver源码阅读

最新推荐文章于 2024-08-12 08:51:35 发布

cbmljs

最新推荐文章于 2024-08-12 08:51:35 发布

阅读量1.7k

点赞数

分类专栏： kubernetes 文章标签： k8s kube-apiserver etcd rbac RESTStorage

本文链接：https://blog.csdn.net/cbmljs/article/details/103241274

版权

kubernetes 专栏收录该内容

75 篇文章 16 订阅

订阅专栏

前言

Kubernetes API Server 是有kube-apiserver进程实现的，他运行在Kubernetes的管理节点--Master上，并对外提供Kubernetes Restful API，它提供的主要是与集群管理相关的API服务，例如校验pod、service、rc controller的配置并存储到后端的etcd server上。下面我们分别对其启动过程、关键代码分析及设计总结等进行深入讲解。

API Server是Kubernetes的核心组件之一，其作用是通过RESTFUL的方式，向所有客户端提供一个集群内资源的统一的增改删查的接口，并将资源的状态存储在etcd中。API Server入口函数的位置在cmd/kube-apiserver/apiserver.go中，也是通过cobra注册了kube-apiserver的命令。

启动分析

Kube-api 进程的入口源码位置如下：
kubernetes/cmd/kube-apiserver/apiserver.go

func main() {
        rand.Seed(time.Now().UnixNano())

        command := app.NewAPIServerCommand(server.SetupSignalHandler())

        // TODO: once we switch everything over to Cobra commands, we can go back to calling
        // utilflag.InitFlags() (by removing its pflag.Parse() call). For now, we have to set the
        // normalize func and add the go flag set by hand.
        // utilflag.InitFlags()
        logs.InitLogs()
        defer logs.FlushLogs()

        if err := command.Execute(); err != nil {
                fmt.Fprintf(os.Stderr, "error: %v\n", err)
                os.Exit(1)
        }
}

同Kubernetes所有的组件启动代码一致，apiserver启动使用的是cobra的命令行方式

          RunE: func(cmd *cobra.Command, args []string) error {
                        verflag.PrintAndExitIfRequested()
                        utilflag.PrintFlags(cmd.Flags())

                        // set default options
                        // 完成apiserver默认参数配置
                        completedOptions, err := Complete(s)
                        if err != nil {
                                return err
                        }

                        // validate options
                        // 判断配置是否合法
                        if errs := completedOptions.Validate(); len(errs) != 0 {
                                return utilerrors.NewAggregate(errs)
                        }

                        return Run(completedOptions, stopCh)
                },

启动主要完成三个步骤：
1、完成参数的配置；
2、判断配置是否合法；
3、执行最终的Run方法。Run方法比较简单

// Run runs the specified APIServer.  This should never exit.// 通过CreateServerChain创建完server后，继续调用GenericAPIServer的Run方法完成最终的启动工作。// 首先通过PrepareRun方法完成启动前的路由收尾工作，该方法主要完成了Swagger和OpenAPI路由的注册工作// （Swagger和OpenAPI主要包含了Kubernetes API的所有细节与规范），并完成/healthz路由的注册工作。// 完成后，开始最终的server启动工作。func Run(completeOptions completedServerRunOptions, stopCh <-chan struct{}) error {
        // To help debugging, immediately log version
        klog.Infof("Version: %+v", version.Get())
        // 创建API server
        server, err := CreateServerChain(completeOptions, stopCh)
        if err != nil {
                return err
        }
        // 运行API Server
        return server.PrepareRun().Run(stopCh)
}

主要执行两个步骤：1、创建server端；2、启动server。因为apiserver本质上就是一个server服务器，所有代码核心就是如何配置server，包括路由、访问权限以及同数据库（etcd）的交互等。先看一下server端是如何创建起来的。

Server端创建

Server端的创建集中在CreateServerChain方法。方法代码如下：

// CreateServerChain creates the apiservers connected via delegation.// CreateServerChain创建通过委托连接的apiservers，创建一系列的serverfunc CreateServerChain(completedOptions completedServerRunOptions, stopCh <-chan struct{}) (*genericapiserver.GenericAPIServer, error) {
        nodeTunneler, proxyTransport, err := CreateNodeDialer(completedOptions)
        if err != nil {
                return nil, err
        }

        // 1.创建kubeAPIServerConfig配置
        kubeAPIServerConfig, insecureServingInfo, serviceResolver, pluginInitializer, admissionPostStartHook, err := CreateKubeAPIServerConfig(completedOptions, nodeTunneler, proxyTransport)
        if err != nil {
                return nil, err
        }

        // If additional API servers are added, they should be gated.
        // 2.判断是否配置了扩展API server，创建apiExtensionsConfig配置
        apiExtensionsConfig, err := createAPIExtensionsConfig(*kubeAPIServerConfig.GenericConfig, kubeAPIServerConfig.ExtraConfig.VersionedInformers, pluginInitializer, completedOptions.ServerRunOptions, completedOptions.MasterCount,
                serviceResolver, webhook.NewDefaultAuthenticationInfoResolverWrapper(proxyTransport, kubeAPIServerConfig.GenericConfig.LoopbackClientConfig))
        if err != nil {
                return nil, err
        }

        // apiExtensionsServer，可扩展的API server
        // 3.启动扩展的API server
        apiExtensionsServer, err := createAPIExtensionsServer(apiExtensionsConfig, genericapiserver.NewEmptyDelegate())
        if err != nil {
                return nil, err
        }

        // 4.启动最核心的kubeAPIServer
        kubeAPIServer, err := CreateKubeAPIServer(kubeAPIServerConfig, apiExtensionsServer.GenericAPIServer, admissionPostStartHook)
        if err != nil {
                return nil, err
        }

        // otherwise go down the normal path of standing the aggregator up in front of the API server
        // this wires up openapi
        kubeAPIServer.GenericAPIServer.PrepareRun()

        // This will wire up openapi for extension api server
        apiExtensionsServer.GenericAPIServer.PrepareRun()

        // aggregator comes last in the chain
        // 5.聚合层的配置aggregatorConfig
        aggregatorConfig, err := createAggregatorConfig(*kubeAPIServerConfig.GenericConfig, completedOptions.ServerRunOptions, kubeAPIServerConfig.ExtraConfig.VersionedInformers, serviceResolver, proxyTransport, pluginInitializer)
        if err != nil {
                return nil, err
        }
        // 6.aggregatorServer,聚合服务器，对所有的服务器访问的整合
        aggregatorServer, err := createAggregatorServer(aggregatorConfig, kubeAPIServer.GenericAPIServer, apiExtensionsServer.Informers)
        if err != nil {
                // we don't need special handling for innerStopCh because the aggregator server doesn't create any go routines
                return nil, err
        }

        // 7.启动非安全端口的server
        if insecureServingInfo != nil {
                insecureHandlerChain := kubeserver.BuildInsecureHandlerChain(aggregatorServer.GenericAPIServer.UnprotectedHandler(), kubeAPIServerConfig.GenericConfig)
                if err := insecureServingInfo.Serve(insecureHandlerChain, kubeAPIServerConfig.GenericConfig.RequestTimeout, stopCh); err != nil {
                        return nil, err
                }
        }

        // 8.返回GenericAPIServer，后续启动安全端口的server
        return aggregatorServer.GenericAPIServer, nil
}

主要进行了以下几件事：
（1）调用CreateNodeDialer，创建与节点交互的工具。
（2）配置API Server的Config。这里同时还配置了Extension API Server的Config，用于配置用户自己编写的API Server。
（3）根据Config，创建API Server和Extension API Server。
（4）运行API Server。通过调用PrepareRun方法实现。
（5）创建并运行aggregator（将API Server和Extension API Server整合在一起，暂时不提）。

创建server，包括扩展的apiserver和原生的apiserver，调用方法为createAPIExtensionsServer和CreateKubeAPIServer。主要就是将各个handler的路由方法注册到Container中去，完全遵循go-restful的设计模式，即将处理方法注册到Route中去，同一个根路径下的Route注册到WebService中去，WebService注册到Container中，Container负责分发。访问的过程为Container-->WebService-->Route。更加详细的go-restful使用可以参考其代码；

创建

CreateKubeAPIServer方法如下：

// CreateKubeAPIServer creates and wires a workable kube-apiserverfunc CreateKubeAPIServer(kubeAPIServerConfig *master.Config, delegateAPIServer genericapiserver.DelegationTarget, admissionPostStartHook genericapiserver.PostStartHookFunc) (*master.Master, error) {
//ljs：通过给定的配置，新建一个master实例
        kubeAPIServer, err := kubeAPIServerConfig.Complete().New(delegateAPIServer)
        if err != nil {
                return nil, err
        }

        kubeAPIServer.GenericAPIServer.AddPostStartHookOrDie("start-kube-apiserver-admission-initializer", admissionPostStartHook)

        return kubeAPIServer, nil
}

首先调用kubeAPIServerConfig.Complete().New方法生成一个kubeAPIServer实例，之后为这个实例添加启动后执行的钩子函数。可以看到，这个API Server将前面创建的Extension API Server作为了代理Server。New方法位于pkg/master/master.go中。进入New方法：

// New returns a new instance of Master from the given config.// Certain config fields will be set to a default value if unset.// Certain config fields must be specified, including://   KubeletClientConfig// 通过给定的配置，返回一个新的Master实例。对于部分未配置的选项，可以使用默认配置；但是对于KubeletClientConfig这样的配置，必须手动指定func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Master, error) {
        if reflect.DeepEqual(c.ExtraConfig.KubeletClientConfig, kubeletclient.KubeletClientConfig{}) {
                return nil, fmt.Errorf("Master.New() called with empty config.KubeletClientConfig")
        }

        // 1. 初始化，创建go-restful的Container，初始化apiServerHandler，API Server预先注册了一些默认path
        s, err := c.GenericConfig.New("kube-apiserver", delegationTarget)
        if err != nil {
                return nil, err
        }

        if c.ExtraConfig.EnableLogsSupport {
                routes.Logs{}.Install(s.Handler.GoRestfulContainer)
        }

        m := &Master{
                GenericAPIServer: s,
        }

        // install legacy rest storage
        // /api开头的版本api注册到Container中去，如Pod、Namespace等资源
        if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
                legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
                        StorageFactory:              c.ExtraConfig.StorageFactory,
                        ProxyTransport:              c.ExtraConfig.ProxyTransport,
                        KubeletClientConfig:         c.ExtraConfig.KubeletClientConfig,
                        EventTTL:                    c.ExtraConfig.EventTTL,
                        ServiceIPRange:              c.ExtraConfig.ServiceIPRange,
                        ServiceNodePortRange:        c.ExtraConfig.ServiceNodePortRange,
                        LoopbackClientConfig:        c.GenericConfig.LoopbackClientConfig,
                        ServiceAccountIssuer:        c.ExtraConfig.ServiceAccountIssuer,
                        ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
                        APIAudiences:                c.GenericConfig.Authentication.APIAudiences,
                }
                m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider)
        }

        // The order here is preserved in discovery.
        // If resources with identical names exist in more than one of these groups (e.g. "deployments.apps"" and "deployments.extensions"),
        // the order of this list determines which group an unqualified resource name (e.g. "deployments") should prefer.
        // This priority order is used for local discovery, but it ends up aggregated in `k8s.io/kubernetes/cmd/kube-apiserver/app/aggregator.go
        // with specific priorities.
        // TODO: describe the priority all the way down in the RESTStorageProviders and plumb it back through the various discovery
        // handlers that we have.
        // /apis开头版本的api注册到Container中
        restStorageProviders := []RESTStorageProvider{
                auditregistrationrest.RESTStorageProvider{},
                authenticationrest.RESTStorageProvider{Authenticator: c.GenericConfig.Authentication.Authenticator, APIAudiences: c.GenericConfig.Authentication.APIAudiences},
                authorizationrest.RESTStorageProvider{Authorizer: c.GenericConfig.Authorization.Authorizer, RuleResolver: c.GenericConfig.RuleResolver},
                autoscalingrest.RESTStorageProvider{},
                batchrest.RESTStorageProvider{},
                certificatesrest.RESTStorageProvider{},
                coordinationrest.RESTStorageProvider{},
                extensionsrest.RESTStorageProvider{},
                networkingrest.RESTStorageProvider{},
                policyrest.RESTStorageProvider{},
                rbacrest.RESTStorageProvider{Authorizer: c.GenericConfig.Authorization.Authorizer},
                schedulingrest.RESTStorageProvider{},
                settingsrest.RESTStorageProvider{},
                storagerest.RESTStorageProvider{},
                // keep apps after extensions so legacy clients resolve the extensions versions of shared resource names.
                // See https://github.com/kubernetes/kubernetes/issues/42392
                appsrest.RESTStorageProvider{},
                admissionregistrationrest.RESTStorageProvider{},
                eventsrest.RESTStorageProvider{TTL: c.ExtraConfig.EventTTL},
        }
        m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...)

        if c.ExtraConfig.Tunneler != nil {
                m.installTunneler(c.ExtraConfig.Tunneler, corev1client.NewForConfigOrDie(c.GenericConfig.LoopbackClientConfig).Nodes())
        }

        m.GenericAPIServer.AddPostStartHookOrDie("ca-registration", c.ExtraConfig.ClientCARegistrationHook.PostStartHook)

        return m, nil
}

首先，通过GenericConfig.New，基于前面配置的Config创建一个API Server对象（并为这个对象添加默认的path，如/versions、/metrics等），再基于这个API Server对象创建一个master对象。
其次，为这个master创建RestStorageProvider，并注册API。这通过调用InstallLegacyAPI和InstallAPIs方法实现。前者用于注册k8s前期的核心API（在/api路径下），后者用于注册k8s新增的API（在/apis路径下）。
最后，为master添加钩子函数，并返回。下面对New、InstallLegacyAPI方法进行分析。InstallAPIs方法与InstallLegacyAPI的调用逻辑几乎一致。

New

进入GenericConfig.New方法，这个方法创建了一个GenericAPIServer：

func (c completedConfig) New(name string, delegationTarget DelegationTarget) (*GenericAPIServer, error) {
        if c.Serializer == nil {
                return nil, fmt.Errorf("Genericapiserver.New() called with config.Serializer == nil")
        }
        if c.LoopbackClientConfig == nil {
                return nil, fmt.Errorf("Genericapiserver.New() called with config.LoopbackClientConfig == nil")
        }

        handlerChainBuilder := func(handler http.Handler) http.Handler {
                return c.BuildHandlerChainFunc(handler, c.Config)
        }
        // 新建一个gorestfulContainer
        apiServerHandler := NewAPIServerHandler(name, c.Serializer, handlerChainBuilder, delegationTarget.UnprotectedHandler())

        s := &GenericAPIServer{
                discoveryAddresses:     c.DiscoveryAddresses,
                LoopbackClientConfig:   c.LoopbackClientConfig,
                legacyAPIGroupPrefixes: c.LegacyAPIGroupPrefixes,
                admissionControl:       c.AdmissionControl,
                Serializer:             c.Serializer,
                AuditBackend:           c.AuditBackend,
                Authorizer:             c.Authorization.Authorizer,
                delegationTarget:       delegationTarget,
                HandlerChainWaitGroup:  c.HandlerChainWaitGroup,

                minRequestTimeout: time.Duration(c.MinRequestTimeout) * time.Second,
                ShutdownTimeout:   c.RequestTimeout,

                SecureServingInfo: c.SecureServing,
                ExternalAddress:   c.ExternalAddress,

                Handler: apiServerHandler,

                listedPathProvider: apiServerHandler,

                swaggerConfig: c.SwaggerConfig,
                openAPIConfig: c.OpenAPIConfig,

                postStartHooks:         map[string]postStartHookEntry{},
                preShutdownHooks:       map[string]preShutdownHookEntry{},
                disabledPostStartHooks: c.DisabledPostStartHooks,

                healthzChecks: c.HealthzChecks,

                DiscoveryGroupManager: discovery.NewRootAPIsHandler(c.DiscoveryAddresses, c.Serializer),

                enableAPIResponseCompression: c.EnableAPIResponseCompression,
                maxRequestBodyBytes:          c.MaxRequestBodyBytes,
        }

        for {
                if c.JSONPatchMaxCopyBytes <= 0 {
                        break
                }
                existing := atomic.LoadInt64(&jsonpatch.AccumulatedCopySizeLimit)
                if existing > 0 && existing < c.JSONPatchMaxCopyBytes {
                        break
                }
                if atomic.CompareAndSwapInt64(&jsonpatch.AccumulatedCopySizeLimit, existing, c.JSONPatchMaxCopyBytes) {
                        break
                }
        }

        for k, v := range delegationTarget.PostStartHooks() {
                s.postStartHooks[k] = v
        }

        for k, v := range delegationTarget.PreShutdownHooks() {
                s.preShutdownHooks[k] = v
        }

        genericApiServerHookName := "generic-apiserver-start-informers"
        if c.SharedInformerFactory != nil && !s.isPostStartHookRegistered(genericApiServerHookName) {
                err := s.AddPostStartHook(genericApiServerHookName, func(context PostStartHookContext) error {
                        c.SharedInformerFactory.Start(context.StopCh)
                        return nil
                })
                if err != nil {
                        return nil, err
                }
        }

        for _, delegateCheck := range delegationTarget.HealthzChecks() {
                skip := false
                for _, existingCheck := range c.HealthzChecks {
                        if existingCheck.Name() == delegateCheck.Name() {
                                skip = true
                                break
                        }
                }
                if skip {
                        continue
                }

                s.healthzChecks = append(s.healthzChecks, delegateCheck)
        }

        s.listedPathProvider = routes.ListedPathProviders{s.listedPathProvider, delegationTarget}

        installAPI(s, c.Config)

        // use the UnprotectedHandler from the delegation target to ensure that we don't attempt to double authenticator, authorize,
        // or some other part of the filter chain in delegation cases.
        if delegationTarget.UnprotectedHandler() == nil && c.EnableIndex {
                s.Handler.NonGoRestfulMux.NotFoundHandler(routes.IndexLister{
                        StatusCode:   http.StatusNotFound,
                        PathProvider: s.listedPathProvider,
                })
        }

        return s, nil
}

这一方法的核心是NewAPIServerHandler方法，此方法初始化了一个Container。这里的Container并非k8s中的容器，而是go-restful的一个概念，可参考https://www.cnblogs.com/ldaniel/p/5868384.html。大体说来，Container是一组WebService的集合，可以监听在不同的端口上，而WebService又是一组Route的集合，为这些Route创建统一的root path等。这个go-restful项目来源于https://github.com/emicklei/go-restful。

本质就是创建一个go-restful的container（NewAPIServerHandler），以及注册handler（installAPI）。

NewAPIServerHandler

k8s.io/aposerver/pkg/server/handler.go

func NewAPIServerHandler(name string, s runtime.NegotiatedSerializer, handlerChainBuilder HandlerChainBuilderFn, notFoundHandler http.Handler) *APIServerHandler {
    nonGoRestfulMux := mux.NewPathRecorderMux(name)
    if notFoundHandler != nil {
        nonGoRestfulMux.NotFoundHandler(notFoundHandler)
    }

    gorestfulContainer := restful.NewContainer()
    gorestfulContainer.ServeMux = http.NewServeMux()
    gorestfulContainer.Router(restful.CurlyRouter{}) // e.g. for proxy/{kind}/{name}/{*}
    gorestfulContainer.RecoverHandler(func(panicReason interface{}, httpWriter http.ResponseWriter) {
        logStackOnRecover(s, panicReason, httpWriter)
    })
    gorestfulContainer.ServiceErrorHandler(func(serviceErr restful.ServiceError, request *restful.Request, response *restful.Response) {
        serviceErrorHandler(s, serviceErr, request, response)
    })

    director := director{
        name:               name,
        goRestfulContainer: gorestfulContainer,
        nonGoRestfulMux:    nonGoRestfulMux,
    }

    return &APIServerHandler{
        FullHandlerChain:   handlerChainBuilder(director),
        GoRestfulContainer: gorestfulContainer,
        NonGoRestfulMux:    nonGoRestfulMux,
        Director:           director,
    }
}

可以看到，方法调用NewContainer方法初始化了一个gorestfulContainer，还调用了Router方法注册了路由类型为CurlyRouter。这些都与官方https://github.com/emicklei/go-restful/blob/master/examples/restful-curly-router.go上的操作相近。此外，方法还添加了路由选择器Mux，以及RecoverHandler和ServiceErrorHandler两个回调函数。回到New方法，在New方法中通过调用installAPI方法，为API Server预先注册了一些默认path。

installAPI

回到New方法，在New方法中通过调用installAPI方法，为API Server预先注册了一些默认path。

func installAPI(s *GenericAPIServer, c *Config) {
        if c.EnableIndex { //ljs:// 添加"/"与"/index.html"路由
                routes.Index{}.Install(s.listedPathProvider, s.Handler.NonGoRestfulMux)
        } //ljs: 添加"/swagger-ui/"路由
        if c.SwaggerConfig != nil && c.EnableSwaggerUI {
                routes.SwaggerUI{}.Install(s.Handler.NonGoRestfulMux)
        } //ljs: 添加"/debug"相关路由
        if c.EnableProfiling {
                routes.Profiling{}.Install(s.Handler.NonGoRestfulMux)
                if c.EnableContentionProfiling {
                        goruntime.SetBlockProfileRate(1)
                }
                // so far, only logging related endpoints are considered valid to add for these debug flags.
                routes.DebugFlags{}.Install(s.Handler.NonGoRestfulMux, "v", routes.StringFlagPutHandler(logs.GlogSetter))
        } //ljs: 添加"/metrics"路由
        if c.EnableMetrics {
                if c.EnableProfiling {
                        routes.MetricsWithReset{}.Install(s.Handler.NonGoRestfulMux)
                } else {
                        routes.DefaultMetrics{}.Install(s.Handler.NonGoRestfulMux)
                }
        }
        //ljs: 添加"/version"路由
        routes.Version{Version: c.Version}.Install(s.Handler.GoRestfulContainer)

        if c.EnableDiscovery {
                s.Handler.GoRestfulContainer.Add(s.DiscoveryGroupManager.WebService())
        }
}

InstallLegacyAPI

创建了API Server实例，初始化了Container，接下来就是最关键的注册路由的部分了。回到master.go，进入InstallLegacyAPI方法：

//ljs: 用于注册k8s前期的核心API（在/api路径下）func (m *Master) InstallLegacyAPI(c *completedConfig, restOptionsGetter generic.RESTOptionsGetter, legacyRESTStorageProvider corerest.LegacyRESTStorageProvider) {
        // 调用NewLegacyRESTStorage方法，生成RESTStorage和APIGroupInfo。
        legacyRESTStorage, apiGroupInfo, err := legacyRESTStorageProvider.NewLegacyRESTStorage(restOptionsGetter)
        if err != nil {
                klog.Fatalf("Error building core storage: %v", err)
        }

        controllerName := "bootstrap-controller"
        coreClient := corev1client.NewForConfigOrDie(c.GenericConfig.LoopbackClientConfig)
        bootstrapController := c.NewBootstrapController(legacyRESTStorage, coreClient, coreClient, coreClient)
        m.GenericAPIServer.AddPostStartHookOrDie(controllerName, bootstrapController.PostStartHook)
        m.GenericAPIServer.AddPreShutdownHookOrDie(controllerName, bootstrapController.PreShutdownHook)

        if err := m.GenericAPIServer.InstallLegacyAPIGroup(genericapiserver.DefaultLegacyAPIPrefix, &apiGroupInfo); err != nil {
                klog.Fatalf("Error in registering group versions: %v", err)
        }
}

包括以下几步：
（1）通过NewLegacyRESTStorage方法创建各个资源的RESTStorage。RESTStorage是一个结构体，具体的定义在vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go下，结构体内主要包含NewFunc返回特定资源信息、NewListFunc返回特定资源列表、CreateStrategy特定资源创建时的策略、UpdateStrategy更新时的策略以及DeleteStrategy删除时的策略等重要方法。
常见的像event、secret、namespace、endpoints等，统一调用NewREST方法构造相应的资源。待所有资源的store创建完成之后，使用restStorageMap的Map类型将每个资源的路由和对应的store对应起来，方便后续去做路由的统一规划，代码如下：
在NewLegacyRESTStorage内部，可以看到创建了多种资源的RESTStorage。
（2）定义钩子函数。
（3）调用InstallLegacyAPIGroup方法，为API Server注册“v1”路由。

NewLegacyRESTStorage

进入NewLegacyRESTStorage方法

pkg/registry/core/rest/storage_core.go

func (c LegacyRESTStorageProvider) NewLegacyRESTStorage(restOptionsGetter generic.RESTOptionsGetter) (LegacyRESTStorage, genericapiserver.APIGroupInfo, error) {
    apiGroupInfo := genericapiserver.APIGroupInfo{
        PrioritizedVersions:          legacyscheme.Scheme.PrioritizedVersionsForGroup(""),
        VersionedResourcesStorageMap: map[string]map[string]rest.Storage{},
        Scheme:                       legacyscheme.Scheme,
        ParameterCodec:               legacyscheme.ParameterCodec,
        NegotiatedSerializer:         legacyscheme.Codecs,
    }

    var podDisruptionClient policyclient.PodDisruptionBudgetsGetter
    if policyGroupVersion := (schema.GroupVersion{Group: "policy", Version: "v1beta1"}); legacyscheme.Scheme.IsVersionRegistered(policyGroupVersion) {
        var err error
        podDisruptionClient, err = policyclient.NewForConfig(c.LoopbackClientConfig)
        if err != nil {
            return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, err
        }
    }
    restStorage := LegacyRESTStorage{}

    podTemplateStorage := podtemplatestore.NewREST(restOptionsGetter)

    eventStorage := eventstore.NewREST(restOptionsGetter, uint64(c.EventTTL.Seconds()))
    limitRangeStorage := limitrangestore.NewREST(restOptionsGetter)

    resourceQuotaStorage, resourceQuotaStatusStorage := resourcequotastore.NewREST(restOptionsGetter)
    secretStorage := secretstore.NewREST(restOptionsGetter)
    persistentVolumeStorage, persistentVolumeStatusStorage := pvstore.NewREST(restOptionsGetter)
    persistentVolumeClaimStorage, persistentVolumeClaimStatusStorage := pvcstore.NewREST(restOptionsGetter)
    configMapStorage := configmapstore.NewREST(restOptionsGetter)

    namespaceStorage, namespaceStatusStorage, namespaceFinalizeStorage := namespacestore.NewREST(restOptionsGetter)

    endpointsStorage := endpointsstore.NewREST(restOptionsGetter)

    nodeStorage, err := nodestore.NewStorage(restOptionsGetter, c.KubeletClientConfig, c.ProxyTransport)
    if err != nil {
        return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, err
    }

    podStorage := podstore.NewStorage(
        restOptionsGetter,
        nodeStorage.KubeletConnectionInfo,
        c.ProxyTransport,
        podDisruptionClient,
    )

    ...

    restStorageMap := map[string]rest.Storage{
        "pods":             podStorage.Pod,
        "pods/attach":      podStorage.Attach,
        "pods/status":      podStorage.Status,
        "pods/log":         podStorage.Log,
        "pods/exec":        podStorage.Exec,
        "pods/portforward": podStorage.PortForward,
        "pods/proxy":       podStorage.Proxy,
        "pods/binding":     podStorage.Binding,
        "bindings":         podStorage.Binding,

        "podTemplates": podTemplateStorage,

        "replicationControllers":        controllerStorage.Controller,
        "replicationControllers/status": controllerStorage.Status,

        "services":        serviceRest,
        "services/proxy":  serviceRestProxy,
        "services/status": serviceStatusStorage,

        "endpoints": endpointsStorage,

        "nodes":        nodeStorage.Node,
        "nodes/status": nodeStorage.Status,
        "nodes/proxy":  nodeStorage.Proxy,

        "events": eventStorage,

        "limitRanges":                   limitRangeStorage,
        "resourceQuotas":                resourceQuotaStorage,
        "resourceQuotas/status":         resourceQuotaStatusStorage,
        "namespaces":                    namespaceStorage,
        "namespaces/status":             namespaceStatusStorage,
        "namespaces/finalize":           namespaceFinalizeStorage,
        "secrets":                       secretStorage,
        "serviceAccounts":               serviceAccountStorage,
        "persistentVolumes":             persistentVolumeStorage,
        "persistentVolumes/status":      persistentVolumeStatusStorage,
        "persistentVolumeClaims":        persistentVolumeClaimStorage,
        "persistentVolumeClaims/status": persistentVolumeClaimStatusStorage,
        "configMaps":                    configMapStorage,

        "componentStatuses": componentstatus.NewStorage(componentStatusStorage{c.StorageFactory}.serversToValidate),
    }
    if legacyscheme.Scheme.IsVersionRegistered(schema.GroupVersion{Group: "autoscaling", Version: "v1"}) {
        restStorageMap["replicationControllers/scale"] = controllerStorage.Scale
    }
    if legacyscheme.Scheme.IsVersionRegistered(schema.GroupVersion{Group: "policy", Version: "v1beta1"}) {
        restStorageMap["pods/eviction"] = podStorage.Eviction
    }
    if serviceAccountStorage.Token != nil {
        restStorageMap["serviceaccounts/token"] = serviceAccountStorage.Token
    }
    apiGroupInfo.VersionedResourcesStorageMap["v1"] = restStorageMap

    return restStorage, apiGroupInfo, nil
}

InstallLegacyAPIGroup

注册handler

//首先，判断传入的前缀是否合法。//其次，调用installAPIResources方法。也就是为route注册一个handler//最后，在/api路径下生成一个WebService，并添加进Container。func (s *GenericAPIServer) InstallLegacyAPIGroup(apiPrefix string, apiGroupInfo *APIGroupInfo) error {
        // 判断前缀是否合法
        if !s.legacyAPIGroupPrefixes.Has(apiPrefix) {
                return fmt.Errorf("%q is not in the allowed legacy API prefixes: %v", apiPrefix, s.legacyAPIGroupPrefixes.List())
        }
        if err := s.installAPIResources(apiPrefix, apiGroupInfo); err != nil {
                return err
        }

        // Install the version handler.
        // Add a handler at /<apiPrefix> to enumerate the supported api versions.      //ljs：container.Add(ws)
        s.Handler.GoRestfulContainer.Add(discovery.NewLegacyRootAPIHandler(s.discoveryAddresses, s.Serializer, apiPrefix).WebService())

        return nil
}

installAPIResources

vendor/k8s.io/apiserver/pkg/server/genericapiserver.go

func (s *GenericAPIServer) installAPIResources(apiPrefix string, apiGroupInfo *APIGroupInfo) error {
        openAPIGroupModels, err := s.getOpenAPIModelsForGroup(apiPrefix, apiGroupInfo)
        if err != nil {
                return fmt.Errorf("unable to get openapi models for group %v: %v", apiPrefix, err)
        }
        for _, groupVersion := range apiGroupInfo.PrioritizedVersions {
                if len(apiGroupInfo.VersionedResourcesStorageMap[groupVersion.Version]) == 0 {
                        klog.Warningf("Skipping API %v because it has no resources.", groupVersion)
                        continue
                }

                apiGroupVersion := s.getAPIGroupVersion(apiGroupInfo, groupVersion, apiPrefix)
                if apiGroupInfo.OptionsExternalVersion != nil {
                        apiGroupVersion.OptionsExternalVersion = apiGroupInfo.OptionsExternalVersion
                }
                apiGroupVersion.OpenAPIModels = openAPIGroupModels

                if err := apiGroupVersion.InstallREST(s.Handler.GoRestfulContainer); err != nil {
                        return fmt.Errorf("unable to setup API %v: %v", apiGroupInfo, err)
                }
        }

        return nil
}

前面的逻辑类似于对传入的apiGroupInfo进行标准格式化，并循环地遍历所有可用version，包装成为APIGroupVersion对象，并对其调用InstallREST方法，为对象注册Handler。核心在于InstallREST方法。

InstallREST

func (g *APIGroupVersion) InstallREST(container *restful.Container) error {
        prefix := path.Join(g.Root, g.GroupVersion.Group, g.GroupVersion.Version)
        installer := &APIInstaller{
                group:                        g,
                prefix:                       prefix,
                minRequestTimeout:            g.MinRequestTimeout,
                enableAPIResponseCompression: g.EnableAPIResponseCompression,
        }

        apiResources, ws, registrationErrors := installer.Install()
        versionDiscoveryHandler := discovery.NewAPIVersionHandler(g.Serializer, g.GroupVersion, staticLister{apiResources})
        versionDiscoveryHandler.AddToWebService(ws)
        // go-restful, container添加一个webservice
        container.Add(ws)
        return utilerrors.NewAggregate(registrationErrors)
}

方法逻辑是，基于这个GroupVersion对象的元素新建一个前缀字符串，然后建立一个APIInstaller结构体，调用Install方法，然后将返回的WebService添加进Container中。

Install

vendor/k8s.io/apiserver/pkg/endpoints/installer.go

// Install handlers for API resources.func (a *APIInstaller) Install() ([]metav1.APIResource, *restful.WebService, []error) {
    var apiResources []metav1.APIResource
    var errors []error
    ws := a.newWebService()

    // Register the paths in a deterministic (sorted) order to get a deterministic swagger spec.
    paths := make([]string, len(a.group.Storage))
    var i int = 0
    for path := range a.group.Storage {
        paths[i] = path
        i++
    }
    sort.Strings(paths)
    for _, path := range paths {
        apiResource, err := a.registerResourceHandlers(path, a.group.Storage[path], ws)
        if err != nil {
            errors = append(errors, fmt.Errorf("error in registering resource: %s, %v", path, err))
        }
        if apiResource != nil {
            apiResources = append(apiResources, *apiResource)
        }
    }
    return apiResources, ws, errors
}

这个方法的逻辑是，先创建一个WebService，然后将APIInstaller结构体中保存的所有API路径存入数组paths，对数组排序，遍历数组所有元素并调用registerResourceHandlers方法，将路径、对应的storage和WebService建立对应关系。最后，将这些对应关系存入apiResources数组中，并返回APIResource和WebService。

registerResourceHandlers

registerResourceHandlers方法长达数百行，是创建API的核心方法。节选最重要的部分如下：

vendor/k8s.io/apiserver/pkg/endpoints/installer.go

func (a *APIInstaller) registerResourceHandlers(path string, storage rest.Storage, ws *restful.WebService) (*metav1.APIResource, error) {
    
　　　　...

　　　　creater, isCreater := storage.(rest.Creater)
　　　　namedCreater, isNamedCreater := storage.(rest.NamedCreater)
　　　　lister, isLister := storage.(rest.Lister)
　　　　getter, isGetter := storage.(rest.Getter)
　　　　getterWithOptions, isGetterWithOptions := storage.(rest.GetterWithOptions)
　　　　gracefulDeleter, isGracefulDeleter := storage.(rest.GracefulDeleter)
　　　　collectionDeleter, isCollectionDeleter := storage.(rest.CollectionDeleter)
　　　　updater, isUpdater := storage.(rest.Updater)
　　　　patcher, isPatcher := storage.(rest.Patcher)
　　　　watcher, isWatcher := storage.(rest.Watcher)
　　　　connecter, isConnecter := storage.(rest.Connecter)
　　　　storageMeta, isMetadata := storage.(rest.StorageMetadata)

　　　　if !isMetadata {
   　　　　　　storageMeta = defaultStorageMetadata{}
　　　　}
　　　　...


　　　　// Handler for standard REST verbs (GET, PUT, POST and DELETE).
　　　　// Add actions at the resource path: /api/apiVersion/resource
　　　　actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
　　　　actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)
　　　　actions = appendIf(actions, action{"DELETECOLLECTION", resourcePath, resourceParams, namer, false}, isCollectionDeleter)
　　　　
　　　　// Add actions at the item path: /api/apiVersion/resource/{name}
　　　　actions = appendIf(actions, action{"GET", itemPath, nameParams, namer, false}, isGetter)
　　　　if getSubpath {
   　　　　　　actions = appendIf(actions, action{"GET", itemPath + "/{path:*}", proxyParams, namer, false}, isGetter)
　　　　}
　　　　actions = appendIf(actions, action{"PUT", itemPath, nameParams, namer, false}, isUpdater)
　　　　actions = appendIf(actions, action{"PATCH", itemPath, nameParams, namer, false}, isPatcher)
　　　　actions = appendIf(actions, action{"DELETE", itemPath, nameParams, namer, false}, isGracefulDeleter)

　　　　actions = appendIf(actions, action{"CONNECT", itemPath, nameParams, namer, false}, isConnecter)
　　　　actions = appendIf(actions, action{"CONNECT", itemPath + "/{path:*}", proxyParams, namer, false}, isConnecter && connectSubpath)

　　　　...
　　　　
　　　　routes := []*restful.RouteBuilder{}

　　　　case "GET": // Get a resource.
　　　　　　　　var handler restful.RouteFunction
　　　　　　　　if isGetterWithOptions {
   　　　　　　　　　　handler = restfulGetResourceWithOptions(getterWithOptions, reqScope, isSubresource)
　　　　　　　　} else {
   　　　　　　　　　　handler = restfulGetResource(getter, exporter, reqScope)
　　　　　　　　}

　　　　　　　　if needOverride {
   　　　　　　　　　　// need change the reported verb
　　　　　　　　　　　　handler = metrics.InstrumentRouteFunc(verbOverrider.OverrideMetricsVerb(action.Verb), group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
　　　　　　　　} else {
　　　　　　　　　　　　handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
　　　　　　　　}

　　　　　　　　if a.enableAPIResponseCompression {
   　　　　　　　　　　handler = genericfilters.RestfulWithCompression(handler)
　　　　　　　　}
　　　　　　　　doc := "read the specified " + kind
　　　　　　　　if isSubresource {
   　　　　　　　　　　doc = "read " + subresource + " of the specified " + kind
　　　　　　　　}
　　　　　　　　route := ws.GET(action.Path).To(handler).Doc(doc).
　　　　　　　　Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
　　　　　　　　Operation("read"+namespaced+kind+strings.Title(subresource)+operationSuffix).
　　　　　　　　Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
　　　　　　　　Returns(http.StatusOK, "OK", producedObject).Writes(producedObject)
　　　　　　　　...
　　　　　　　　routes = append(routes, route)

　　　　...

　　　　
　　　　for _, route := range routes {
   　　　　　　route.Metadata(ROUTE_META_GVK, metav1.GroupVersionKind{
      　　　　　　　　Group:   reqScope.Kind.Group,
      　　　　　　　　Version: reqScope.Kind.Version,
      　　　　　　　　Kind:    reqScope.Kind.Kind,
   　　　　　　})
   　　　　　　route.Metadata(ROUTE_META_ACTION, strings.ToLower(action.Verb))
   　　　　　　ws.Route(route)
　　　　}
　　　　...
　　　　return &apiResource, nil
}

代码虽长，逻辑很清晰。大致的执行逻辑是：
1.判断storage是否支持create、list、get等方法，并对所有支持的方法进行进一步的处理，如if !isMetadata这一块一样，内容过多不一一贴出；
2.将所有支持的方法存入actions数组中；
3.遍历actions数组，在一个switch语句中，为所有元素定义路由。如贴出的case "GET"这一块，首先创建并包装一个handler对象，然后调用WebService的一系列方法，创建一个route对象，将handler绑定到这个route上。后面还有case "PUT"、case "DELETE"等一系列case，不一一贴出。最后，将route加入routes数组中。
4.遍历routes数组，将route加入WebService中。
5.最后，返回一个APIResource结构体。
这样，Install方法就通过调用registerResourceHandlers方法，完成了WebService与APIResource的绑定。至此，InstallLegacyAPI方法的逻辑就分析完了。总的来说，这个方法遵循了go-restful的设计模式，在/api路径下注册了WebService，并将WebService加入Container中。

路由添加(apis开头)

api开头的路由主要是对基础资源的路由实现，而对于其他附加的资源，如认证相关、网络相关等各种扩展的api资源，统一以apis开头命名，实现入口为InstallAPIs。InstallAPIs与InstallLegacyAPIGroup主要的区别是获取RESTStorage的方式。对于api开头的路由来说，都是/api/v1这种统一的格式；而对于apis开头路由则不一样，它包含了多种不同的格式（Kubernetes代码内叫groupName），如/apis/apps、/apis/certificates.k8s.io等各种无规律的groupName。为此，kubernetes提供了一种RESTStorageProvider的工厂模式的接口

// RESTStorageProvider is a factory type for REST storage.type RESTStorageProvider interface {
        GroupName() string
        NewRESTStorage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) (genericapiserver.APIGroupInfo, bool)
}

Server端启动

通过CreateServerChain创建完server后，继续调用GenericAPIServer的Run方法完成最终的启动工作。首先通过PrepareRun方法完成启动前的路由收尾工作，该方法主要完成了Swagger和OpenAPI路由的注册工作（Swagger和OpenAPI主要包含了Kubernetes API的所有细节与规范），并完成/healthz路由的注册工作。完成后，开始最终的server启动工作。Run方法里通过NonBlockingRun方法启动安全的http server（非安全方式的启动在CreateServerChain方法已经完成）

// Run spawns the secure http server. It only returns if stopCh is closed// or the secure port cannot be listened on initially.// Run方法会创建一个安全的http server。只有在stopCh关闭或最初无法监听安全端口时返回func (s preparedGenericAPIServer) Run(stopCh <-chan struct{}) error {
        // NonBlockingRun创建一个安全的http server
        err := s.NonBlockingRun(stopCh)
        if err != nil {
                return err
        }

        <-stopCh

        // 接收到stopCh之后的处理动作
        err = s.RunPreShutdownHooks()
        if err != nil {
                return err
        }

        // Wait for all requests to finish, which are bounded by the RequestTimeout variable.
        s.HandlerChainWaitGroup.Wait()

        return nil
}

启动主要工作包括配置各种证书认证、时间参数、报文大小参数之类，之后通过调用net/http库的启动方式启动，代码比较简洁，不一一列出了。

权限相关

ApiServer中与权限相关的主要有三种机制，即常用的认证、鉴权和准入控制。对apiserver来说，主要提供的就是rest风格的接口，所以各种权限最终还是集中到对接口的权限判断上。
以最核心的kubeAPIServerConfig举例，在CreateServerChain方法中，调用了CreateKubeAPIServerConfig的方法，该方法主要的作用是创建kubeAPIServer的配置。进入该方法，调用了buildGenericConfig创建一些通用的配置，在NewConfig下，返回了DefaultBuildHandlerChain，该方法主要就是用来对apiserver rest接口的链式判断，即俗称的filter操作，先记录下，后续分析。

func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler {
        handler := genericapifilters.WithAuthorization(apiHandler, c.Authorization.Authorizer, c.Serializer)
        handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc)
        handler = genericapifilters.WithImpersonation(handler, c.Authorization.Authorizer, c.Serializer)
        handler = genericapifilters.WithAudit(handler, c.AuditBackend, c.AuditPolicyChecker, c.LongRunningFunc)
        failedHandler := genericapifilters.Unauthorized(c.Serializer, c.Authentication.SupportsBasicAuth)
        failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, c.AuditBackend, c.AuditPolicyChecker)
        handler = genericapifilters.WithAuthentication(handler, c.Authentication.Authenticator, failedHandler, c.Authentication.APIAudiences)
        handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true")
        handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc, c.RequestTimeout)
        handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup)
        handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver)
        handler = genericfilters.WithPanicRecovery(handler)
        return handler
}

RBAC启动

Kubernetes中比较重要的用的比较多的可能就是RBAC了。在DefaultBuildHandlerChain方法内，通过调用genericapifilters.WithAuthorization方法，实现对每个接口的权限的filter操作。WithAuthorization方法如下

func WithAuthorization(handler http.Handler, a authorizer.Authorizer, s runtime.NegotiatedSerializer) http.Handler {
        if a == nil {
                klog.Warningf("Authorization is disabled")
                return handler
        }
        return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
                ctx := req.Context()
                ae := request.AuditEventFrom(ctx)

                attributes, err := GetAuthorizerAttributes(ctx)
                if err != nil {
                        responsewriters.InternalError(w, req, err)
                        return
                }
                authorized, reason, err := a.Authorize(attributes)
                // an authorizer like RBAC could encounter evaluation errors and still allow the request, so authorizer decision is checked before error here.
                if authorized == authorizer.DecisionAllow {
                        audit.LogAnnotation(ae, decisionAnnotationKey, decisionAllow)
                        audit.LogAnnotation(ae, reasonAnnotationKey, reason)
                        handler.ServeHTTP(w, req)
                        return
                }
                if err != nil {
                        audit.LogAnnotation(ae, reasonAnnotationKey, reasonError)
                        responsewriters.InternalError(w, req, err)
                        return
                }

                klog.V(4).Infof("Forbidden: %#v, Reason: %q", req.RequestURI, reason)
                audit.LogAnnotation(ae, decisionAnnotationKey, decisionForbid)
                audit.LogAnnotation(ae, reasonAnnotationKey, reason)
                responsewriters.Forbidden(ctx, attributes, w, req, reason, s)
        })
}

1、调用GetAuthorizerAttributes方法获取配置的各种属性值；
2、调用Authorize方法判断权限是否通过，不同的权限实现其接口，完成鉴权任务；
3、如果鉴权成功通过，则调用handler.ServeHTTP方法继续下一步的filter操作；否则，直接返回错误信息。
以RBAC为例，Authorize方法最终调用VisitRulesFor方法实现权限的判断，方法在kubernetes/pkg/registry/rbac/validation/rule.go文件内。

数据库操作

ApiServer与数据库的交互主要指的是与etcd的交互。Kubernetes所有的组件不直接与etcd交互，都是通过请求apiserver，apiserver与etcd进行交互完成数据的最终落盘。
在之前的路由实现已经说过，apiserver最终实现的handler对应的后端数据是以Store的结构保存的。这里以api开头的路由举例，在NewLegacyRESTStorage方法中，通过NewREST或者NewStorage会生成各种资源对应的Storage，以endpoints为例，生成的方法如下

func NewREST(optsGetter generic.RESTOptionsGetter) *REST {
        store := &genericregistry.Store{
                NewFunc:                  func() runtime.Object { return &api.Endpoints{} },
                NewListFunc:              func() runtime.Object { return &api.EndpointsList{} },
                DefaultQualifiedResource: api.Resource("endpoints"),

                CreateStrategy: endpoint.Strategy,
                UpdateStrategy: endpoint.Strategy,
                DeleteStrategy: endpoint.Strategy,

                TableConvertor: printerstorage.TableConvertor{TablePrinter: printers.NewTablePrinter().With(printersinternal.AddHandlers)},
        }
        options := &generic.StoreOptions{RESTOptions: optsGetter}
        if err := store.CompleteWithOptions(options); err != nil {
                panic(err) // TODO: Propagate error up
        }
        return &REST{store}
}

对etcd操作更具体的可以参考这篇文章。
https://www.jianshu.com/p/daa4ff387a78