Install Kubernetes From a Scratch

Install Kubernetes From a Scratch

一个完整的kubernetes服务需要许多组件支持. 比如我们需要安装kubelet, api server, scheduler, kube-proxy等等.

不过除了kubelet, docker需要通过system service来启动外, 其他的组件,比如api server, scheduler, etcd等等都可以通过static pod的方式, 让kubernetes本身来为我们启动.

本文着重介绍通过static pod如何启动kubernetes 服务, 并添加一个add-on

节点角色介绍

Master :

Master节点上运行了特定的组件提供集群的管理控制中心,如api-server。
在Master节点上不会运行用户pod/service.

Slave:

区别于Master,除了kubelet, kube-proxy等服务, 还运行用户Pod/Service

准备工作

首先每台虚拟机将要关闭防火墙和SELinux, 同时安装docker服务, 同时生成相关的CA证书, 这里将不在赘述. 具体可参考 [ 部署开源k8s-1.8.5方法]

安装kubelet服务

通过以下kubelete.service文件, 在所有节点启动kubelet服务:

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/bin/kubelet \
  --address=10.0.10.3 \ ## kublet所服务的ip地址
  --hostname-override=10.0.10.3 \
  --pod-infra-container-image=172.16.1.41:5000/google_containers/pause:3.0 \
  --kubeconfig=/etc/kubernetes/kubeconfig \ ##指定kubeconfig的地址
  --require-kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \ ##制定CA证书地址
  --cluster-dns=10.10.10.10 \
  --cluster-domain=cluster.local \
  --hairpin-mode=promiscuous-bridge \
  --allow-privileged=true \
  --logtostderr=true \
  --fail-swap-on=false \
  --pod-manifest-path=/etc/kubernetes/manifests \ ##制定kubelet监控的pod manifests的地址(重要)
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

关于kubelet的配置项可参考 [ kubelet官方配置文档]

其中–pod-manifest-path制定kubelet监控的pod manifest的路径

kubelet 会每隔几秒钟(20s)去查看该路径下的pod manifest, 会去启动或更新对应的pod. 常见的Pod manifest格式为yaml或者jason, 官方推荐yaml

执行以下命令,启动kubelet

cp kubelet.service /usr/lib/systemd/system/
systemctl daemon-reload && systemctl restart kubelet

配置kubeconfig

kubeConfig定义了kubelet如何访问集群的配置, 其中包含了cluster, user的信息.具体配置如下:

apiVersion: v1
kind: Config
clusters:
  - cluster:
      certificate-authority: /etc/kubernetes/ssl/ca.pem
      server: https://10.0.10.3:6443
    name: kubernetes
contexts:
  - context:
      cluster: kubernetes
      user: k8s
    name: kubelet-to-kubernetes
current-context: kubelet-to-kubernetes
users:
  - name: k8s
    user:
      client-certificate: /etc/kubernetes/ssl/admin.pem
      client-key: /etc/kubernetes/ssl/admin-key.pemc

上述的kubeConfig定义了一个cluster叫kubernetes, 该集群的server地址, ca证书由clusters域定义.

user定义了用户认证的相关信息. context定义了两者的关系.

在所有节点完成上述配置我们就可以生成对应的Manifest来启动各个组件服务

配置对应组件的Pod Manifest

当docker以及kubelet服务已经启动完毕后,我们只需要生成etcd, kube-controller-manager, kube-proxy, kube-apiserver, kube-scheduler的manifest,并放到刚才kubelet监控的目录, 也就是/etc/kubernetes/manifests.

在Master节点上我们需要配置以下Pod Manifest:

etcd.manifests
kube-apiserver.manifests
kube-controller-manager.manifests
kube-scheduler.manifests

在所有节点我们需要配置:

kube-proxy.manifest

etcd.manifest

apiVersion: v1
kind: Pod
metadata:
  name: k8s-etcd
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: etcd
    image: 172.16.1.41:5000/coreos/etcd:3.2.11
    imagePullPolicy: IfNotPresent
    command:
    - etcd
    - --data-dir=/var/etcd/data
    env:
    - name: ETCD_INITIAL_CLUSTER_STATE  
      value: new
    - name: ETCD_NAME
      value: ETCD0
    - name: ETCD_INITIAL_ADVERTISE_PEER_URLS
      value: http://10.0.10.3:2380
    - name: ETCD_INITIAL_CLUSTER
      value: ETCD0=http://10.0.10.3:2380
    - name: ETCD_INITIAL_CLUSTER_TOKEN
      value: token
    - name: ETCD_LISTEN_PEER_URLS
      value: http://0.0.0.0:2380
    - name: ETCD_ADVERTISE_CLIENT_URLS
      value: http://10.0.10.3:2379
    - name: ETCD_LISTEN_CLIENT_URLS
      value: http://0.0.0.0:2379
    volumeMounts:
    - name: varetcd
      mountPath: /var/etcd
      readOnly: false
  volumes:
  - name: varetcd
    hostPath:
      path: /var/etcd/data

关于ETCD的参数配置,可参考 [ ETCD部署]

kube-apiserver.manifest

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
    - name: apiserver
      image: 172.16.1.41:5000/google_containers/kube-apiserver:v1.8.5
      imagePullPolicy: IfNotPresent
      command:
        - kube-apiserver
        - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
        - --advertise-address=10.0.10.3
        - --allow-privileged=true
        - --audit-log-maxage=30
        - --audit-log-maxbackup=3
        - --audit-log-maxsize=100
        - --audit-log-path=/var/log/kubernetes/audit.log
        - --authorization-mode=RBAC
        - --bind-address=10.0.10.3
        - --client-ca-file=/etc/kubernetes/ssl/ca.pem
        - --enable-swagger-ui=true
        - --etcd-servers=http://10.0.10.3:2379
        - --event-ttl=1h
        - --kubelet-https=true
        - --insecure-bind-address=0.0.0.0
        - --runtime-config=rbac.authorization.k8s.io/v1alpha1
        - --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem
        - --service-cluster-ip-range=10.10.10.1/24
        - --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem
        - --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem
        - --v=2
      volumeMounts:
      - name: apikubelog
        mountPath: /var/log/kubernetes/
        readOnly: false
      - name: kubernetes
        mountPath: /var/run/kubernetes/
        readOnly: false
      - name: etckube
        mountPath: /etc/kubernetes/
        readOnly: false
  volumes:
    - name: apikubelog
      hostPath:
        path: /var/log/kubernetes/apiserver
    - name: kubernetes
      hostPath:
        path: /var/run/kubernetes/
    - name: etckube
      hostPath:
        path: /etc/kubernetes/

参数解释

– admission-control
定义admission controller(准入)插件, api server会对相应的请求进行校验, 检查请求是否符合插件定义的规则. 详细请参考Adimission Controller

– advertise-address:
广播API Server给所有集群成员的IP地址

–client-ca-file
如果设置后,任何带ca认证的请求都会用指定的ca file进行的比对

–etcd-servers
指定etcd server地址

–kubelet-https
是否使用https建立连接

–service-cluster-ip-range
CIDR标记的IP范围，从中分配IP给服务集群

更多设置解释请参考kube-apiserver配置

kube-controller-manager.manifest

kube-controller-manager是一个daemon, 它会通过api-server不停的轮询监控集群的状态,并在适当的情况下将集群状态迁移到期望的状态
他的manifest如下:

apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
    - name: kube-controller-manager
      image: 172.16.1.41:5000/google_containers/kube-controller-manager:v1.8.5
      imagePullPolicy: IfNotPresent
      command:
        - kube-controller-manager
        - --kubeconfig=/etc/kubernetes/kubeconfig ##指定kubeConfig所在地址
        - --cluster-name=kubernetes
        - --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem ##部署服务所需的秘钥文件
        - --root-ca-file=/etc/kubernetes/ssl/ca.pem
        - --v=2
      volumeMounts:
      - name: kubelog
        mountPath: /var/log/kubernetes/
        readOnly: false
      - name: kubernetes
        mountPath: /var/run/kubernetes/
        readOnly: false
      - name: etckube
        mountPath: /etc/kubernetes/
        readOnly: false
  volumes:
    - name: kubelog
      hostPath:
        path: /var/log/kubernetes/apiserver
    - name: kubernetes
      hostPath:
        path: /var/run/kubernetes/
    - name: etckube
      hostPath:
        path: /etc/kubernetes/

kube-scheduler.manifests

kube-scheduler 监视新创建没有分配到Node的Pod，为Pod选择一个Node

apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
    - name: kube-scheduler
      image: 172.16.1.41:5000/google_containers/kube-scheduler:v1.8.5
      imagePullPolicy: IfNotPresent
      command:
        - kube-scheduler
        - --kubeconfig=/etc/kubernetes/kubeconfig
        - --v=2
      volumeMounts:
      - name: kubelog
        mountPath: /var/log/kubernetes/
        readOnly: false
      - name: kubernetes
        mountPath: /var/run/kubernetes/
        readOnly: false
      - name: etckube
        mountPath: /etc/kubernetes/
        readOnly: false
  volumes:
    - name: kubelog
      hostPath:
        path: /var/log/kubernetes/apiserver
    - name: kubernetes
      hostPath:
        path: /var/run/kubernetes/
    - name: etckube
      hostPath:
        path: /etc/kubernetes/

将以上manifest拷贝到Master节点的kubelet所监控的目录/etc/kubernetes/manifests中

kube-proxy.manifest

在所有节点我们还需要创建kube-proxy.manifest:

kube-proxy通过在主机上维护网络规则并执行连接转发来实现Kubernetes服务抽象

apiVersion: v1
kind: Pod
metadata:
  name: k8s-proxy
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-proxy
    image: 172.16.1.41:5000/google_containers/kube-proxy:v1.8.5
    imagePullPolicy: Always
    command:
    - proxy
    - --kubeconfig=/etc/kubernetes/kubeconfig
    - --v=2
    - --masquerade-all ##对所有service ip
    - --resource-container=""
    securityContext:
      privileged: true
    volumeMounts:
    - name: apikubelog
      mountPath: /var/log/kubernetes/
      readOnly: false
    - name: kubernetes
      mountPath: /var/run/kubernetes/
      readOnly: false
    - name: etckube
      mountPath: /etc/kubernetes/
      readOnly: false
  volumes:
    - name: apikubelog
      hostPath:
        path: /var/log/kubernetes/apiserver
    - name: kubernetes
      hostPath:
        path: /var/run/kubernetes/
    - name: etckube
      hostPath:
        path: /etc/kubernetes/

配置好所有的组件的Manifest, 一个kubernetes集群就启动了, 在本文中有一个master, 1个slave.

如何添加插件Add-on

待添加