kubernetes从零到有,集群部署使用

互相交流

此文档为本人搭建过程中记录,基本按照文档来。都能够搭建出自己的k8s集群,如果按照文档还不能解决问题,那么可以通过262735860@qq.com进行交流。

集群环境部署


master: kube-apiserver\kube-controller-manager\kube-scheduler
node: kube-proxy\kubelet

集群机器

master:10.1.0.71
node1:10.1.0.78
node2:10.1.0.74(后续增加node)

**由于部署过程中需要声明环境变量及生成一系列证书文件,建议大家找一个部署工作目录,笔者此套部署集群的工作目录为/home/k8s-1.6/file(自行创建)**

定义环境变量

cat /home/k8s-1.6/file/environment.sh

export BOOTSTRAP_TOKEN="01139e361493d869eeb3704432e036ff"
# TLS Bootstrapping 使用的 Token,可以使用命令 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成
export SERVICE_CIDR="10.254.0.0/16"
# 建议用 未用的网段 来定义服务网段和 Pod 网段
# 服务网段 (Service CIDR),部署前路由不可达,部署后集群内使用 IP:Port 可达
export CLUSTER_CIDR="172.30.0.0/16"
# POD 网段 (Cluster CIDR),部署前路由不可达,**部署后**路由可达 (flanneld 保证)
export NODE_PORT_RANGE="8400-9000"
# 服务端口范围 (NodePort Range)
export ETCD_ENDPOINTS="https://10.1.0.71:2379,https://10.1.0.78:2379"
# etcd 集群服务地址列表
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
# flanneld 网络配置前缀
export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"
# kubernetes 服务 IP (预分配,一般是 SERVICE_CIDR 中第一个IP)
export CLUSTER_DNS_SVC_IP="10.254.0.2"
# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
export CLUSTER_DNS_DOMAIN="cluster.local."
# 集群 DNS 域名
export NODE_NAME=etcd-host0
 # 当前部署的机器名称(随便定义,只要能区分不同机器即可)
export NODE_IP=10.1.0.71
# 当前部署的机器 IP
export NODE_IPS="10.1.0.71 10.1.0.78"
 # etcd 集群所有机器 IP
export ETCD_NODES=etcd-host0=https://10.1.0.71:2380,etcd-host1=https://10.1.0.78:2380
 # 导入用到的其它全局变量:ETCD_ENDPOINTS、FLANNEL_ETCD_PREFIX、CLUSTER_CIDR
export MASTER_IP=10.1.0.71
# 替换为 kubernetes master 集群机器 IP
export KUBE_APISERVER="https://${MASTER_IP}:6443"

执行文件使环境变量生效

source /home/k8s-1.6/file/environment.sh

制造TSL证书


kubernetes 系统各组件需要使用 TLS 证书对通信进行加密,本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 证书和秘钥文件,CA 是自签名的证书,用来签名后续创建的其它 TLS 证书。

安装CFSSL

$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
$ chmod +x cfssl_linux-amd64
$ sudo mv cfssl_linux-amd64 /bin/cfssl

$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
$ chmod +x cfssljson_linux-amd64
$ sudo mv cfssljson_linux-amd64 /bin/cfssljson

$ wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
$ chmod +x cfssl-certinfo_linux-amd64
$ sudo mv cfssl-certinfo_linux-amd64 /bin/cfssl-certinfo

$ cfssl print-defaults config > config.json
$ cfssl print-defaults csr > csr.json
集群搭建中可以先在master上生成所有证书,然后再将node需要的证书拷贝过去,集群中所需要的证书:(具体证书生成内容及步骤在后面体现)
[root@LOC-XAN-PRO-032 ssl]# ll *.pem
-rw------- 1 root root 1679 89 19:15 admin-key.pem
-rw-r--r-- 1 root root 1399 89 19:15 admin.pem
-rw------- 1 root root 1675 89 17:21 ca-key.pem
-rw-r--r-- 1 root root 1359 89 17:21 ca.pem
-rw------- 1 root root 1675 89 19:12 etcd-key.pem
-rw-r--r-- 1 root root 1436 89 19:12 etcd.pem
-rw------- 1 root root 1679 89 19:22 flanneld-key.pem
-rw-r--r-- 1 root root 1391 89 19:22 flanneld.pem
-rw------- 1 root root 1675 810 10:10 harbor-key.pem
-rw-r--r-- 1 root root 1419 810 10:10 harbor.pem
-rw------- 1 root root 1675 89 20:11 kube-proxy-key.pem
-rw-r--r-- 1 root root 1403 89 20:11 kube-proxy.pem
-rw------- 1 root root 1679 89 19:41 kubernetes-key.pem
-rw-r--r-- 1 root root 1627 89 19:41 kubernetes.pem

生成ca证书和密钥

$ cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}
EOF

$ cat > ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ ls ca*
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

将生成的 CA 证书、秘钥文件、配置文件拷贝到所有机器的 /etc/kubernetes/ssl 目录下

$ sudo mkdir -p /etc/kubernetes/ssl
$ sudo cp ca* /etc/kubernetes/ssl

etcd集群


kuberntes 系统使用 etcd 存储所有数据,笔者部署一个双节点高可用 etcd 集群的步骤,这两个节点复用 kubernetes master 机器,分别命名为etcd-host0、etcd-host1:
etcd-host0:10.1.0.71
etcd-host1:10.1.0.78

下载etcd二进制文件

$ wget https://github.com/coreos/etcd/releases/download/v3.1.6/etcd-v3.1.6-linux-amd64.tar.gz
$ tar -xvf etcd-v3.1.6-linux-amd64.tar.gz
$ sudo mv etcd-v3.1.6-linux-amd64/etcd* /bin

生成etcd证书和私钥

$ cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.1.0.71",
    "10.1.0.78"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

**#需要注意的是在写etcd-csr.json时,hosts内容需要包含集群的ip**。

$ cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem -config=./ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
$ ls etcd*
etcd.csr  etcd-csr.json  etcd-key.pem etcd.pem
$ sudo mkdir -p /etc/etcd/ssl
$ sudo cp etcd*.pem /etc/etcd/ssl

将生成的etcd证书和密钥文件拷贝到etcd集群的各个机器的/etc/etcd/ssl目录下

$ sudo mkdir -p /etc/etcd/ssl
$ sudo cp etcd*.pem /etc/etcd/ssl

创建etcd.service服务

$ sudo mkdir -p /var/lib/etcd  **# 必须先创建工作目录**
$ cat > etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/etcd \\
  --name=${NODE_NAME} \\
  --cert-file=/etc/etcd/ssl/etcd.pem \\
  --key-file=/etc/etcd/ssl/etcd-key.pem \\
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\
  --initial-advertise-peer-urls=https://${NODE_IP}:2380 \\
  --listen-peer-urls=https://${NODE_IP}:2380 \\
  --listen-client-urls=https://${NODE_IP}:2379,http://127.0.0.1:2379 \\
  --advertise-client-urls=https://${NODE_IP}:2379 \\
  --initial-cluster-token=etcd-cluster-0 \\
  --initial-cluster=${ETCD_NODES} \\
  --initial-cluster-state=new \\
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

**#需要注意的是,要先启动master下etcd服务,再启动node上的etcd服务,否则会有报错。
#一般存在报错时,检查环境变量的NODE_NAME、NODE_IP以及/var/lib/etcd是否存在。**

启动etcd服务

$ sudo mv etcd.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable etcd
$ sudo systemctl start etcd
$ systemctl status etcd

验证etcd服务

for ip in ${NODE_IPS}; do
  ETCDCTL_API=3 /bin/etcdctl \
  --endpoints=https://${ip}:2379  \
  --cacert=/etc/kubernetes/ssl/ca.pem \
  --cert=/etc/etcd/ssl/etcd.pem \
  --key=/etc/etcd/ssl/etcd-key.pem \
  endpoint health; done

预期结果

2017-11-07 17:58:15.954431 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
https://10.1.0.71:2379 is healthy: successfully committed proposal: took = 2.9803ms
2017-11-07 17:58:16.043153 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
https://10.1.0.78:2379 is healthy: successfully committed proposal: took = 3.154291ms

部署kubectl


下载kubectl

$ wget https://dl.k8s.io/v1.6.2/kubernetes-client-linux-amd64.tar.gz
$ tar -xzvf kubernetes-client-linux-amd64.tar.gz
$ sudo cp kubernetes/client/bin/kube* /bin/
$ chmod a+x /bin/kube*

#生成damin证书
$ cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF


$ cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem -config=./ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
$ ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem

将生成好的admin*.pem证书拷贝到集群节点/etc/kubernetes/ssl下

$ sudo cp admin*.pem /etc/kubernetes/ssl/

创建 kubectl kubeconfig 文件

$ **# 设置集群参数**
$ kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER}
$ **# 设置客户端认证参数**
$ kubectl config set-credentials admin \
  --client-certificate=/etc/kubernetes/ssl/admin.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/admin-key.pem
$ **# 设置上下文参数**
$ kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=admin
$ **# 设置默认上下文**
$ kubectl config use-context kubernetes
**#分发 kubeconfig 文件:将 ~/.kube/config 文件拷贝到运行 kubelet 命令的机器的 ~/.kube/ 目录下。**

部署flanneld集群


kubernetes 要求集群内各节点能通过 Pod 网段互联互通,需要在所有节点 (Master、Node) 上创建互联互通的 Pod 网段

生成flanneld证书

$ cat > flanneld-csr.json <<EOF
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

$ cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem -config=./ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

$ ls flanneld*
flanneld.csr  flanneld-csr.json  flanneld-key.pem flanneld.pem

将生成的flanneld*.pem证书拷贝到集群节点/etc/flanneld/ssl下

$ sudo mkdir -p /etc/flanneld/ssl
$ sudo cp flanneld*.pem /etc/flanneld/ssl

向 etcd 写入集群 Pod 网段信息(本步骤只需在第一次部署 Flannel 网络时执行,后续在其它节点上部署 Flannel 时无需再写入该信息!)

$ /bin/etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/flanneld/ssl/flanneld.pem \
  --key-file=/etc/flanneld/ssl/flanneld-key.pem \
  set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'

下载安装flanneld

$ wget https://github.com/coreos/flannel/releases/download/v0.7.1/flannel-v0.7.1-linux-amd64.tar.gz
$ tar -xzvf flannel-v0.7.1-linux-amd64.tar.gz
$ sudo cp ./{flanneld,mk-docker-opts.sh} /bin

创建 flanneld 的 systemd unit 文件

$ cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/ssl/ca.pem \\
  -etcd-certfile=/etc/flanneld/ssl/flanneld.pem \\
  -etcd-keyfile=/etc/flanneld/ssl/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX}
ExecStartPost=/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

$ sudo mv flanneld.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable flanneld
$ sudo systemctl start flanneld
$ systemctl status flanneld

验证服务

$ journalctl  -u flanneld |grep 'Lease acquired'

预期结果

Nov 07 17:53:37 LOC-XAN-TEST-071 flanneld[43520]: I1107 17:53:37.228939   43520 manager.go:250] Lease acquired: 172.30.16.0/24
Nov 07 17:56:24 LOC-XAN-TEST-071 flanneld[43732]: I1107 17:56:24.061383   43732 manager.go:250] Lease acquired: 172.30.16.0/24

检查分配给各 flanneld 的 Pod 网段信息

$ **# 查看集群 Pod 网段(/16)**
$ /bin/etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/flanneld/ssl/flanneld.pem \
  --key-file=/etc/flanneld/ssl/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/config

  2017-11-07 18:01:16.414771 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated

$ **# 查看已分配的 Pod 子网段列表(/24)**
$ /bin/etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/flanneld/ssl/flanneld.pem \
  --key-file=/etc/flanneld/ssl/flanneld-key.pem \
  ls ${FLANNEL_ETCD_PREFIX}/subnets

2017-11-07 18:02:01.873705 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated

$ **# 查看某一 Pod 网段对应的 flanneld 进程监听的 IP 和网络参数**
$ /bin/etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/flanneld/ssl/flanneld.pem \
  --key-file=/etc/flanneld/ssl/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.16.0-24

  2017-11-07 18:02:50.694103 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
{"PublicIP":"10.1.0.71","BackendType":"vxlan","BackendData":{"VtepMAC":"be:3f:33:a0:15:b9"}}

部署master节点


kubernetes master 节点包含的组件:
kube-apiserver
kube-scheduler
kube-controller-manager

#**下载安装kubernetes有两种方法:**
#######1########
$ wget https://github.com/kubernetes/kubernetes/releases/download/v1.6.2/kubernetes.tar.gz
$ tar -xzvf kubernetes.tar.gz
...
$ cd kubernetes
$ ./cluster/get-kube-binaries.sh

#######2#########
$ # wget https://dl.k8s.io/v1.6.2/kubernetes-client-linux-amd64.tar.gz
$ wget https://dl.k8s.io/v1.6.2/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
...
$ cd kubernetes
$ tar -xzvf  kubernetes-src.tar.gz


#######
$ sudo cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /bin/

生成kubernetes证书

$ cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "${MASTER_IP}",
    "${CLUSTER_KUBERNETES_SVC_IP}",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

**#需要注意的是hosts字段内容:(当多master节点时,hosts字段应该列出对应的ip列表)**
$ cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem -config=./ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

$ ls kubernetes*
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem

将实现生成的kubernetes*.pem拷贝到节点/etc/kubernetes/ssl下

$ sudo cp kubernetes*.pem /etc/kubernetes/ssl/

创建 kube-apiserver 使用的客户端 token 文件
kubelet 首次启动时向 kube-apiserver 发送 TLS Bootstrapping 请求,kube-apiserver 验证 kubelet 请求中的 token 是否与它配置的 token.csv 一致,如果一致则自动为 kubelet生成证书和秘钥。

$ **# 导入的 environment.sh 文件定义了 BOOTSTRAP_TOKEN 变量**
$ cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
$ cp token.csv /etc/kubernetes/

创建 kube-apiserver 的 systemd unit 文件

$ cat  > kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/bin/kube-apiserver \\
  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --advertise-address=${MASTER_IP} \\
  --bind-address=${MASTER_IP} \\
  --insecure-bind-address=${MASTER_IP} \\
  --authorization-mode=RBAC \\
  --runtime-config=rbac.authorization.k8s.io/v1alpha1 \\
  --kubelet-https=true \\
  --experimental-bootstrap-token-auth \\
  --token-auth-file=/etc/kubernetes/token.csv \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --service-node-port-range=${NODE_PORT_RANGE} \\
  --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\
  --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \\
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \\
  --etcd-cafile=/etc/kubernetes/ssl/ca.pem \\
  --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \\
  --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \\
  --etcd-servers=${ETCD_ENDPOINTS} \\
  --enable-swagger-ui=true \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/var/lib/audit.log \\
  --event-ttl=1h \\
  --v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

启动kube-apiserver服务

$ sudo cp kube-apiserver.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable kube-apiserver
$ sudo systemctl start kube-apiserver
$ sudo systemctl status kube-apiserver

创建 kube-controller-manager 的 systemd unit 文件

$ cat > kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/bin/kube-controller-manager \\
  --address=127.0.0.1 \\
  --master=http://${MASTER_IP}:8080 \\
  --allocate-node-cidrs=true \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --cluster-cidr=${CLUSTER_CIDR} \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \\
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \\
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \\
  --leader-elect=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

启动kube-controller-manager服务

$ sudo mv kube-controller-manager.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable kube-controller-manager
$ sudo systemctl start kube-controller-manager

创建 kube-scheduler 的 systemd unit 文件

$ cat > kube-scheduler.service <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/bin/kube-scheduler \\
  --address=127.0.0.1 \\
  --master=http://${MASTER_IP}:8080 \\
  --leader-elect=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

启动kube-scheduler服务

$ sudo cp kube-scheduler.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable kube-scheduler
$ sudo systemctl start kube-scheduler

验证master节点功能

$ kubectl get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health": "true"}
etcd-1               Healthy   {"health": "true"}

部署node节点


kubernetes Node 节点包含如下组件:
flanneld
docker
kubelet
kube-proxy

下载安装docker(建议不要使用ce版本)

$ wget https://get.docker.com/builds/Linux/x86_64/docker-1.13.1.tgz
$ tar -xvf docker-1.13.1.tgz
$ cp docker/docker* /bin
$ cp docker/completion/bash/docker /etc/bash_completion.d/

创建 docker 的 systemd unit 文件

$ cat > docker.service <<EOF
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
Environment="PATH=/bin:/sbin:/usr/bin:/usr/sbin"
EnvironmentFile=-/run/flannel/docker
ExecStart=/bin/dockerd --log-level=error $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

docker 从 1.13 版本开始,可能将 iptables FORWARD chain的默认策略设置为DROP,从而导致 ping 其它 Node 上的 Pod IP 失败,遇到这种情况时,需要手动设置策略为 ACCEPT

$ sudo iptables -P FORWARD ACCEPT

**#并且把以下命令写入/etc/rc.local文件中,防止节点重启iptables FORWARD chain的默认策略又还原为DROP**

sleep 60 && /sbin/iptables -P FORWARD ACCEPT

为了加快 pull image 的速度,可以使用国内的仓库镜像服务器,同时增加下载的并发数。(如果 dockerd 已经运行,则需要重启 dockerd 生效。)

$ mkdir /etc/docker
$ cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn", "hub-mirror.c.163.com"],
  "max-concurrent-downloads": 10
}
EOF

启动docker

$ sudo mv docker.service /etc/systemd/system/docker.service
$ sudo systemctl daemon-reload
$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld
$ sudo iptables -F && sudo iptables -X && sudo iptables -F -t nat && sudo iptables -X -t nat
$ sudo systemctl enable docker
$ sudo systemctl start docker

验证服务

$ docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 08:47:51 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 08:47:51 2017
 OS/Arch:      linux/amd64
 Experimental: false
通过ifconfig查看网卡信息,保证docker0 与 flanneld网卡的在一个网段,如果不在一个网段集群服务是不正常的。导致这块原因有可能是粘贴上述docker.service信息时,ExecStart=/root/local/bin/dockerd --log-level=error $DOCKER_NETWORK_OPTIONS ExecReload=/bin/kill -s HUP $MAINPID被遗漏,所以需要手动加上,并删除docker0网卡重新加载启动。 虚拟网卡docker0其实是一个网桥,是启动Docker Daemon时创建的。传统的删除网卡方式不好用,下次daemon启动(假设没有指定-b参数)时,又会自动创建docker0网桥,可以按照如下步骤进行删除:
$ systemctl stop docker
$ vim /etc/systemd/system/docker.service
$ systemctl daemon-reload
$ yum install -y net-tools bridge-utils
$ ifconfig docker0 down
$ brctl delbr docker0
$ systemctl start docker

安装和配置 kubelet


以下操作只用在master上执行

kubelet 启动时向 kube-apiserver 发送 TLS bootstrapping 请求,需要先将 bootstrap token 文件中的 kubelet-bootstrap 用户赋予 system:node-bootstrapper 角色,然后 kubelet 才有权限创建认证请求(certificatesigningrequests)

$ kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
--user=kubelet-bootstrap 是文件 /etc/kubernetes/token.csv 中指定的用户名,同时也写入了文件 /etc/kubernetes/bootstrap.kubeconfig;

以上操作只用在master上执行

下载最新的 kubelet 和 kube-proxy 二进制文件

$ wget https://dl.k8s.io/v1.6.2/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes
$ tar -xzvf  kubernetes-src.tar.gz
$ sudo cp -r ./server/bin/{kube-proxy,kubelet} /bin/

创建 kubelet bootstrapping kubeconfig 文件

$ **# 设置集群参数**
$ kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
$ **# 设置客户端认证参数**
$ kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
$ **# 设置上下文参数**
$ kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig
$ **# 设置默认上下文**
$ kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
$ cp bootstrap.kubeconfig /etc/kubernetes/

创建 kubelet 的 systemd unit 文件

$ sudo mkdir /var/lib/kubelet **# 必须先创建工作目录**
$ cat > kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/bin/kubelet \\
  --address=${NODE_IP} \\
  --hostname-override=${NODE_IP} \\
  --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \\
  --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \\
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
  --require-kubeconfig \\
  --cert-dir=/etc/kubernetes/ssl \\
  --cluster-dns=${CLUSTER_DNS_SVC_IP} \\
  --cluster-domain=${CLUSTER_DNS_DOMAIN} \\
  --hairpin-mode promiscuous-bridge \\
  --allow-privileged=true \\
  --serialize-image-pulls=false \\
  --logtostderr=true \\
  --v=2
ExecStartPost=/sbin/iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -s 172.16.0.0/12 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -s 192.168.0.0/16 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -p tcp --dport 4194 -j DROP
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

启动kubelet服务

$ sudo mv kubelet.service /etc/systemd/system/kubelet.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable kubelet
$ sudo systemctl start kubelet
$ systemctl status kubelet

通过 kubelet 的 TLS 证书请求

kubelet 首次启动时向 kube-apiserver 发送证书签名请求,必须通过后 kubernetes 系统才会将该 Node 加入到集群。

查看未授权的 CSR 请求

$ kubectl get csr
NAME        AGE       REQUESTOR           CONDITION
csr-5g5t6   48s       kubelet-bootstrap   Pending
$ kubectl get nodes
No resources found.

通过 CSR 请求

$ kubectl certificate approve csr-5g5t6
certificatesigningrequest "csr-5g5t6" approved

$ kubectl get nodes
NAME        STATUS    AGE       VERSION
10.1.0.71   Ready     31s       v1.6.2
#自动生成了 kubelet kubeconfig 文件和公私钥:
$ ls -l /etc/kubernetes/kubelet.kubeconfig
-rw-------. 1 root root 2276 Nov  8 11:49 /etc/kubernetes/kubelet.kubeconfig
$ ls -l /etc/kubernetes/ssl/kubelet*
-rw-r--r--. 1 root root 1042 Nov  8 11:49 /etc/kubernetes/ssl/kubelet-client.crt
-rw-------. 1 root root  227 Nov  8 11:46 /etc/kubernetes/ssl/kubelet-client.key
-rw-r--r--. 1 root root 1103 Nov  8 11:49 /etc/kubernetes/ssl/kubelet.crt
-rw-------. 1 root root 1679 Nov  8 11:49 /etc/kubernetes/ssl/kubelet.key

生成kube-proxy证书

$ cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

$ cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem -config=./ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

将事先创建好的kubernetes-proxy*.pem拷贝到集群/etc/kubernetes/ssl下

$ sudo cp kube-proxy*.pem /etc/kubernetes/ssl/

创建 kube-proxy kubeconfig 文件

$ **# 设置集群参数**
$ kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
$ **# 设置客户端认证参数**
$ kubectl config set-credentials kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
$ **# 设置上下文参数**
$ kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
$ **# 设置默认上下文**
$ kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
$ cp kube-proxy.kubeconfig /etc/kubernetes/

创建 kube-proxy 的 systemd unit 文件

$ sudo mkdir -p /var/lib/kube-proxy # 必须先创建工作目录
$ cat > kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/bin/kube-proxy \\
  --bind-address=${NODE_IP} \\
  --hostname-override=${NODE_IP} \\
  --cluster-cidr=${SERVICE_CIDR} \\
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

启动kube-proxy服务

$ sudo mv kube-proxy.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable kube-proxy
$ sudo systemctl start kube-proxy
$ systemctl status kube-proxy

全新node加入集群主要注意事项

export export NODE_IP=10.1.0.74  #环境变量需要修改当前节点ip

1.安装flanneld注意
确保/etc/flanneld/ssl下的证书与/etc/kubernetes/ssl/ca.pem  
先安装docker并启动的话,可能docker0和flanneld不在一个网段,所以需要先启动flanneld

2.安装docker
如果存在服务起不来的现象,删除/var/lib/docker目录,再启动一次

3.一定确保/etc/flanneld/ssl下有其证书及密钥文件
执行/bin/etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/flanneld/ssl/flanneld.pem \
  --key-file=/etc/flanneld/ssl/flanneld-key.pem \
  set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'

 4.安装kubelet注意
 切忌不能将其他/etc/kubernetes/ssl下的kubelet-client.crt和kubelet-client.key直接拷贝到新的node上(因为这两个及/etc/kubernetes/kubelet.kubeconfig是当master进行csr验证后自动生成的)
 5.安装kube-proxy

dns插件部署

按照文档http://blog.csdn.net/qq_35904833/article/details/78522599

部署私有库

#安装docker(大于1.10.0版本)
yum -y install docker

#安装docker-compose(大于1.6版本)
curl -L https://github.com/docker/compose/releases/download/1.14.0/docker-compose-`uname -s`-`uname -m` > /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose

#下载harbor安装,在有网络的情况下,下载online(小文件即可)
https://github.com/vmware/harbor/releases
#源码 git clone https://github.com/vmware/harbor.git
#解压
tar zxvf harbor-online-installer-v1.1.2.tgz
#根据需求修改harbor.cfg 为方便下序安装,这里定义:
hostname = harbor.test.com
#需要注意的是,同样需要通过cfssl直造TLS证书:
$ cat > harbor-csr.json <<EOF
{
  "CN": "harbor",
  "hosts": [
    "127.0.0.1",
    "$NODE_IP"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

$ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=/etc/kubernetes/ssl/ca-config.json \
  -profile=kubernetes harbor-csr.json | cfssljson -bare harbor
$ ls harbor*
harbor.csr  harbor-csr.json  harbor-key.pem harbor.pem
$ sudo mkdir -p /etc/harbor/ssl
$ sudo mv harbor*.pem /etc/harbor/ssl

#注意将ui_url_protocol = https(修改为https)
#但凡涉及到修改harbor.cfg文件后,都需要在停止服务后执行./prepare使之重新加载配置文件。
#执行安装脚本
./install.sh
#安装完毕后即可通过harbor.test.com登录(端口、挂在目录均可通过docker-compose.yml进行修改)
#由于采用了默认的 http 方式连接,而 Docker 认为这是不安全的,所以在 push 之前需要调整一下 docker 配置
vim /etc/systemd/system/docker.service
#增加DOCKER_OPTS=--insecure-registry harbor.test.com
确保有/etc/docker/certs.d/10.1.0.78/ca.crt文件
#ca.crt为/etc/kubernetes/ssl/ca.pem
即可通过docker login harbor.test.com进行登录,随后push
push的原则是docker push harbor.test.com/项目名称/镜像名称:版本号。
至此,harbor私有库搭建完毕。
参考https://github.com/vmware/harbor/blob/master/docs/installation_guide.md
https://blog.csdn.net/john_f_lau/article/details/78249540

问题&&答案

1.node超过指定时间未上报自己状态时。master会判定其为“失联”,从而标记成notready,并且开始转移负载(既失联node上的实例转移到ready的node上)
2.为什么镜像用docker run启动时没问题,而用k8s的rc\deployment启动时会一直处于死循环。
因为k8s对于长期运行的容器要求是:其程序要一直在前台运行,如果镜像的启动命令为后台执行,那么在Pod建立的第一时间,k8s认为pod执行结束。将立即销毁pod。所以对于无法改造成前台运行的应用。应该借助Supervisor辅助进行前台运行
3.dns服务,进入容器后无法ping通非默认systemspaces的service服务地址。比如在kube-system组里的服务,ping的时候需要是 ping kube-dns**.kube-system.svc.cluster.local** ,若要直接ping kube-dns实现。则在kubernetes-csr.json中需要提前定义。"kubernetes.**kube-system**.svc.cluster.local"

理解

k8s访问集群外独立的服务最好的方式是采用Endpoint方式(可以看作是将k8s集群之外的服务抽象为内部服务),以mysql服务为例:
创建mysql-endpoints.yaml

apiVersion: v1
kind: Endpoints
metadata:
  name: mysql-test
  namespace: default
subsets:
  - addresses:
    - ip: 10.1.0.32
    ports:
      - port: 3306
多个端口的话可以在此处列出

创建mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql-test
spec:
  ports:
    - port: 3306
同样多端口需要列出

测试连接数据库:
[root@LOC-XAN-PRO-032 run]# kubectl exec -it mysql-1225577607-rv8c1 sh
/data # ping mysql-test
PING mysql-test (10.254.121.162): 56 data bytes
^C
--- mysql-test ping statistics ---
11 packets transmitted, 0 packets received, 100% packet loss
/data # mysql -hmysql-test -ubill -ppass
ERROR 1045 (28000): Access denied for user 'bill'@'172.30.95.10' (using password: YES)
/data # mysql -hmysql-test -ulienhua -p123
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 5.5.52-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| lienhua            |
| test               |
+--------------------+
3 rows in set (0.00 sec)

MariaDB [(none)]>


查看这个service
[root@LOC-XAN-PRO-032 mysql]# kubectl describe svc mysql-test
Name:           mysql-test
Namespace:      default
Labels:         <none>
Annotations:        <none>
Selector:       <none>
Type:           ClusterIP
IP:         10.254.121.162
Port:           <unset> 3306/TCP
Endpoints:      10.1.0.32:3306
Session Affinity:   None
Events:         <none>


参考连接:https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/03-%E9%83%A8%E7%BD%B2%E9%AB%98%E5%8F%AF%E7%94%A8Etcd%E9%9B%86%E7%BE%A4.md

  • 1
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值