kubernetes-----部署单master的二进制群集

EoinXu

已于 2022-07-08 16:48:04 修改

阅读量492

点赞数 1

分类专栏： kubernetes 文章标签： kubernetes java 容器

于 2020-04-28 16:33:46 首次发布

本文链接：https://blog.csdn.net/qq_42761527/article/details/105799430

版权

kubernetes 专栏收录该内容

12 篇文章 1 订阅

订阅专栏

一.单master的二进制集群分析

单节点的架构图

master组件

我们需要再master部署三大核心组件
kube-apiserver:是集群的统一入口，各个组件的协调者，所有对象资源的增删改查和监听操作都交给APIserver处理，再提交给etcd存储。
kue-controller-manager：处理群集中常规的后台任务，一个资源对应一个控制器，而controller-manager就是负责管理这些控制器。
kube-scheduler：根据调度算法为新创建的pod选择一个node节点，可以任意部署，可以部署同一个节点上，也可以部署在不同节点上。

node组件

kublet：kube是master在node节点上的Agent，管理本机运行容器的生命周期，比如创建容器、Pod挂载数据卷、下载secret、获取容器和节点状态等工作。kubelet将每个pod转换成一组容器
kebe-proxy：在node节点上实现pod网络代理，维护网络规划和四层负载均衡的工作
docker engine：docker引擎

etcd群集

etcd是CoreOS团队于为2013年6月发起的开源项目，它的目标是构建一个高可用的分布式键值（key-value）数据库。etcd内部采用raft协议作为一致性算法，etcd基于go语言开发。
etcd群集属于无中心化集群，应用于区块链技术
etcd作为服务发现系统有如下特点：

简单：安装配置简单，而且提供了HTTP进行交互，使用也很简单

安全：支持SSL证书验证

快速：根据官方提供的benchmark数据，单实例支持每秒2k+读操作

可靠：采用raft算法，实现分布式数据的可用性和一致性

etcd的三大特性

1.一个强一致性、高可用的服务存储目录

2.具有注册服务和健康服务健康状况的机制：用户可以在etcd中注册服务，并且对注册的服务配置key TTL，定时保持服务的心跳以达到监控健康状态的效果

3.一种查找和连接服务的机制：通过在etcd指定的主题下注册的服务业能在对应的主题下查找到。为了确保连接，我们可以在每个服务器器上部署一个proxy模式的etcd，这样就可以确保访问etcd集群的服务都能够互相连接。

证书认证

由于各个组件会在集群内部向apiserver通信，所以需要通信需要做CA认证

组件	使用的证书
etcd	ca.pem，server.pem，server-key.pem
flannel	ca.pem，server.pem，server-key.pem
kube-apiserver	ca.pem，server.pem，server-key.pem
kubelet	ca.pem，ca-key.pem
kube-proxy	ca.pem，kube-proxy.pem，kube-proxy-key.pem
kubectl	ca.pem，admin-pem，admin-key.pem

二.实验环境分析

角色	IP地址	系统与资源	需要安装的组件
master	192.168.43.101/24	centos7.4（单核cpu，2G内存）	kube-apiserver、kube-controller-manager、kube-scheduler、etcd
node1	192.168.43.102/24	centos7.4（单核cpu，2G内存）	kubelet、kube-proxy、docker、flnanel、etcd
node2	192.168.43.103/24	centos7.4（单核cpu，2G内存）	kubelet、kube-proxy、docker、flannel、etcd

此处将etcd集群分别安装在master和node节点上
在官网获取相关软件包
etcd下载网址：Releases · etcd-io/etcd · GitHub

kubernetes软件包下载地址：https://github.com/kubernetes/kubernetes/releases?after=v1.13.1

百度网盘分享：

链接：https://pan.baidu.com/s/1aIB_TP_n89gMZaM82p-rCg
提取码：pxf3

三.具体部署

基础环境配置

设置主机名，永久关闭安全性功能

hostnamectl set-hostname master/node1/node2


systemctl stop firewalld
systemctl disable firewalld

setenforce 0
sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config

部署etcd集群

在master上部署etcd

编辑证书、签名、密钥的脚本

[root@master ]# vi etcd-cert.sh 
##定义ca证书的配置文件
##ca-config.json,可以定义多个文件，分别制定不同的过期时间、使用场景等参数；后续在签名证书时使用某个文件
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
##定义使用时长
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
##表示server、client可以用该CA进行验证
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
##实现证书签名
cat > ca-csr.json <<EOF
{
#CN”：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；
    "CN": "etcd CA",
#定义算法和长度
    "key": {
        "algo": "rsa",
        "size": 2048
    },
#定义名称
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
  }
}
EOF
##cfssl是生成证书的工具
#cfssljson,通过json文件生成证书
#下述命令是生成ca证书和私钥
##initca表示初始化
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------
#指定etcd三个节点之间的通信验证
cat > server-csr.json <<EOF
{
    "CN": "etcd",
##定义三个节点的地址
    "hosts": [
    "192.168.43.101",
    "192.168.43.102",
    "192.168.43.103"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF
##使用cfssl工具生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

编辑etcd服务集群开启的脚本

[root@master ]# vi etcd.sh 

#!/bin/bash
# example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380

##位置变量指定节点名字，IP地址，和集群
ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd

##创建etcd的配置文件
cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"        //etcd内部端口
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"        //etcd对外提供的端口

#[Clustering]，定义集群信息
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
##生成服务执行脚本
cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
##定义服务之间的证书认证
[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
##开启服务
systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

下载cfssl工具

##创建一个k8s的目录，存储相关的文件
[root@master ~]# mkdir k8s
[root@master ~]# cd k8s/
##编辑的证书脚本和etcd服务开启脚本
[root@master k8s]# ls
etcd-cert.sh  etcd.sh
##创建此目录存储证书，以及相关材料
[root@master k8s]# mkdir ectd-cert
##将etcd服务开启脚本移到新创建的目录下
[root@master k8s]# mv etcd-cert.sh etcd-cert

##创建cfssl工具下载的脚本
##curl -L表示下载，-o表示到处
[root@master k8s]# vim cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo


#执行脚本，生成相关工具
[root@master k8s]# bash cfssl.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9.8M  100  9.8M    0     0  1371k      0  0:00:07  0:00:07 --:--:-- 2042k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2224k  100 2224k    0     0   576k      0  0:00:03  0:00:03 --:--:--  576k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6440k  100 6440k    0     0  1721k      0  0:00:03  0:00:03 --:--:-- 1721k


[root@master k8s]# ls /usr/local/bin/
cfssl  cfssl-certinfo  cfssljson
//cfssl：生成证书的工具
//cfssl-certinfo:查看证书信息
//cfssljson：通过传入json文件生成证书

[root@master k8s]# cd /usr/local/bin/
##添加权限
[root@master bin]# chmod 777 cfssl cfssl-certinfo cfssljson 
[root@master bin]# ll
总用量 18808
-rwxrwxrwx. 1 root root 10376657 4月  27 16:38 cfssl
-rwxrwxrwx. 1 root root  6595195 4月  27 16:38 cfssl-certinfo
-rwxrwxrwx. 1 root root  2277873 4月  27 16:38 cfssljson

生成ca证书、证书签名、密钥、节点通信验证等

#进入etcd-cert目录
[root@master] cd /root/k8s/etcd-cert
[root@master etcd-cert]# ls
etcd-cert.sh
#创建CA配置文件
[root@master etcd-cert]# cat > ca-config.json <<EOF
> {
>   "signing": {
>     "default": {
>       "expiry": "87600h"        //证书的有效期
>     },
>     "profiles": {
>       "www": {
>          "expiry": "87600h",
>          "usages": [
>             "signing",
>             "key encipherment",
>             "server auth",
>             "client auth"
>         ]  
>       } 
>     }
>   }
> }
> EOF

#生成CA证书签名，.csr为后缀的文件为签名文件
[root@master etcd-cert]# cat > ca-csr.json <<EOF 
> {   
>     "CN": "etcd CA",
>     "key": {
>         "algo": "rsa",
>         "size": 2048
>     },
>     "names": [
>         {
>             "C": "CN",
>             "L": "Beijing",
>             "ST": "Beijing"
>         }
>     ]
> }
> EOF
[root@master etcd-cert]# ls
ca-config.json  ca-csr.json  etcd-cert.sh

##利用cfssl工具，生成CA证书和私钥
[root@master etcd-cert]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2020/04/27 16:42:51 [INFO] generating a new CA key and certificate from CSR
2020/04/27 16:42:51 [INFO] generate received request
2020/04/27 16:42:51 [INFO] received CSR
2020/04/27 16:42:51 [INFO] generating key: rsa-2048
2020/04/27 16:42:52 [INFO] encoded CSR
2020/04/27 16:42:52 [INFO] signed certificate with serial number 303976521847226843130741301784704905842128877733
[root@master etcd-cert]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd-cert.sh

##生成服务端的证书
[root@master etcd-cert]# cat > server-csr.json <<EOF
> {
>     "CN": "etcd",        //定义项目名称
>     "hosts": [            //定义节点
>     "192.168.43.101",
>     "192.168.43.102",
>     "192.168.43.103"
>     ],
>     "key": {
>         "algo": "rsa",        //密钥算法
>         "size": 2048
>     },
>     "names": [
>         {
>             "C": "CN",        //名称需要和ca处一致
>             "L": "BeiJing",
>             "ST": "BeiJing"
>         }
>     ]
> }
> EOF
[root@master etcd-cert]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd-cert.sh  server-csr.json

##生成etcd集群的证书与私钥
##.pem后缀为证书标识
[root@master etcd-cert]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2020/04/27 16:44:01 [INFO] generate received request
2020/04/27 16:44:01 [INFO] received CSR
2020/04/27 16:44:01 [INFO] generating key: rsa-2048
2020/04/27 16:44:02 [INFO] encoded CSR
2020/04/27 16:44:02 [INFO] signed certificate with serial number 141201637904782202236349266630353395425516635789
2020/04/27 16:44:02 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

##最后CA认证文件如下
[root@master etcd-cert]# ls
ca-config.json  ca-csr.json  ca.pem        server.csr       server-key.pem
ca.csr          ca-key.pem   etcd-cert.sh  server-csr.json  server.pem

解压etcd压缩包

##上传etcd、kubernetes、flannel软件包
[root@master k8s]# ls
cfssl.sh   etcd.sh                          flannel-v0.10.0-linux-amd64.tar.gz
etcd-cert  etcd-v3.3.10-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz

##在当前目录下解压etcd软件包
[root@master k8s]# tar xzf etcd-v3.3.10-linux-amd64.tar.gz 

##解压后具体文件如下
[root@master k8s]# ls etcd-v3.3.10-linux-amd64
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md

配置文件、执行文件、证书

##创建etcd工作目录，底下分别有配置目录、执行目录、证书目录
[root@master k8s]# mkdir /opt/etcd/{cfg,bin,ssl} -p

[root@master k8s]# ls
cfssl.sh   etcd-v3.3.10-linux-amd64            kubernetes-server-linux-amd64.tar.gz
etcd-cert  etcd-v3.3.10-linux-amd64.tar.gz
etcd.sh    flannel-v0.10.0-linux-amd64.tar.gz

##将etcd的执行脚本移动到新创建的/opt/etcd/bin目录下
[root@master k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/

##复制所有证书到新创建的/opt/etcd/ssl目录下
##此处有ca的证书和ca-key的证书，一共四个证书
[root@master k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/

卡住etcd状态等待其他节点加入，等待时间一般为5分钟，这个操作的目的时生成etcd的配置文件/opt/etcd/cfg/etcd

##在/root/k8s目录下，执行etcd.sh脚本
##192.168.43.101为master
##192.168.43.102和192.168.43.103为node
##2380端口时etcd对内的通信端口

[root@master k8s]# bash etcd.sh etcd01 192.168.43.101 etcd02=https://192.168.43.102:2380,etcd03=https://192.168.43.103:2380

在另外一个远程终端上，查看服务是否开启

##查看etcd的进程
[root@master ~]# ps -ef | grep etcd
root      36534   1605  0 16:52 pts/0    00:00:00 bash etcd.sh etcd01 192.168.43.101 etcd02=https://192.168.43.102:2380,etcd03=https://192.168.43.103:2380
root      36585  36534  0 16:52 pts/0    00:00:00 systemctl restart etcd
root      36594      1  3 16:52 ?        00:00:00 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://192.168.43.101:2380 --listen-client-urls=https://192.168.43.101:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.43.101:2379 --initial-advertise-peer-urls=https://192.168.43.101:2380 --initial-cluster=etcd01=https://192.168.43.101:2380,etcd02=https://192.168.43.102:2380,etcd03=https://192.168.43.103:2380 --initial-cluster-token=etc-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
root      36658  36615  0 16:52 pts/1    00:00:00 grep --color=auto etcd

将master节点上的etcd的相关证书、脚本、配置文件，复制到node节点上，提高效率，但是复制过去的配置文件需要修改IP地址和名称

[root@master ~]# ls /opt/etcd/
bin  cfg  ssl

##复制/opt/etcd目录下的文件到node节点上
##-r为递归复制
[root@master ~]# scp -r /opt/etcd/ root@192.168.43.102:/opt/
The authenticity of host '192.168.43.102 (192.168.43.102)' can't be established.
ECDSA key fingerprint is SHA256:utNBPHSX6ZDikmo/1d6F8cyC1KPYCxhLxdlMF+WC/FM.
ECDSA key fingerprint is MD5:fe:4f:57:8a:80:d6:81:4e:e3:c7:ba:f4:85:69:a7:dd.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.43.102' (ECDSA) to the list of known hosts.
root@192.168.43.102's password: 
etcd                                                                                       100%  516   650.5KB/s   00:00    
etcd                                                                                       100%   18MB  57.9MB/s   00:00    
etcdctl                                                                                    100%   15MB  80.6MB/s   00:00    
ca-key.pem                                                                                 100% 1675   361.8KB/s   00:00    
ca.pem                                                                                     100% 1265   366.3KB/s   00:00    
server-key.pem                                                                             100% 1675   683.9KB/s   00:00    
server.pem                                                                                 100% 1338     1.0MB/s   00:00    
[root@master ~]# scp -r /opt/etcd/ root@192.168.43.103:/opt/
The authenticity of host '192.168.43.103 (192.168.43.103)' can't be established.
ECDSA key fingerprint is SHA256:Zwn4nEP/mb/GP2Ac9x3UNiISW/qpSIRgW4FcpD5qrA8.
ECDSA key fingerprint is MD5:98:d0:52:af:68:79:8d:c3:11:89:cf:aa:dc:2e:ae:49.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.43.103' (ECDSA) to the list of known hosts.
root@192.168.43.103's password: 
etcd                                                                                       100%  516   328.2KB/s   00:00    
etcd                                                                                       100%   18MB  65.9MB/s   00:00    
etcdctl                                                                                    100%   15MB  75.3MB/s   00:00    
ca-key.pem                                                                                 100% 1675     1.8MB/s   00:00    
ca.pem                                                                                     100% 1265   434.2KB/s   00:00    
server-key.pem                                                                             100% 1675   792.1KB/s   00:00    
server.pem                                                                                 100% 1338     1.2MB/s   00:00   

##复制etcd执行文件到node节点上
[root@master ~]# scp /usr/lib/systemd/system/etcd.service root@192.168.43.102:/usr/lib/systemd/system/
root@192.168.43.102's password: 
etcd.service                                                                               100%  923     1.1MB/s   00:00    
[root@master ~]# scp /usr/lib/systemd/system/etcd.service root@192.168.43.103:/usr/lib/systemd/system/
root@192.168.43.103's password: 
etcd.service                                                                               100%  923     1.0MB/s   00:00    
[root@master ~]#

在node上部署etcd

在node1上修改etcd文件，并且启动服务

##因为是从master中复制过去的，所以地址和名字需要修改一下
[root@node1 ~]# vi /opt/etcd/cfg/etcd 

#[Member]
ETCD_NAME="etcd02"        ##修改为etcd02
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.43.102:2380"        ##修改地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.43.102:2379"         ##修改地址

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.43.102:2380"  ##修改地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.43.102:2379"  ##修改地址，2379端口对外提供端口
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.43.101:2380,etcd02=https://192.168.43.102:2380,etcd03=https://192.168.43.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

##启动服务
[root@node1 ~]# systemctl start etcd.service

在node2上修改etcd文件，并且启动etcd服务

[root@node2 ~]# vi /opt/etcd/cfg/etcd 

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.43.103:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.43.103:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.43.103:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.43.103:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.43.101:2380,etcd02=https://192.168.43.102:2380,etcd03=https://192.168.43.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"


[root@node2 ~]# systemctl start etcd.service

在master上检测etcd的健康状况

查看etcd集群的健康状态

##进入/root/k8s/etcd-cert/目录
[root@master ~]# cd k8s/etcd-cert/

[root@master etcd-cert]# ls
ca-config.json  ca-csr.json  ca.pem        server.csr       server-key.pem
ca.csr          ca-key.pem   etcd-cert.sh  server-csr.json  server.pem

##使用该目录下的证书文件
##endpoints指向端点
[root@master etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.43.101:2379,https://192.168.43.102:2379,https://192.168.43.103:2379" cluster-health

member 8e0db64fe8ec891a is healthy: got healthy result from https://192.168.43.101:2379
member 935e3fc4cd001836 is healthy: got healthy result from https://192.168.43.102:2379
member a014504f8a161729 is healthy: got healthy result from https://192.168.43.103:2379
cluster is healthy

##到目前为止etcd集群安装成功！！
##k8s集群的所有信息都存储在etcd，所以必须先安装etcd
##etcd为中心化集群，应用于区块链

部署flannel网络

flannel简述

部署flannel网络之前需要安装docker引擎
flannel网络是overlay网络的一种，也是将源数据包封装在另外一种网络包里面进行路由转发和通信，目前支持UDP、VXALN、AWX vpc和GCE路由等数据转发方式

Overlay Network：覆盖网络，在基础网络上叠加的一种网络技术模式，该网络中的主机通过虚拟链路连接起来

VXLAN：将源数据包封装到UDP中，并使用基础网络的IP/MAC作为外层报文进行封装，然后在以太网上传输，达到目的地后有隧道端点解封并且将数据发给目标地址

VTEP相当于docker0，而VTEP和eth0之间时NAT地址转换

在node1和node2中安装docker

以下步骤在node1和node2中都操作

##安装docker的依赖包
yum install -y yum-utils device-mapper-persistent-data lvm2
##设置docker源（docker-ce为docker的社区版本）
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

##安装docker
yum install -y docker-ce

##开启docker服务
systemctl start docker.service
systemctl enable docker.service

##加速镜像

tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://dnntzrw4.mirror.aliyuncs.com"]
}
EOF

##网络优化###
vi /etc/sysctl.conf
net.ipv4.ip_forward=1

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p

##重启加载docker
systemctl daemon-reload
systemctl restart docker

部署flannel网络

在master端分配子网段到etcd中，供给flannel使用

##进入有证书的目录
[root@master k8s]# cd etcd-cert/
[root@master etcd-cert]# ls
ca-config.json  ca-csr.json  ca.pem        server.csr       server-key.pem
ca.csr          ca-key.pem   etcd-cert.sh  server-csr.json  server.pem

##写入分配的子网段到etcd，以供flannel使用
##set写入
##指定网络类型为vxaln
[root@master etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.43.101:2379,https://192.168.43.102:2379,https://192.168.43.103:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
##执行命令后，出现下列信息
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

##查看写入的信息是否正确
##get查看
[root@master etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.43.101:2379,https://192.168.43.102:2379,https://192.168.43.103:2379" get /coreos.com/network/config

{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}  ##出现此条信息说明信息无误


[root@master etcd-cert]# cd ..
[root@master k8s]# ls
cfssl.sh   etcd.sh                   etcd-v3.3.10-linux-amd64.tar.gz     kubernetes-server-linux-amd64.tar.gz
etcd-cert  etcd-v3.3.10-linux-amd64  flannel-v0.10.0-linux-amd64.tar.gz

##将软件包复制到node1和node2节点
[root@master k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz root@192.168.43.102:/root
root@192.168.43.102's password: 
flannel-v0.10.0-linux-amd64.tar.gz                                                         100% 9479KB  51.2MB/s   00:00    
[root@master k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz root@192.168.43.103:/root
root@192.168.43.103's password: 
flannel-v0.10.0-linux-amd64.tar.gz                                                         100% 9479KB  48.1MB/s   00:00

在node1和node2节点安装flannel

##########################以下以node2为例，node1做相同操作########################
[root@node2 ~]# ls
anaconda-ks.cfg  flannel-v0.10.0-linux-amd64.tar.gz  initial-setup-ks.cfg  公共  模板  视频  图片  文档  下载  音乐  桌面

##解压flannel软件包到当前目录下
[root@node2 ~]# tar xzf flannel-v0.10.0-linux-amd64.tar.gz 
[root@node2 ~]# ls
anaconda-ks.cfg  flannel-v0.10.0-linux-amd64.tar.gz  mk-docker-opts.sh  公共  视频  文档  音乐
flanneld         initial-setup-ks.cfg                README.md          模板  图片  下载  桌面

##创建kubernetes的目录，方便执行服务
##-p创建多个子目录
[root@node2 ~]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p

##将flannel解压后的执行文件，移动到/opt/kubernetes/bin目录下
[root@node2 ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/
[root@node2 ~]# ls
anaconda-ks.cfg                     initial-setup-ks.cfg  公共  视频  文档  音乐
flannel-v0.10.0-linux-amd64.tar.gz  README.md             模板  图片  下载  桌面

##编辑flannel脚本，生成flannel相关的配置文件和执行文件
[root@node2 ~]# vi flannel.sh 

#!/bin/bash

##指定etcd的地址和对外提供端口
ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}

##参照证书，制定flannel相关配置文件
cat <<EOF >/opt/kubernetes/cfg/flanneld

FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"

EOF

##生成flannel的执行文件
cat <<EOF >/usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

#定义服务
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS
##DOCKER_NETWORK_OPTIONS指定子网段
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

EOF
##开启flannel服务
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld


###执行flannel.sh脚本，指定各个节点的地址端口，开启flannel网络功能
[root@node2 ~]# bash flannel.sh https://192.168.43.101:2379,https://192.168.43.102:2379,https://192.168.43.103:2379

Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.

##配置docker来连接flannel网络
[root@node2 ~]# vi /usr/lib/systemd/system/docker.service 
[Servi
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/subnet.env        ##指定环境文件，包含子网段信息
##添加变量
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always


##查看flannel子网文件信息
[root@node2 ~]# cat /run/flannel/subnet.env 
DOCKER_OPT_BIP="--bip=172.17.12.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.12.1/24 --ip-masq=false --mtu=1450"

##重启docker，加载新的文件，使得环境文件生效
[root@node2 ~]# systemctl daemon-reload 
[root@node2 ~]# systemctl restart docker

验证flannel网络

node1和node2中都应该存在flannel网络

分别在node1和node2节点创建容器

##在node1和node2分别创建一个容器，且进入容器
docker run -it centos:7 /bin/bash

##在容器中安装工具，使之能够使用ifconfig，查看地址信息
yum install -y net-tools

##node1处的容器地址信息
[root@d180afa14db1 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.36.2  netmask 255.255.255.0  broadcast 172.17.36.255
        ether 02:42:ac:11:24:02  txqueuelen 0  (Ethernet)
        RX packets 20202  bytes 15821012 (15.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9328  bytes 508450 (496.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@d180afa14db1 /]# exit
exit
[root@node1 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
d180afa14db1        centos:7            "/bin/bash"         25 minutes ago      Exited (0) 4 seconds ago                       practical_spence


##node2处的容器地址信息
[root@0dd995a58a76 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.12.2  netmask 255.255.255.0  broadcast 172.17.12.255
        ether 02:42:ac:11:0c:02  txqueuelen 0  (Ethernet)
        RX packets 20312  bytes 15831233 (15.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9432  bytes 514259 (502.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@0dd995a58a76 /]# exit
exit
[root@node2 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
0dd995a58a76        centos:7            "/bin/bash"         25 minutes ago      Exited (0) 4 seconds ago                       confident_knuth

在node1处的容器ping通node2处的节点

##node1容器地址为172.17.36.2
##node2容器地址为172.17.12.2

[root@d180afa14db1 /]# ping 172.17.12.2 -c3
PING 172.17.12.2 (172.17.12.2) 56(84) bytes of data.
64 bytes from 172.17.12.2: icmp_seq=1 ttl=62 time=0.525 ms
64 bytes from 172.17.12.2: icmp_seq=2 ttl=62 time=0.501 ms
64 bytes from 172.17.12.2: icmp_seq=3 ttl=62 time=0.568 ms

--- 172.17.12.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.501/0.531/0.568/0.033 ms
[root@d180afa14db1 /]# exit
exit
[root@node1 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
d180afa14db1        centos:7            "/bin/bash"         25 minutes ago      Exited (0) 4 seconds ago                       practical_spence
[root@node1 ~]# docker rm d180afa14db1 
d180afa14db1

flannel网络的作用就是保证不同的节点上的容器相互通信，以上完成了flannel组件的部署

部署master组件

master组件有

kube-apiserver：服务的唯一接口（证书、权限、角色捆绑）

schedule（调度器）服务

contrller-manager：控制管理器

在master上生成apai-server的证书，以便提供服务

#上传master.zip,并且解压
[root@master k8s]# rz

[root@master k8s]# ls
cfssl.sh   etcd.sh                   etcd-v3.3.10-linux-amd64.tar.gz     kubernetes-server-linux-amd64.tar.gz
etcd-cert  etcd-v3.3.10-linux-amd64  flannel-v0.10.0-linux-amd64.tar.gz  master.zip

##master.zip中包含三个重要组件的脚本文件
[root@master k8s]# unzip master.zip 
Archive:  master.zip
  inflating: apiserver.sh            
  inflating: controller-manager.sh   
  inflating: scheduler.sh  


##创建存储kuernetes二进制集群的工作目录
##cfg配置文件、bin命令文件、ssl证书文件
[root@master k8s]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p

##创建kubernetes集群证书的目录
[root@master k8s]# mkdir k8s-cert
[root@master k8s]# cd k8s-cert/
[root@master k8s-cert]# ls

##创建kubernetes集群证书的脚本
[root@master k8s-cert]# vi k8s-cert.sh 

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
#O代表对象
            "O": "k8s",
#OU代表单元
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------

cat > server-csr.json <<EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
##此处不声明node节点的地址，k8s自动发现node
#指定第一台master的地址
      "192.168.43.101",
#指定第二台master的地址，以便后面做多master集群
      "192.168.43.104",
#指定vip地址
      "192.168.43.100",
#指定第一台调度服务器地址（master）
      "192.168.43.105",
#指定第二台调度服务器地址（backup）
      "192.168.43.106",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF


#-----------------------

cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF
#admin-csr.json生成管理员的证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

在master上生成k8s的证书

##执行k8s-cert.sh脚本，生成证书
[root@master k8s-cert]# bash k8s-cert.sh 
2020/04/27 18:29:59 [INFO] generating a new CA key and certificate from CSR
2020/04/27 18:29:59 [INFO] generate received request
2020/04/27 18:29:59 [INFO] received CSR
2020/04/27 18:29:59 [INFO] generating key: rsa-2048
2020/04/27 18:29:59 [INFO] encoded CSR
2020/04/27 18:29:59 [INFO] signed certificate with serial number 491488893423855096948315247324707987947191926707
2020/04/27 18:29:59 [INFO] generate received request
2020/04/27 18:29:59 [INFO] received CSR
2020/04/27 18:29:59 [INFO] generating key: rsa-2048
2020/04/27 18:30:00 [INFO] encoded CSR
2020/04/27 18:30:00 [INFO] signed certificate with serial number 263234971884691589051198252567446440632621797072
2020/04/27 18:30:00 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
2020/04/27 18:30:00 [INFO] generate received request
2020/04/27 18:30:00 [INFO] received CSR
2020/04/27 18:30:00 [INFO] generating key: rsa-2048
2020/04/27 18:30:00 [INFO] encoded CSR
2020/04/27 18:30:00 [INFO] signed certificate with serial number 127328426445398142630529412013673856084328373788
2020/04/27 18:30:00 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
2020/04/27 18:30:00 [INFO] generate received request
2020/04/27 18:30:00 [INFO] received CSR
2020/04/27 18:30:00 [INFO] generating key: rsa-2048
2020/04/27 18:30:00 [INFO] encoded CSR
2020/04/27 18:30:00 [INFO] signed certificate with serial number 640453806279864621306204301145468690527984065435
2020/04/27 18:30:00 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

##生成的具体证书如下
##此处应该生成8张证书（.pem）
[root@master k8s-cert]# ls
admin.csr       admin.pem       ca-csr.json  k8s-cert.sh          kube-proxy-key.pem  server-csr.json
admin-csr.json  ca-config.json  ca-key.pem   kube-proxy.csr       kube-proxy.pem      server-key.pem
admin-key.pem   ca.csr          ca.pem       kube-proxy-csr.json  server.csr          server.pem


##复制证书到/opt/kubernetes/ssl目录下
[root@master k8s-cert]# cp ca*pem server*pem /opt/kubernetes/ssl/

解压kubernetes-server-linux-amd64.tar.gz生成各个节点所需要的组件，创建token

#解压kubernetes-server-linux-amd64.tar.gz到当前目录
[root@master k8s]# ls
apiserver.sh           etcd-cert                 etcd-v3.3.10-linux-amd64.tar.gz     kubernetes-server-linux-amd64.tar.gz
cfssl.sh               etcd.sh                   flannel-v0.10.0-linux-amd64.tar.gz  master.zip
controller-manager.sh  etcd-v3.3.10-linux-amd64  k8s-cert                            scheduler.sh
[root@master k8s]# tar xzf kubernetes-server-linux-amd64.tar.gz 


##将解压后的kubernetes的执行文件，复制到/opt/kubernetes/bin目录中
[root@master k8s]# cd kubernetes/server/bin
[root@master bin]# ls
apiextensions-apiserver              kube-apiserver                      kubectl                kube-scheduler.docker_tag
cloud-controller-manager             kube-apiserver.docker_tag           kubelet                kube-scheduler.tar
cloud-controller-manager.docker_tag  kube-apiserver.tar                  kube-proxy             mounter
cloud-controller-manager.tar         kube-controller-manager             kube-proxy.docker_tag
hyperkube                            kube-controller-manager.docker_tag  kube-proxy.tar
kubeadm                              kube-controller-manager.tar         kube-scheduler
[root@master bin]# cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/


##创建token文件
[root@master k8s]# cd /opt/kubernetes/cfg/
[root@master cfg]# ls
#head -c 16 /dev/urandom | od -An -t x | tr -d ' '，随机生成序列号
[root@master cfg]# export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
[root@master cfg]# cat > token.csv << EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

##在master上创建角色，管理node中kubelet
##kubelet-bootstrap管理、授权system:kubelet-bootstrap
##system:kubelet-bootstrap管理node中的kubelet
##token就是授权给角色，以管理kubelet，没有token就不能管理下面的kubelet
[root@master cfg]# ls
token.csv
[root@master cfg]# cat token.csv 
978741334e39eb21d1c13050d6e8bb79,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
#序列号，用户名，id，角色身份

利用证书、二进制文件、token，开启apaiserver

##进入存在apiserver的目录中
[root@master cfg]# cd /root/k8s/
[root@master k8s]# ls
apiserver.sh           etcd.sh                             k8s-cert                              scheduler.sh
cfssl.sh               etcd-v3.3.10-linux-amd64            kubernetes
controller-manager.sh  etcd-v3.3.10-linux-amd64.tar.gz     kubernetes-server-linux-amd64.tar.gz
etcd-cert              flannel-v0.10.0-linux-amd64.tar.gz  master.zip

##开启apiserver
[root@master k8s]# bash apiserver.sh 192.168.43.101 https://192.168.43.101:2379,https://192.168.43.102:2379,https://192.168.43.103:2379

##查看apiserver的配置文件是否正常
[root@master k8s-cert]# cat /opt/kubernetes/cfg/kube-apiserver

KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.43.101:2379,https://192.168.43.102:2379,https://192.168.43.103:2379 \
--bind-address=192.168.43.101 \
--secure-port=6443 \
--advertise-address=192.168.43.101 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
##验证模式
--authorization-mode=RBAC,Node \
--kubelet-https=true \
##开启bootstrap角色授权
--enable-bootstrap-token-auth \
##指定令牌路径
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"


##apiserver配置文件完成

##查看进程端口是否开启
[root@master k8s]# netstat -natp | grep 6443
tcp        0      0 192.168.43.101:6443     0.0.0.0:*               LISTEN      37947/kube-apiserve 
tcp        0      0 192.168.43.101:6443     192.168.43.101:58510    ESTABLISHED 37947/kube-apiserve 
tcp        0      0 192.168.43.101:58510    192.168.43.101:6443     ESTABLISHED 37947/kube-apiserve 
[root@master k8s]# netstat -natp | grep 8080
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      37947/kube-apiserve

开启schedule（调度器）服务、contrller-manager（控制管理器）服务

##执行schedule脚本，以开启schedule服务
[root@master k8s]# ./scheduler.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.

##查看进程，是否存在schedule进程
[root@master k8s]# ps aux | grep ku
postfix   37069  0.0  0.1  91732  1232 ?        S    17:26   0:00 pickup -l -t unix -u
root      37947 13.3 33.4 430920 333984 ?       Ssl  18:37   0:11 /opt/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://192.168.43.101:2379,https://192.168.43.102:2379,https://192.168.43.103:2379 --bind-address=192.168.43.101 --secure-port=6443 --advertise-address=192.168.43.101 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem
root      38033  1.6  2.0  44300 20168 ?        Ssl  18:39   0:00 /opt/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect
root      38044  0.0  0.0 112676   980 pts/0    S+   18:39   0:00 grep --color=auto ku


[root@master k8s]# ls
apiserver.sh           etcd.sh                             k8s-cert                              scheduler.sh
cfssl.sh               etcd-v3.3.10-linux-amd64            kubernetes
controller-manager.sh  etcd-v3.3.10-linux-amd64.tar.gz     kubernetes-server-linux-amd64.tar.gz
etcd-cert              flannel-v0.10.0-linux-amd64.tar.gz  master.zip
##添加控制管理器脚本的执行权限
[root@master k8s]# chmod +x controller-manager.sh 

##执行控制管理器脚本，开启控制管理器服务
[root@master k8s]# ./controller-manager.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

查看master节点的状态

[root@master k8s]# /opt/kubernetes/bin/kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
etcd-2               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
controller-manager   Healthy   ok

部署node组件

node节点上的三大核心组件

kubelet：是master在node节点上的agent，可以管理本机运行容器的生命周期。例如创建容器、Pod挂载数据卷、下载secret、获取容器和节点状态等工作，kubelet 将每个 Pod转换成一组容器。
kube-proxy：在node节点上实现pod网络代理，维护网络规划和四层负载均衡工作
docker engine:已安装

具体部署情况

在master复制kubelet、kube-proxy的执行文件到node上

##进入k8s集群的执行文件目录
[root@master k8s]# cd kubernetes/server/bin/
[root@master bin]# ls
apiextensions-apiserver              kube-apiserver                      kubectl                kube-scheduler.docker_tag
cloud-controller-manager             kube-apiserver.docker_tag           kubelet                kube-scheduler.tar
cloud-controller-manager.docker_tag  kube-apiserver.tar                  kube-proxy             mounter
cloud-controller-manager.tar         kube-controller-manager             kube-proxy.docker_tag
hyperkube                            kube-controller-manager.docker_tag  kube-proxy.tar
kubeadm                              kube-controller-manager.tar         kube-scheduler

##将kubelet、kube-proxy这两个执行文件推送到node1和node2上
[root@master bin]# scp kubelet kube-proxy root@192.168.43.102:/opt/kubernetes/bin/
root@192.168.43.102's password: 
kubelet                                                                                    100%  168MB  66.4MB/s   00:02    
kube-proxy                                                                                 100%   48MB  70.8MB/s   00:00    
[root@master bin]# scp kubelet kube-proxy root@192.168.43.103:/opt/kubernetes/bin/
root@192.168.43.103's password: 
kubelet                                                                                    100%  168MB  64.6MB/s   00:02    
kube-proxy

在node1节点上，上传解压node.zip，对于node2的部署我们可以复制node1节点的文件，增加效率

[root@node1 ~]# rz

[root@node1 ~]# ls
anaconda-ks.cfg  flannel-v0.10.0-linux-amd64.tar.gz  node.zip   公共  视频  文档  音乐
flannel.sh       initial-setup-ks.cfg                README.md  模板  图片  下载  桌面
##这个压缩文件，kubelet和proxy脚本，用于连接master
[root@node1 ~]# unzip node.zip 
Archive:  node.zip
  inflating: proxy.sh                
  inflating: kubelet.sh      


##以下是脚本的详细情况
[root@node1 ~]# cat kubelet.sh 
#!/bin/bash

NODE_ADDRESS=$1
DNS_SERVER_IP=${2:-"10.0.0.2"}

cat <<EOF >/opt/kubernetes/cfg/kubelet

KUBELET_OPTS="--logtostderr=true \\
--v=4 \\
--hostname-override=${NODE_ADDRESS} \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet.config \\
--cert-dir=/opt/kubernetes/ssl \\
##指定容器镜像仓库
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

EOF

cat <<EOF >/opt/kubernetes/cfg/kubelet.config

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: ${NODE_ADDRESS}
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- ${DNS_SERVER_IP} 
clusterDomain: cluster.local.
failSwapOn: false
authentication:
##匿名功能
  anonymous:
    enabled: true
EOF

cat <<EOF >/usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet


[root@node1 ~]# cat proxy.sh 
#!/bin/bash

NODE_ADDRESS=$1

cat <<EOF >/opt/kubernetes/cfg/kube-proxy

KUBE_PROXY_OPTS="--logtostderr=true \\
--v=4 \\
--hostname-override=${NODE_ADDRESS} \\
--cluster-cidr=10.0.0.0/24 \\
--proxy-mode=ipvs \\
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"

EOF

cat <<EOF >/usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy
[root@node1 ~]#

在master上创建bootstrap.kubeconfig、kube-proxy kubeconfig文件，并且推送给node1和node2

[root@master bin]# cd /root
[root@master ~]# ls
anaconda-ks.cfg  initial-setup-ks.cfg  k8s  公共  模板  视频  图片  文档  下载  音乐  桌面
[root@master ~]# cd k8s/

##创建kubeconfig目录，用于编辑node配置文件的路径
[root@master k8s]# mkdir kubeconfig
[root@master k8s]# cd kubeconfig/

##查看令牌，并且复制token信息，用于下面的kubeconfig文件内编辑
[root@master kubeconfig]# cat /opt/kubernetes/cfg/token.csv 
978741334e39eb21d1c13050d6e8bb79,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

#编辑kubeconfig文件
[root@master kubeconfig]# vi kubeconfig 

##指定apiserver的地址
APISERVER=$1
##指定证书路径
SSL_DIR=$2

# 创建kubelet bootstrapping kubeconfig
export KUBE_APISERVER="https://$APISERVER:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=$SSL_DIR/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
##此处token复制于 /opt/kubernetes/cfg/token.csv
  --token=978741334e39eb21d1c13050d6e8bb79 \
  --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig


# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#----------------------

# 创建kube-proxy kubeconfig文件

kubectl config set-cluster kubernetes \
  --certificate-authority=$SSL_DIR/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=$SSL_DIR/kube-proxy.pem \
  --client-key=$SSL_DIR/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

##将kubectl写到全局变量中，方便使用，这样写是临时的
##可以直接追加到/etc/profile中
[root@master kubeconfig]# export PATH=$PATH:/opt/kubernetes/bin/
##使用kubectl查看master状态
[root@master kubeconfig]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
controller-manager   Healthy   ok                  

##生成配置文件，在/root/k8s/kubeconfig目录中
[root@master kubeconfig]# bash kubeconfig 192.168.43.101 /root/k8s/k8s-cert/
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".

[root@master kubeconfig]# ls
bootstrap.kubeconfig  kubeconfig  kube-proxy.kubeconfig

##推送配置文件bootstrap.kubeconfig和kube-proxy.kubeconfig到node1与node2
[root@master kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.43.102:/opt/kubernetes/cfg/
[root@master kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.43.103:/opt/kubernetes/cfg/

在master上创建bootstrap角色，赋予权限，用于连接apiserver签名，master通过bootstrap来管理node节点

[root@master kubeconfig]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

在node1上启动kubelet.sh脚本，请求连接master

##启动kubelet.sh脚本请求连接master，192.168.43.102为node1的本地地址
[root@node1 ~]# bash kubelet.sh 192.168.43.102
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.


[root@node1 ~]# ps aux | grep kube
root      39168  0.0  0.7 300512  7712 ?        Ssl  18:01   0:04 /opt/kubernetes/bin/flanneld --ip-masq --etcd-endpoints=https://192.168.43.101:2379,https://192.168.43.102:2379,https://192.168.43.103:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem
root      52741 14.5  4.7 374116 47088 ?        Ssl  20:35   0:01 /opt/kubernetes/bin/kubelet --logtostderr=true --v=4 --hostname-override=192.168.43.102 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --config=/opt/kubernetes/cfg/kubelet.config --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
root      52775  0.0  0.0 112676   984 pts/0    S+   20:36   0:00 grep --color=auto kube

在master上同意node1的请求，颁发证书

##使用kubectl get csr命令查看node1的证书请求
##pending表示等待集群给node1节点颁发证书
[root@master kubeconfig]# kubectl get csr
NAME                                                   AGE   REQUESTOR           CONDITION
node-csr-Ct8vYC2i9SqNGNPCJHkHQ8IXQlXk9dayRYiPmDjica4   23s   kubelet-bootstrap   Pending

##使用approve同意请求
[root@master kubeconfig]# kubectl certificate approve node-csr-Ct8vYC2i9SqNGNPCJHkHQ8IXQlXk9dayRYiPmDjica4
certificatesigningrequest.certificates.k8s.io/node-csr-Ct8vYC2i9SqNGNPCJHkHQ8IXQlXk9dayRYiPmDjica4 approved

##再次查看时，状态变为Approved，Issued
[root@master kubeconfig]# kubectl get csr
NAME                                                   AGE    REQUESTOR           CONDITION
node-csr-Ct8vYC2i9SqNGNPCJHkHQ8IXQlXk9dayRYiPmDjica4   104s   kubelet-bootstrap   Approved,Issued

##查看群集节点，node1已经被加入进来
[root@master kubeconfig]# kubectl get node
NAME             STATUS   ROLES    AGE   VERSION
192.168.43.102   Ready    <none>   26s   v1.12.3

在node1上启动porxy服务

##启动proxy服务
[root@node1 ~]# bash proxy.sh 192.168.43.102
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

##查看proxy服务状态
[root@node1 ~]# systemctl status kube-proxy.service 
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2020-04-27 20:38:16 CST; 14s ago
 Main PID: 53283 (kube-proxy)
   Memory: 9.2M
   CGroup: /system.slice/kube-proxy.service
           ‣ 53283 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=192.168.43.102 --cluster-ci...

4月 27 20:38:21 node1 kube-proxy[53283]: I0427 20:38:21.057764   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:21 node1 kube-proxy[53283]: I0427 20:38:21.069746   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:23 node1 kube-proxy[53283]: I0427 20:38:23.066704   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:23 node1 kube-proxy[53283]: I0427 20:38:23.081185   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:25 node1 kube-proxy[53283]: I0427 20:38:25.078317   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:25 node1 kube-proxy[53283]: I0427 20:38:25.092374   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:27 node1 kube-proxy[53283]: I0427 20:38:27.089776   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:27 node1 kube-proxy[53283]: I0427 20:38:27.102458   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:29 node1 kube-proxy[53283]: I0427 20:38:29.101396   53283 config.go:141] Calling handler.OnEndpointsUpdate
4月 27 20:38:29 node1 kube-proxy[53283]: I0427 20:38:29.111403   53283 config.go:141] Calling handler.OnEndpointsUpdat

##将node1的文件传给node2
[root@node1 ~]# scp -r /opt/kubernetes/ root@192.168.43.103:/opt
[root@node1 ~]# scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.43.103:/usr/lib/systemd/system/

部署node2节点

[root@node2 ~]# cd /opt/kubernetes/
[root@node2 kubernetes]# cd
##由于这些证书是被复制过来的，而node2会申请自己的证书，所以这里删除
[root@node2 ~]# cd /opt/kubernetes/ssl
[root@node2 ssl]# ls
kubelet-client-2020-04-27-20-37-32.pem  kubelet-client-current.pem  kubelet.crt  kubelet.key
[root@node2 ssl]# rm -rf *
[root@node2 ssl]# ls


##修改配置文件kubelet.config、kubelet、kube-proxy的IP地址
[root@node2 ssl]# cd ../cfg/
[root@node2 cfg]# ls
bootstrap.kubeconfig  flanneld  kubelet  kubelet.config  kubelet.kubeconfig  kube-proxy  kube-proxy.kubeconfig

[root@node2 cfg]# vi kubelet.config 


kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.43.103
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
##后面做DNS解析需要这个地址
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true



[root@node2 cfg]# vi kubelet


KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.43.103 \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"



[root@node2 cfg]# vi kube-proxy


KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.43.103 \
--cluster-cidr=10.0.0.0/24 \
--proxy-mode=ipvs \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"



##启动服务
[root@node2 cfg]# systemctl start kubelet.service 
[root@node2 cfg]# systemctl start kube-proxy.service
[root@node2 cfg]# 



##在master查看node2的请求，并且授权，颁发证书
[root@master kubeconfig]# kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-Ct8vYC2i9SqNGNPCJHkHQ8IXQlXk9dayRYiPmDjica4   9m45s   kubelet-bootstrap   Approved,Issued
node-csr-QWvHGaxIVxabiv7YaV5hlAIydy4b7QDZ5IdKJPEjBsU   30s     kubelet-bootstrap   Pending
[root@master kubeconfig]# kubectl certificate approve node-csr-QWvHGaxIVxabiv7YaV5hlAIydy4b7QDZ5IdKJPEjBsU
certificatesigningrequest.certificates.k8s.io/node-csr-QWvHGaxIVxabiv7YaV5hlAIydy4b7QDZ5IdKJPEjBsU approved

在master中查看集群节点情况

查看集群节点

[root@master ~]# kubectl get nodes
NAME             STATUS   ROLES    AGE   VERSION
192.168.43.102   Ready    <none>   18h   v1.12.3
192.168.43.103   Ready    <none>   18h   v1.12.3
[root@master ~]#