k8s1部署

最新推荐文章于 2025-08-04 10:56:20 发布

2201_76119904

最新推荐文章于 2025-08-04 10:56:20 发布

阅读量308

点赞数 8

CC 4.0 BY-SA版权

文章标签： kubernetes 容器云原生

本文链接：https://blog.csdn.net/2201_76119904/article/details/146053638

机器配置

ubuntu0 192.168.23.99
ubuntu1 192.168.23.98
ubuntu2 192.168.23.97

 etcd分布式集群搭建
	1.etcd概述 
etcd是分布式系统中最关键数据的分布式可靠键值存储，重点是：
	简单：定义明确、面向用户的API（gRPC）
	安全：自动TLS，可选客户端证书身份验证
	快速：以每秒10000次写入为基准
	可靠：使用Raft正确分布

etcd是用Go编写的，并使用Raft共识算法来管理高可用的复制日志。
etcd被许多公司用于生产环境，开发团队在关键部署场景中支持它，其中etcd经常与Kubernetes、locksmith、vulcan、Doorman等应用程序合作。严格的稳健性测试进一步确保了可靠性。
etcd使用http协议工作，支持tls，生产环境中一般使用的是https访问。
etcdctl是一个用于与etcd服务器交互的命令行工具，用户大多通过输入或获取键的值与etcd进行交互。
etcdctl用来与etcd对话的API版本可以通过etcdctl_API环境变量设置为版本2或3。
默认情况下，"etcdctl 3.4+"的etcdctl使用v3 API，早期版本"etcdctl 3.3-"默认为v2 API。
值得注意的是，etcd v2版本写入的数据无法通过v3的版本进行查询，说白了，就是高版本不兼容低版本，相当于v3重写了etcd。

生产环境中，官方推荐使用v3版本。

推荐阅读：
	https://github.com/etcd-io/etcd
	https://etcd.io/docs/


2.安装etcd的方式
	包管理方式安装： 安装简单，但是版本低
      apt -y install etcd-client
    二进制下载etcd的软件包
wget https://github.com/etcd-io/etcd/releases/download/v3.5.17/etcd-v3.5.17-linux-amd64.tar.gz   
	 解压etcd的二进制程序包到PATH环境变量路径
	 root@localhost:~# tar -xf etcd-v3.5.17-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.17-linux-amd64/etcd{,ctl}
root@localhost:~# ls /usr/local/bin/etcd
etcd     etcdctl 
	root@localhost:~# etcdctl version
	etcdctl version: 3.5.17
	API version: 3.5
	
 将软件包下发到所有节点	
root@ubuntu0:~# ls /usr/local/bin/etcd
etcd     etcdctl  
root@ubuntu0:~# scp /usr/local/bin/etcd*  192.168.23.98:/usr/local/bin/
etcd                                                                                                            100%   23MB 192.0MB/s   00:00    
etcdctl                                                                                                         100%   17MB 364.0MB/s   00:00    
root@ubuntu0:~# scp /usr/local/bin/etcd*  192.168.23.97:/usr/local/bin/
etcd                                                                                                            100%   23MB 203.1MB/s   00:00    
etcdctl                                                                                                         100%   17MB 426.9MB/s   00:00    

安装整数管理工具
root@ubuntu0:~# ll cfssl-v1.6.5.zip 
-rw-r--r-- 1 root root 10022492 Mar  5 13:17 cfssl-v1.6.5.zip
解压
root@ubuntu0:~# unzip cfssl-v1.6.5.zip 
Archive:  cfssl-v1.6.5.zip
  inflating: cfssl-certinfo_1.6.5_linux_amd64  
  inflating: cfssljson_1.6.5_linux_amd64  
  inflating: cfssl_1.6.5_linux_amd64  
root@ubuntu0:~# rm -rf cfssl-v1.6.5.zip 
root@ubuntu0:~# ll -l cfssl*
-rw-r--r-- 1 root root  8413336 Jun 15  2024 cfssl-certinfo_1.6.5_linux_amd64
-rw-r--r-- 1 root root 11890840 Jun 15  2024 cfssl_1.6.5_linux_amd64
-rw-r--r-- 1 root root  6205592 Jun 15  2024 cfssljson_1.6.5_linux_amd64

重命名，把后缀给去掉
root@ubuntu0:~# rename -v 's/_1.6.5_linux_amd64//g' cfssl*
root@ubuntu0:~# ls
1.txt  cfssl  cfssl-certinfo  cfssljson  etcd-v3.5.17-linux-amd64.tar.gz  snap
root@ubuntu0:~# ll cfssl*
-rw-r--r-- 1 root root 11890840 Jun 15  2024 cfssl
-rw-r--r-- 1 root root  8413336 Jun 15  2024 cfssl-certinfo
-rw-r--r-- 1 root root  6205592 Jun 15  2024 cfssljson
将整移到PATH环境变量下面
root@ubuntu0:~# mv cfssl* /usr/local/bin/
root@ubuntu0:~# chmod +x /usr/local/bin/cfssl*
root@ubuntu0:~#  ll /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root 11890840 Jun 15  2024 /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root  8413336 Jun 15  2024 /usr/local/bin/cfssl-certinfo*
-rwxr-xr-x 1 root root  6205592 Jun 15  2024 /usr/local/bin/cfssljson*


创建整数目录
root@ubuntu0:~#  mkdir -pv /th/certs/etcd && cd /th/certs/etcd/
mkdir: created directory '/th'
mkdir: created directory '/th/certs'
mkdir: created directory '/th/certs/etcd'

生成证书的CSR文件： 证书签发请求文件，配置了一些域名，公司，单位
root@ubuntu0:/th/certs/etcd# cat > etcd-ca-csr.json <<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

生成etcd CA证书和CA证书的key
root@ubuntu0:/th/certs/etcd# cfssl gencert -initca etcd-ca-csr.json|cfssljson -bare /th/certs/etcd/etcd-ca          
2025/03/06 01:10:34 [INFO] generating a new CA key and certificate from CSR
2025/03/06 01:10:34 [INFO] generate received request
2025/03/06 01:10:34 [INFO] received CSR
2025/03/06 01:10:34 [INFO] generating key: rsa-2048
2025/03/06 01:10:34 [INFO] encoded CSR
2025/03/06 01:10:34 [INFO] signed certificate with serial number 567088261002005575620106251375351103422358887693
root@ubuntu0:/th/certs/etcd# 

生成etcd证书的有效期为100年
root@ubuntu0:/th/certs/etcd# cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

生成证书的CSR文件： 证书签发请求文件，配置了一些域名，公司，单位
root@ubuntu0:/th/certs/etcd# cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

基于自建的ectd ca证书生成etcd的证书
root@ubuntu0:/th/certs/etcd# cfssl gencert \
>   -ca=/th/certs/etcd/etcd-ca.pem \
>   -ca-key=/th/certs/etcd/etcd-ca-key.pem \
>  -config=ca-config.json \
>   --hostname=127.0.0.1,ubuntu0,ubuntu1,ubuntu2,192.168.23.99,192.168.23.98,192.168.23.97 \  #主机名，ip，集群有三个主机，相当于添加信任的机器
>   --profile=kubernetes \  #名字就是ca-config.json里面的名字，可以更改，但要保持一致
>   etcd-csr.json  | cfssljson -bare /th/certs/etcd/etcd-server
2025/03/06 01:16:33 [INFO] generate received request
2025/03/06 01:16:33 [INFO] received CSR
2025/03/06 01:16:33 [INFO] generating key: rsa-2048
2025/03/06 01:16:33 [INFO] encoded CSR
2025/03/06 01:16:33 [INFO] signed certificate with serial number 207604995991302902138944823956042184795215360861

root@ubuntu0:/th/certs/etcd#  ll /th/certs/etcd/etcd-server*
-rw------- 1 root root 1675 Mar  6 01:23 /th/certs/etcd/etcd-server-key.pem
-rw-r--r-- 1 root root 1110 Mar  6 01:23 /th/certs/etcd/etcd-server.csr
-rw-r--r-- 1 root root 1444 Mar  6 01:23 /th/certs/etcd/etcd-server.pem

将etcd证书拷贝到其他两个master节点
root@ubuntu0:~# scp -r /th/certs 192.168.23.97:/th/certs/
root@ubuntu0:~# scp -r /th/certs 192.168.23.98:/th/certs/

创建etcd集群各节点配置文件

ubuntu1节点的配置文件
root@ubuntu1:~#  mkdir -pv /th/softwares/etcd
 cat > /th/softwares/etcd/etcd.config.yml <<'EOF'
name: 'ubuntu1'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.23.98:2380'
listen-client-urls: 'https://192.168.23.98:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.23.98:2380'
advertise-client-urls: 'https://192.168.23.98:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'ubuntu1=https://192.168.23.98:2380,node-ubuntu0=https://192.168.23.99:2380,ubuntu2=https://192.168.23.97:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/th/certs/etcd/etcd-server.pem'
  key-file: '/th/certs/etcd/etcd-server-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/th/certs/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/th/certs/etcd/etcd-server.pem'
  key-file: '/th/certs/etcd/etcd-server-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/th/certs/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF






ubuntu2节点的配置文件
root@ubuntu2:~#  mkdir -pv /th/softwares/etcd
 cat > /th/softwares/etcd/etcd.config.yml <<'EOF'
name: 'ubuntu2'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.23.97:2380'
listen-client-urls: 'https://192.168.23.97:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.23.97:2380'
advertise-client-urls: 'https://192.168.23.97:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'ubuntu1=https://192.168.23.98:2380,ubuntu0=https://192.168.23.99:2380,ubuntu2=https://192.168.23.97:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/th/certs/etcd/etcd-server.pem'
  key-file: '/th/certs/etcd/etcd-server-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/th/certs/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/th/certs/etcd/etcd-server.pem'
  key-file: '/th/certs/etcd/etcd-server-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/th/certs/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF


ubuntu0节点的配置文件
root@ubuntu0:~#  mkdir -pv /th/softwares/etcd
 cat > /th/softwares/etcd/etcd.config.yml <<'EOF'
name: 'ubuntu0'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.23.99:2380'
listen-client-urls: 'https://192.168.23.99:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.23.99:2380'
advertise-client-urls: 'https://192.168.23.99:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'ubuntu1=https://192.168.23.98:2380,ubuntu0=https://192.168.23.99:2380,ubuntu2=https://192.168.23.97:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/th/certs/etcd/etcd-server.pem'
  key-file: '/th/certs/etcd/etcd-server-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/th/certs/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/th/certs/etcd/etcd-server.pem'
  key-file: '/th/certs/etcd/etcd-server-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/th/certs/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

编写etcd的启动脚本
cat > /usr/lib/systemd/system/etcd.service <<'EOF'
[Unit]
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/th/softwares/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF

启动etcd集群
systemctl daemon-reload && systemctl enable --now etcd
systemctl status etcd


查看etcd集群状态
etcdctl --endpoints="192.168.23.99:2379,192.168.23.98:2379,192.168.23.97:2379" --cacert=/th/certs/etcd/etcd-ca.pem --cert=/th/certs/etcd/etcd-server.pem --key=/th/certs/etcd/etcd-server-key.pem  endpoint status --write-out=table
root@ubuntu0:~# etcdctl --endpoints="192.168.23.99:2379,192.168.23.98:2379,192.168.23.97:2379" --cacert=/th/certs/etcd/etcd-ca.pem --cert=/th/certs/etcd/etcd-server.pem --key=/th/certs/etcd/etcd-server-key.pem  endpoint status --write-out=table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.23.99:2379 | 8dafaf3d20cf2448 |  3.5.17 |   20 kB |     false |      false |         5 |         16 |                 16 |        |
| 192.168.23.98:2379 | 488036da648f79f1 |  3.5.17 |   20 kB |      true |      false |         5 |         16 |                 16 |        |
| 192.168.23.97:2379 | 1325c00f4d6b1e18 |  3.5.17 |   25 kB |     false |      false |         5 |         16 |                 16 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+


验证etcd高可用集群
	 停止leader节点
root@ubuntu1:~#  ss -ntl | egrep "2379|2380"
LISTEN 0      4096   192.168.23.98:2379      0.0.0.0:*          
LISTEN 0      4096   192.168.23.98:2380      0.0.0.0:*          
LISTEN 0      4096       127.0.0.1:2379      0.0.0.0:* 



查看现有集群环境，发现新leader诞生
root@ubuntu1:~# systemctl stop etcd
root@ubuntu1:~# ss -ntl | egrep "2379|2380"
root@ubuntu1:~# etcdctl --endpoints="192.168.23.99:2379,192.168.23.98:2379,192.168.23.97:2379" --cacert=/th/certs/etcd/etcd-ca.pem --cert=/th/certs/etcd/etcd-server.pem --key=/th/certs/etcd/etcd-server-key.pem  endpoint status --write-out=table
{"level":"warn","ts":"2025-03-06T07:29:45.976146Z","logger":"etcd-client","caller":"v3@v3.5.17/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004d2000/192.168.23.99:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 192.168.23.98:2379: connect: connection refused\""}
Failed to get the status of endpoint 192.168.23.98:2379 (context deadline exceeded)
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.23.99:2379 | 8dafaf3d20cf2448 |  3.5.17 |   20 kB |     false |      false |         6 |         17 |                 17 |        |
| 192.168.23.97:2379 | 1325c00f4d6b1e18 |  3.5.17 |   25 kB |      true |      false |         6 |         17 |                 17 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+


 再将之前的leader起来,再次验证
 root@ubuntu1:~# systemctl start etcd
root@ubuntu1:~# etcdctl --endpoints="192.168.23.99:2379,192.168.23.98:2379,192.168.23.97:2379" --cacert=/th/certs/etcd/etcd-ca.pem --cert=/th/certs/etcd/etcd-server.pem --key=/th/certs/etcd/etcd-server-key.pem  endpoint status --write-out=table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.23.99:2379 | 8dafaf3d20cf2448 |  3.5.17 |   20 kB |     false |      false |         6 |         18 |                 18 |        |
| 192.168.23.98:2379 | 488036da648f79f1 |  3.5.17 |   20 kB |     false |      false |         6 |         18 |                 18 |        |
| 192.168.23.97:2379 | 1325c00f4d6b1e18 |  3.5.17 |   25 kB |      true |      false |         6 |         18 |                 18 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+



- etcd的基本使用
	1.配置别名
[root@node-exporter42 ~]# vim .bashrc 
...
alias etcdctl='etcdctl --endpoints="10.0.0.41:2379,10.0.0.42:2379,10.0.0.43:2379" --cacert=/oldboyedu/certs/etcd/etcd-ca.pem --cert    =/oldboyedu/certs/etcd/etcd-server.pem --key=/oldboyedu/certs/etcd/etcd-server-key.pem' 
...
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# source .bashrc
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# etcdctl endpoint status --write-out=table

查看数据
root@ubuntu0:~# etcdctl get --prefix /   #默认系统是空的
root@ubuntu0:~# 

2.1 写入数据KEY的school，value等于oldboyedu
root@ubuntu0:~# etcdctl put school oldboyedu
OK
root@ubuntu0:~# 
root@ubuntu0:~# etcdctl put scheduler 调度器
OK
root@ubuntu0:~# 
root@ubuntu0:~# etcdctl put class Linux94
OK
root@ubuntu0:~#
root@ubuntu0:~# etcdctl put service 服务
OK
root@ubuntu0:~# 

2.2 查看数据
root@ubuntu0:~# etcdctl get school
school
oldboyedu
root@ubuntu0:~#
root@ubuntu0:~# etcdctl get school --keys-only   #只显示key
school

root@ubuntu0:~#
root@ubuntu0:~# etcdctl get school --print-value-only   #只显示values
oldboyedu
root@ubuntu0:~# 
root@ubuntu0:~# etcdctl get sch --prefix --keys-only  # 匹配"sch"开头的key
scheduler

school

root@ubuntu0:~# 
root@ubuntu0:~# 


		2.3 修改数据
root@ubuntu0:~# etcdctl get school --print-value-only
oldboyedu
root@ubuntu0:~# 
root@ubuntu0:~# 
root@ubuntu0:~# etcdctl put school laonanhai  # 如果key的值是存在，则直接覆盖
OK
root@ubuntu0:~# 
root@ubuntu0:~# etcdctl get school --print-value-only
laonanhai
root@ubuntu0:~#


		2.4 删除数据 
root@ubuntu0:~#
 etcdctl get sch --prefix --keys-only  
scheduler

school

root@ubuntu0:~# 
root@ubuntu0:~# etcdctl del school
1
root@ubuntu0:~# 
root@ubuntu0:~# etcdctl get sch --prefix --keys-only  
scheduler

root@ubuntu0:~# 
root@ubuntu0:~# etcdctl del sch --prefix
1
root@ubuntu0:~# 
root@ubuntu0:~# etcdctl get sch --prefix --keys-only  
root@ubuntu0:~# 
root@ubuntu0:~#

k8s简史

- K8S简史:
	- 2013年docker开源，IT界的福音，备受关注
	- 2014.06 Google有15年的容器编排Borg(博格，商业产品)使用经验，并将K8S(基于borg系统使用go语言研发)底层基于docker作为容器运行时开源
	- 2014.12 docker inc公司推出了K8S竞品，docker swarm
	- Google kubernets vs docker inc swarm 【3年对抗赛】 2017年年底结束，k8s完胜。（k8s 72% vs swarm %13）
		- 2014 coreOS 公司推出了rkt容器管理工具并站队K8S
		- 2015 Google公司将K8S开源并贡献给了CNCF组织，成为该组织的第一个毕业项目。
		- 2015 docker inc公司推出了OCI提议，主要针对容器运行时和镜像规范，并开源了runc。
		- 2016 Google推出了CRI规范，目前市面上没有任何产品可以满足，于是就开源了docker-shim组件（会调用docker接口并满足cri规范）支持CRI接口;
		- 2016，RedHat公司基于cri-o（既符合CRI也符合OCI规范）开发框架让rkt容器管理工具支持CRI接口;
		- 2017，docker inc公司将containerd从docker engine剥离，并将containerd开源给了CNCF组织，
			- containerd底层调用runc，因此该产品是支持OCI提议的;
			- containerd组件本身不支持CRI，因此社区大佬们（包含国内外）集体开发cri-containerd组件，最后合并到containerd项目
	- 2018 年国内开始流程K8S，各大云厂商已经开始大规模使用K8S集群，
		- 阿里云的ACK的SAAS产品
		- 腾讯云的TKE的SAAS产品
		- 华为云的CCE的SAAS产品
		- ucloud的UK8S的SAAS产品
		- 亚马逊的Amazon EKS的SAAS产品
		- 京东云，百度云等 
	- 2018年，coreOS公司被Redhat以2.5亿美元收购。
	- 2018年10月29日，IBM宣布以340亿美元的价格收购Red Hat。
		- 曾经一度，Docker方面的炒作非常猛。
		- Docker从Greylock Partners、Insight Partners和红杉资本等大牌投资者处筹资超过2.7亿美元，
		- 2018年估值达到最高峰：13.2亿美元。
	- 2019年2，Docker一分为二，将企业业务出售给了云咨询公司Mirantis（对于OpenStack代码贡献量非常大，能排到前3）。
	- 2020年，Google宣布K8S将在1.22+版本后弃用docker容器运行时，当时年底发布的最新版是1.20.X; 
	- 2020年3月11日公布的，当时Docker宣布被云计算软件巨头微软（Microsoft）以 6.7亿美元收购。
	- 2021年底 K8S 1.23的RC版本发布;
	- 2022年初，K8S 1.24横空出世，直接将docker-shim组件移除，而是使用containerd作为容器运行时;
	- 2023年初，K8S 1.27.X发布;
	- 2023年3月，K8S 1.23.17 发布了最后一个支持docker-shim的版本。
		- docker和Mirantis公司作为合作伙伴，将维护该项目，运维小伙伴如果需要在K8S 1.24及以后的版本使用docker的话，需要单独cri-docker组件。
	- 2024年初，K8S 1.30.x版本发布



CSI:
	Container Store Interface。容器存储接口，符合该接口的存储系统均能够在K8S上做存储。

CRI:
	Container Runtime Interface。容器运行接口，符合该接口的容器运行时均能够在K8S上做容器。

CNI:
	Container Network Interface。容器网络接口，符合该接口的网络组件，均可以在K8S集群中为容器提供网络服务。




Kubernetes，简称k8s，首字母K，末字母S，中间间隔8个字母，因此得名K8S。类似于的还有internationalization，I18n。


Kubernetes 也称为 K8s，是用于自动部署、扩缩和管理容器化应用程序的开源系统。

它将组成应用程序的容器组合成逻辑单元，以便于管理和服务发现。Kubernetes 源自Google 15 年生产环境的运维经验，同时凝聚了社区的最佳创意和实践。

Google 每周运行数十亿个容器，Kubernetes 基于与之相同的原则来设计，能够在不扩张运维团队的情况下进行规模扩展。

无论是本地测试，还是跨国公司，Kubernetes 的灵活性都能让你在应对复杂系统时得心应手。


Kubernetes 是开源系统，可以自由地部署在企业内部，私有云、混合云或公有云，让您轻松地做出合适的选择。

官网地址:
	https://kubernetes.io/zh-cn/
	
github开源地址:
	https://github.com/kubernetes/kubernetes




面试题: 你们公司用过K8S没？简单陈述K8S各个组件及作用？

- K8S架构图解 
	- master(主人) | control plane(控制面板)
		- etcd:
			数据存储。此数据库仅能api-server访问。也不是K8S内置的组件，由coreOS公司研发。
			
		- api-server:
			管理K8S集群的访问入口，可以对客户端请求进行认证，鉴权，数据交互。
			比如: 想要创建服务，获取资源都得经过api-server。
	
		- scheduler:
			调度器，负责Pod调度功能。
			
		- controller manager
			控制管理器，负责维护K8S集群状态。
			
		- cloud controller manager
			可选的部署组件，可以忽略，一般是云厂商使用。
	
	- salve(奴隶)  | worker node(工作节点)
		- kubelet
			负责Pod的生命周期管理并监控本地节点和容器状态周期性上报给api-server。
			是将cAdvisor核心功能内置到了kubelet实现了容器的监控。
		
		- kube-proxy		
			代理K8S集群的网络请求，主要是为K8S集群内部和外部提供访问路由。
			底层支持iptables和ipvs代理模式。
			
	- CNI组件：
		K8S本身并没有实现网络部分，而是定义了容器运行接口，典型代表: Flannel，Calico，Cannel，Cilium，...



- Kubernetes的三种网段说明
	- K8S各组件通信的网络
		使用时物理网卡，默认网段: 10.0.0.0/24。
		
	- 跨节点容器实现通信的网段:
		用户可以自定义，学习环境推荐: 10.100.0.0/16。
		
		但是在自定义网段时，要考虑将来能够分片的IP地址数量，"10.100.0.0/16"最多有65536个IP地址。
		
		如果将来容器运行的数量超过该规模时，应该考虑将网段地址调大，比如"10.0.0.0/8"。
		
		
	- Service网段：
		为容器提供负载均衡和服务发现功能。也是需要一个独立的网段，比如"10.200.0.0/16"最多有65536个IP地址。
		
		同理，如果规模较大时，应该考虑网段分配的问题。

请添加图片描述
Kubernetes的三种网段说明

k8s的部署方式

- kubernetes的部署方式
	- 官方默认都有两种部署方式: （在生产环境中都可以使用，且都支持高可用环境。咱们学习过程中，建议选择kubeadm。）
		- 二进制部署K8S集群
			手动部署K8S各个组件，配置文件，启动脚本及证书生成，kubeconfig文件。 
			配置繁琐，对新手不友好，尤其是证书管理。但是可以自定义配置信息，老手部署的话2小时起步，新手20小时+
			
		- kubeadm部署K8S集群
			是官方提供的一种快速部署K8S各组件的搭建，如果镜像准备就绪的情况下，基于容器的方式部署。
			需要提前安装kubelet，docker或者containerd，kubeadm组件。
			配置简单，适合新手。新手在镜像准备好的情况下，仅需要2分钟部署完毕。
			
			
	- 第三方提供的部署方式：
		国内公司:
			- 青云科技： kubesphere
				底层基于kubeadm快速部署K8S，提供了丰富的图形化管理界面。
			- kuboard
				底层基于kubeadm快速部署K8S，提供了丰富的图形化管理界面。
			- kubeasz
				底层基于二进制方式部署，结合ansible的playbook实现的快速部署管理K8S集群。
			
			
		国外的产品:
			- rancher:
				和国内的kubesphere很相似，也是K8S发行商，提供了丰富的图形化管理界面。
				还基于K8S研发出来了K3S，号称轻量级的K8S。
				
		云厂商：
			- 阿里云的ACK的SAAS产品
			- 腾讯云的TKE的SAAS产品
			- 华为云的CCE的SAAS产品
			- ucloud的UK8S的SAAS产品
			- 亚马逊的Amazon EKS的SAAS产品
			- 京东云，百度云的SAAS产品等。
		
		其他部署方式:
			- minikube:
				适合在windows部署K8S，适合开发环境搭建的使用。不建议生产环境部署。
				
			- kind:
				可以部署多套K8S环境，轻量级的命令行管理工具。
				 
			- yum:
				不推荐，版本支持较低，默认是1.5.2。
				
		CNCF技术蓝图：
			https://landscape.cncf.io/

 
	- 二进制部署和kubeadm部署的区别：
		相同点:
			都可以部署K8S高可用集群。
		
		不同点:
			- 1.部署难度: kubeadm简单.
			- 2.部署时间: kubeadm短时间。
			- 3.证书管理: 二进制需要手动生成，而kubeadm自建一个10年的CA证书，各组件证书有效期为1年。
			- 4.软件安装: kubeadm需要单独安装kubeadm,kubectl和kubelet组件，由kubelet组件启动K8S其他相关Pod，而二进制需要安装除了kubeadm的其他K8S组件。
			
			


- K8S版本选择
	我们选择的版本是K8S 1.23.17版本，该版本的第一个rc版本是2021年初，最后一个版本是23年年初结束。

K8S基础环境准备

推荐阅读:
	https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
	
环境准备:
	硬件配置: 2core 4GB
	磁盘: 50GB+
	操作系统: Ubuntu 22.04.04 LTS
	IP和主机名:
     192.168.23.99 ubuntu0
     192.168.23.98 ubuntu1
     192.168.23.97 ubuntu2
所有节点能够上网，机器必须"干净"。


2.关闭swap分区,一定要做
swapoff -a && sysctl -w vm.swappiness=0  # 临时关闭
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab  # 基于配置文件关闭

3.确保各个节点MAC地址或product_uuid唯一
ifconfig  eth0  | grep ether | awk '{print $2}'
cat /sys/class/dmi/id/product_uuid 
    温馨提示:
一般来讲，硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复。 
Kubernetes使用这些值来唯一确定集群中的节点。 如果这些值在每个节点上不唯一，可能会导致安装失败。


4.检查网络节点是否互通
ping baidu.com

5. 允许iptable检查桥接流量
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system

6.所有节点修改cgroup的管理进程为systemd
root@ubuntu0:~# ll autoinstall-docker-docker-compose.tar.gz 
-rw-r--r-- 1 root root 84289496 Mar  7 12:23 autoinstall-docker-docker-compose.tar.gz
root@ubuntu2:~# docker info  | grep "Cgroup Driver:"
 Cgroup Driver: systemd

7.所有节点安装kubeadm，kubelet，kubectl
你需要在每台机器上安装以下的软件包：
	kubeadm：
		用来初始化K8S集群的工具。
	kubelet：
		在集群中的每个节点上用来启动Pod和容器等。
	kubectl：
		用来与K8S集群通信的命令行工具。

kubeadm不能帮你安装或者管理kubelet或kubectl，所以你需要确保它们与通过kubeadm安装的控制平面(master)的版本相匹配。 如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。 

然而，控制平面与kubelet间的相差一个次要版本不一致是支持的，但kubelet的版本不可以超过"API SERVER"的版本。 例如，1.7.0版本的kubelet可以完全兼容1.8.0版本的"API SERVER"，反之则不可以。


8.K8S所有节点配置软件源
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update

9.查看一下当前环境支持的k8s版本
root@ubuntu1:~# apt-cache madison kubeadm
   kubeadm |  1.28.2-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.28.1-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.28.0-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
   kubeadm |  1.27.6-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages


10.安装 kubelet kubeadm kubectl
apt-get -y install kubelet=1.23.17-00 kubeadm=1.23.17-00 kubectl=1.23.17-00


11.检查各组件版本 是否相同
root@ubuntu2:~#  kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:33:14Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
root@ubuntu2:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:34:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
root@ubuntu2:~#  kubelet --version
Kubernetes v1.23.17
参考链接:
	https://kubernetes.io/zh/docs/tasks/tools/install-kubectl-linux/


12.检查时区
root@ubuntu2:~# timedatectl set-timezone Asia/Shanghai
root@ubuntu2:~# date -R 
Fri, 07 Mar 2025 20:57:26 +0800
 或者用下面的也可以
 ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime 
date -R

基于kubeadm组件初始化K8S的master组件

在master节点执行
1.提前导入镜像
root@ubuntu0:~# docker images
REPOSITORY                                                        TAG        IMAGE ID       CREATED       SIZE
registry.aliyuncs.com/google_containers/kube-apiserver            v1.23.17   62bc5d8258d6   2 years ago   130MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.23.17   1dab4fc7b6e0   2 years ago   120MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.23.17   bc6794cb54ac   2 years ago   51.9MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.23.17   f21c8d21558c   2 years ago   111MB
registry.aliyuncs.com/google_containers/etcd                      3.5.6-0    fce326961ae2   2 years ago   299MB
registry.aliyuncs.com/google_containers/coredns                   v1.8.6     a4ca41631cc7   3 years ago   46.8MB
registry.aliyuncs.com/google_containers/pause                     3.6        6270bb605e12   3 years ago   683kB

2.使用kubeadm初始化master节点
 kubeadm init --kubernetes-version=v1.23.17 --image-repository registry.aliyuncs.com/google_containers  --pod-network-cidr=10.100.0.0/16 --service-cidr=192.168.0.0/16  --service-dns-domain=xp.com --apiserver-advertise-address=192.168.23.99  # 如果有多块物理网卡时 --apiserver-advertise-address很有效，单块网卡无需配置此选项。如果配置的话，ip填写master节点的ip，   --service-dns-domain=域名,
如果出现问题的话 kubeadm reset 进行重置

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:



以下的token之类的要先保存好
kubeadm join 192.168.23.99:6443 --token tat4xh.p3oeogh9kk4y2pkd \
	--discovery-token-ca-cert-hash sha256:dc4bad82f07e646037d101eadfa2a8d1a562aad9ea7d40b32a97754871f92d32 

出现以上的提示就代表已经完成了初始化了
mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
把这些命令复制一下即可，如果不敲的话，他会连接本地的8080，因为默认他采用的是http协议
root@ubuntu0:~# kubectl get cs
The connection to the server localhost:8080 was refused - did you specify the right host or port?

把这些命令桥一下就可以了，此时k8s master组件就部署好了
root@ubuntu0:~# mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
root@ubuntu0:~# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
scheduler            Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}   
controller-manager   Healthy   ok           



温馨提示:
	你的token跟我不一致，请保存好，默认保留24小时，因此24小时内你的集群必须启动起来！
相关参数说明:
	--kubernetes-version:
		指定K8S master组件的版本号。		
	--image-repository:
		指定下载k8s master组件的镜像仓库地址。		
	--pod-network-cidr:
		指定Pod的网段地址。		
	--service-cidr:
		指定SVC的网段
	--service-dns-domain:
		指定service的域名。若不指定，默认为"cluster.local"。
使用kubeadm初始化集群时，可能会出现如下的输出信息:
	[init] 
		使用初始化的K8S版本。		
	[preflight] 
		主要是做安装K8S集群的前置工作，比如下载镜像，这个时间取决于你的网速。
	[certs] 
		生成证书文件，默认存储在"/etc/kubernetes/pki"目录哟。
	[kubeconfig]
		生成K8S集群的默认配置文件，默认存储在"/etc/kubernetes"目录哟。
	[kubelet-start] 
		启动kubelet，
		环境变量默认写入:"/var/lib/kubelet/kubeadm-flags.env"
		配置文件默认写入:"/var/lib/kubelet/config.yaml"
	[control-plane]
		使用静态的目录，默认的资源清单存放在:"/etc/kubernetes/manifests"。
		此过程会创建静态Pod，包括"kube-apiserver"，"kube-controller-manager"和"kube-scheduler"
	[etcd] 
		创建etcd的静态Pod，默认的资源清单存放在:""/etc/kubernetes/manifests"		
	[wait-control-plane] 
		等待kubelet从资源清单目录"/etc/kubernetes/manifests"启动静态Pod。
	[apiclient]
		等待所有的master组件正常运行。		
	[upload-config] 
		创建名为"kubeadm-config"的ConfigMap在"kube-system"名称空间中。		
	[kubelet] 
		创建名为"kubelet-config-1.22"的ConfigMap在"kube-system"名称空间中，其中包含集群中kubelet的配置
	[upload-certs] 
		跳过此节点，详情请参考”--upload-certs"		
	[mark-control-plane]
		标记控制面板，包括打标签和污点，目的是为了标记master节点。		
	[bootstrap-token] 
		创建token口令，例如:"kbkgsa.fc97518diw8bdqid"。
		如下图所示，这个口令将来在加入集群节点时很有用，而且对于RBAC控制也很有用处哟。
	[kubelet-finalize] 
		更新kubelet的证书文件信息
	[addons] 
		添加附加组件，例如:"CoreDNS"和"kube-proxy”
- master初始化不成功解决问题的方法 【选做！！不做！如果你做成了，这个步骤跳过！！！！】
可能存在的原因：
	- 由于没有禁用swap分区导致无法完成初始化;
	- 每个2core以上的CPU导致无法完成初始化;
	- 没有手动导入镜像;
	
解决方案:
	- 1.检查上面的是否有上面的情况
free -h 
lscpu
	- 2.重置当前节点环境
[root@master231 ~]# kubeadm reset -f
	- 3.再次尝试初始化master节点 
参考上面步骤。

kubeadm部署worker节点

两个worker节点都要同步部署
root@ubuntu1:~# docker images
REPOSITORY                                           TAG               IMAGE ID       CREATED         SIZE
flannel/flannel                                      v0.24.3           f6f0ee58f497   12 months ago   78.6MB
flannel/flannel-cni-plugin                           v1.4.0-flannel1   77c1250c26d9   13 months ago   9.87MB
registry.aliyuncs.com/google_containers/kube-proxy   v1.23.17          f21c8d21558c   2 years ago     111MB
registry.aliyuncs.com/google_containers/coredns      v1.8.6            a4ca41631cc7   3 years ago     46.8MB
registry.aliyuncs.com/google_containers/pause        3.6               6270bb605e12   3 years ago     683kB

查看主节点是没有任何的node节点的
root@ubuntu0:~# kubectl get nodes
NAME      STATUS     ROLES                  AGE   VERSION
ubuntu0   NotReady   control-plane,master   13h   v1.23.17


在worker节点执行加入的命令【复制你自己的token】
root@ubuntu1:~# kubeadm join 192.168.23.99:6443 --token tat4xh.p3oeogh9kk4y2pkd \
        --discovery-token-ca-cert-hash sha256:dc4bad82f07e646037d101eadfa2a8d1a562aad9ea7d40b32a97754871f92d32

此时在主节点上就把这两个节点加入成功了
root@ubuntu0:~# kubectl get no 
NAME      STATUS     ROLES                  AGE   VERSION
ubuntu0   NotReady   control-plane,master   13h   v1.23.17
ubuntu1   NotReady   <none>                 34s   v1.23.17
ubuntu2   NotReady   <none>                 29s   v1.23.17


温馨提示:
	此时K8S组件就算部署成功了，但是将来容器的网络依旧没有准备就绪，因此各节点处于“NotReady”状态

部署flannel CNI组件

1.所有节点导入镜像
root@ubuntu0:~# scp cni-v1.5.1-flannel-v0.25.6.tar.gz  192.168.23.98:/root
cni-v1.5.1-flannel-v0.25.6.tar.gz                                                                                                  100%   75MB 309.2MB/s   00:00    
root@ubuntu0:~# scp cni-v1.5.1-flannel-v0.25.6.tar.gz  192.168.23.97:/root
cni-v1.5.1-flannel-v0.25.6.tar.gz                                                                                                  100%   75MB 322.9MB/s   00:00    
root@ubuntu0:~# docker load -i cni-v1.5.1-flannel-v0.25.6.tar.gz 
30f8e4588c9e: Loading layer [==================================================>]  8.081MB/8.081MB
05bb1aee8d21: Loading layer [==================================================>]  2.836MB/2.836MB
Loaded image: flannel/flannel-cni-plugin:v1.5.1-flannel2
574b3797bb46: Loading layer [==================================================>]  3.223MB/3.223MB
8047f399ca1f: Loading layer [==================================================>]  16.68MB/16.68MB
e2508fe18818: Loading layer [==================================================>]  1.544MB/1.544MB
2231a338095f: Loading layer [==================================================>]  42.55MB/42.55MB
a5bb967c7bd1: Loading layer [==================================================>]  5.632kB/5.632kB
68105d35769e: Loading layer [==================================================>]  6.144kB/6.144kB
1cb69e15e4b5: Loading layer [==================================================>]  2.042MB/2.042MB
5f70bf18a086: Loading layer [==================================================>]  1.024kB/1.024kB
0b7f2865810b: Loading layer [==================================================>]  2.046MB/2.046MB
Loaded image: flannel/flannel:v0.25.6


修改网段
root@ubuntu0:~# wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
只需要更改这一个即可
root@ubuntu0:~# grep -n '10.100.0.0/16' kube-flannel.yml 
84:      "Network": "10.100.0.0/16",


把镜像改成相同版本的
root@ubuntu0:~# grep image kube-flannel.yml 
        image: ghcr.io/flannel-io/flannel:v0.26.5
        image: ghcr.io/flannel-io/flannel-cni-plugin:v1.6.2-flannel1
        image: ghcr.io/flannel-io/flannel:v0.26.5
root@ubuntu0:~# docker load -i cni-v1.5.1-flannel-v0.25.6.tar.gz 
30f8e4588c9e: Loading layer [==================================================>]  8.081MB/8.081MB
05bb1aee8d21: Loading layer [==================================================>]  2.836MB/2.836MB
Loaded image: flannel/flannel-cni-plugin:v1.5.1-flannel2
574b3797bb46: Loading layer [==================================================>]  3.223MB/3.223MB
8047f399ca1f: Loading layer [==================================================>]  16.68MB/16.68MB
e2508fe18818: Loading layer [==================================================>]  1.544MB/1.544MB
2231a338095f: Loading layer [==================================================>]  42.55MB/42.55MB
a5bb967c7bd1: Loading layer [==================================================>]  5.632kB/5.632kB
68105d35769e: Loading layer [==================================================>]  6.144kB/6.144kB
1cb69e15e4b5: Loading layer [==================================================>]  2.042MB/2.042MB
5f70bf18a086: Loading layer [==================================================>]  1.024kB/1.024kB
0b7f2865810b: Loading layer [==================================================>]  2.046MB/2.046MB
Loaded image: flannel/flannel:v0.25.6



更改为一摸一样的即可
root@ubuntu0:~# sed -i 's#v0.26.5#v0.25.6#g' kube-flannel.yml 
root@ubuntu0:~# grep image kube-flannel.yml 
        image: ghcr.io/flannel-io/flannel:v0.25.6
        image: ghcr.io/flannel-io/flannel-cni-plugin:v1.6.2-flannel1
        image: ghcr.io/flannel-io/flannel:v0.25.6
root@ubuntu0:~# vim kube-flannel.yml 
root@ubuntu0:~# 
root@ubuntu0:~# grep image kube-flannel.yml 
        image: docker.io/flannel/flannel:v0.25.6
        image: docker.io/flannel/flannel-cni-plugin:v1.5.1-flannel2
        image: docker.io/flannel/flannel:v0.25.6



安装flannel组件
root@ubuntu0:~# kubectl apply -f kube-flannel.yml 
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

查看flannel组件是否安装成功,保证STATUS是 Running
root@ubuntu0:~# kubectl get pod -o wide -n kube-flannel  
NAME                    READY   STATUS    RESTARTS   AGE   IP              NODE      NOMINATED NODE   READINESS GATES
kube-flannel-ds-8dbln   1/1     Running   0          32s   192.168.23.99   ubuntu0   <none>           <none>
kube-flannel-ds-9cflr   1/1     Running   0          6s    192.168.23.97   ubuntu2   <none>           <none>
kube-flannel-ds-q8sss   1/1     Running   0          6s    192.168.23.98   ubuntu1   <none>           <none>

此时就变成了Ready
root@ubuntu0:~# kubectl get node
NAME      STATUS   ROLES                  AGE     VERSION
ubuntu0   Ready    control-plane,master   19h     v1.23.17
ubuntu1   Ready    <none>                 5h44m   v1.23.17
ubuntu2   Ready    <none>                 5h43m   v1.23.17


检查网络是否正确，确保这俩网卡在同一个网段
root@ubuntu0:~# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.100.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::54d6:f2ff:feae:a046  prefixlen 64  scopeid 0x20<link>
        ether 56:d6:f2:ae:a0:46  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 12 overruns 0  carrier 0  collisions 0

root@ubuntu0:~# ifconfig cni0     
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.100.0.1  netmask 255.255.255.0  broadcast 10.100.0.255
        inet6 fe80::5491:39ff:fe00:ff5d  prefixlen 64  scopeid 0x20<link>
        ether 56:91:39:00:ff:5d  txqueuelen 1000  (Ethernet)
        RX packets 606  bytes 52059 (52.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 658  bytes 82165 (82.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

fannel网卡cni0网卡缺失解决方案
 1问题描述
部分节点不存在cni0网络设备，仅有flannel.1设备，此时我们需要手动创建cni0网桥设备
 2 解决方案
如果有节点没有cni0网卡，建议大家手动创建相应的网桥设备，但是注意网段要一致
- 手动创建cni0网卡
---> 假设 master231的flannel.1是10.100.0.0网段。
ip link add cni0 type bridge
ip link set dev cni0 up
ip addr add 10.100.0.1/24 dev cni0


---> 假设 worker233的flannel.1是10.100.2.0网段。
ip link add cni0 type bridge
ip link set dev cni0 up
ip addr add 10.100.2.1/24 dev cni0

如果添加错误删除的话
ip link del cni0 



验证Pod的CNI网络是否正常
	1 编写Pod资源清单
root@ubuntu0:~#  cat > network-cni-test.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: xiuxian-v1
spec:
  nodeName: ubuntu1
  containers:
  - image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
    name: xiuxian

---

apiVersion: v1
kind: Pod
metadata:
  name: xiuxian-v2
spec:
  nodeName: ubuntu2
  containers:
  - image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v2
    name: xiuxian
EOF

   2.创建Pod资源
root@ubuntu0:~# kubectl apply -f network-cni-test.yaml 
pod/xiuxian-v1 created
pod/xiuxian-v2 created

3.查看Pod资源你列表
root@ubuntu0:~# kubectl get pods -o wide
NAME         READY   STATUS    RESTARTS   AGE     IP           NODE      NOMINATED NODE   READINESS GATES
xiuxian-v1   1/1     Running   0          3h26m   10.100.1.5   ubuntu1   <none>           <none>
xiuxian-v2   1/1     Running   0          3h26m   10.100.2.5   ubuntu2   <none>           <none>

systemctl restart kubelet

显示这个就是正常的
root@ubuntu0:~# curl 10.100.2.5
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v2</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: red">凡人修仙传 v2 </h1>

配置自动补全

添加环境变量
root@ubuntu0:~# kubectl completion bash > ~/.kube/completion.bash.inc
root@ubuntu0:~# echo source '$HOME/.kube/completion.bash.inc' >> ~/.bashrc 
root@ubuntu0:~# source ~/.bashrc
验证自动补全
root@ubuntu0:~# kubectl 
alpha          attach         completion     debug          edit           help           patch          rollout        top            
annotate       auth           config         delete         exec           kustomize      plugin         run            uncordon       
api-resources  autoscale      cordon         describe       explain        label          port-forward   scale          version        
api-versions   certificate    cp             diff           expose         logs           proxy          set            wait           
apply          cluster-info   create         drain          get            options        replace        taint

K8S主机巡检流程

1.查看master组件是否正常
root@ubuntu0:~#  kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
scheduler            Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}  


2.查看工作节点是否就绪
root@ubuntu0:~# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
ubuntu0   Ready    control-plane,master   23h   v1.23.17
ubuntu1   Ready    <none>                 9h    v1.23.17
ubuntu2   Ready    <none>                 9h    v1.23.17

3.检查falnnel组件是否正常运行
root@ubuntu0:~# kubectl get pods -n kube-flannel -o wide
NAME                    READY   STATUS    RESTARTS      AGE     IP              NODE      NOMINATED NODE   READINESS GATES
kube-flannel-ds-8dbln   1/1     Running   2 (26m ago)   4h15m   192.168.23.99   ubuntu0   <none>           <none>
kube-flannel-ds-9cflr   1/1     Running   2 (26m ago)   4h15m   192.168.23.97   ubuntu2   <none>           <none>
kube-flannel-ds-q8sss   1/1     Running   2             4h15m   192.168.23.98   ubuntu1   <none>           <none>

4.检查cni0和flannel网卡是否存在
若不存在则创建。

5.验证网络是否正常
root@ubuntu0:~# curl 10.100.2.5
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v2</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: red">凡人修仙传 v2 </h1>

在这里插入图片描述

k8s资源

查看k8s内置资源
root@ubuntu0:~# kubectl api-resources
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
bindings                                       v1                                     true         Binding
componentstatuses                 cs           v1                                     false        ComponentStatus
configmaps                        cm           v1                                     true         ConfigMap
root@ubuntu0:~# kubectl api-resources|wc -l
57

name： 名称
SHORTNAMES： 简称
APIVERSION： API版本
KIND： 类型


apiVersion: 
	声明资源API的版本号。	
kind:
	声明资源的类型。
metadata:
	声明资源你的元数据信息。这个元数据信息包括但不限于名称，标签，名称空间，资源注解，资源的创建时间等。
spec:
	声明资源的期望状态，用户希望资源如何运行，比如Pod资源我们要考虑启动容器名称，容器使用的镜像，启动命令，暴露端口，环境变量，资源限制等。
status:
	资源的状态，一般情况下，不需要维护该字段，是K8S集群内部自行维护。
lables:      #标签，筛选操作；一般用于集群内部的筛选功能；

举个例子: 查看ubuntu1节点node资源的资源清单
root@ubuntu0:~# kubectl get no  ubuntu1 -o json  # -o 就是output输出的意思

查看检查状态的资源清单，显示所有的
root@ubuntu0:~# kubectl get   cs -o json
root@ubuntu0:~# kubectl get   cs
#显示单个的
root@ubuntu0:~# kubectl get cs   etcd-0 -o json
Warning: v1 ComponentStatus is deprecated in v1.19+
{
    "apiVersion": "v1",
    "conditions": [
        {
            "message": "{\"health\":\"true\",\"reason\":\"\"}",
            "status": "True",
            "type": "Healthy"
        }
    ],
    "kind": "ComponentStatus",
    "metadata": {
        "creationTimestamp": null,
        "name": "etcd-0"
    }
}


自己编写资源清单
root@ubuntu0:~/manifests/pods# pwd 
/root/manifests/pods
root@ubuntu0:~/manifests/pods# cat 01-xiuxian.yml 
#声明api版本号，可以通过kubectl api-resources,来查看对应的些的版本号
#我们此时写的是pod
apiVersion: v1
#声明资源类型
kind: Pod
#声明元数据
metadata:
  #声明资源的名称
  name: ubuntu-xiuxian-v1
#定义用户期望的资源
spec:
  #定义容器信息
  containers:
  #声明容器的名字
  - name: ubuntu-xiuxian-v1
  #指定镜像的信息
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1

创建资源
root@ubuntu0:~/manifests/pods# kubectl create -f 01-xiuxian.yml 
pod/ubuntu-xiuxian-v1 created

查看资源
root@ubuntu0:~/manifests/pods# kubectl get pods
NAME                READY   STATUS    RESTARTS      AGE
ubuntu-xiuxian-v1   1/1     Running   0             11s
xiuxian-v1          1/1     Running   2 (88m ago)   3d4h
xiuxian-v2          1/1     Running   2 (88m ago)   3d4h
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide  #wide显示更多的信息
NAME                READY   STATUS    RESTARTS      AGE     IP           NODE      NOMINATED NODE   READINESS GATES
ubuntu-xiuxian-v1   1/1     Running   0             2m26s   10.100.1.8   ubuntu1   <none>           <none>
xiuxian-v1          1/1     Running   2 (91m ago)   3d4h    10.100.1.7   ubuntu1   <none>           <none>
xiuxian-v2          1/1     Running   2 (91m ago)   3d4h    10.100.2.7   ubuntu2   <none>           <none>
#此时就部署成功了
root@ubuntu0:~/manifests/pods# curl 10.100.1.8
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v1</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: green">凡人修仙传 v1 </h1>
    <div>
      <img src="1.jpg">
    <div>
  </body>

</html>

#删除
root@ubuntu0:~/manifests/pods# kubectl delete -f 01-xiuxian.yml 
pod "ubuntu-xiuxian-v1" deleted
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide
NAME         READY   STATUS    RESTARTS      AGE    IP           NODE      NOMINATED NODE   READINESS GATES
xiuxian-v1   1/1     Running   2 (92m ago)   3d4h   10.100.1.7   ubuntu1   <none>           <none>
xiuxian-v2   1/1     Running   2 (92m ago)   3d4h   10.100.2.7   ubuntu2   <none>           <none>


对于资源名称，不能出现汉字，大写字母，等特殊符号，有效的值为小写字母，数字，横杠(-)，点(.)。
解决方案:
	- 1.修改名称为合法的字符即可。
		[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*


那k8s集群内部可以访问，但用户又不在k8s环境上，用户该如何访问呢
root@ubuntu0:~/manifests/pods# cat 01-xiuxian.yml 
#声明api版本号，可以通过kubectl api-resources,来查看对应的些的版本号
#我们此时写的是pod
apiVersion: v1
#声明资源类型
kind: Pod
#声明元数据
metadata:
  #声明资源的名称
  name: ubuntu-xiuxian-v2
#定义用户期望的资源
spec:
  #使用宿主机网络
  hostNetwork: true
  #定义容器信息
  containers:
  #声明容器的名字
  - name: ubuntu-xiuxian-v1
  #指定镜像的信息
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1

#此时用的就是宿主机网络了
root@ubuntu0:~/manifests/pods# kubectl create -f 01-xiuxian.yml 
pod/ubuntu-xiuxian-v2 created
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide
NAME                READY   STATUS    RESTARTS       AGE    IP              NODE      NOMINATED NODE   READINESS GATES
ubuntu-xiuxian-v2   1/1     Running   0              12s    192.168.23.97   ubuntu2   <none>           <none>
xiuxian-v1          1/1     Running   2 (117m ago)   3d4h   10.100.1.7      ubuntu1   <none>           <none>
xiuxian-v2          1/1     Running   2 (117m ago)   3d4h   10.100.2.7      ubuntu2   <none>           <none>


如果你镜像名称写错了，在创建的时候他也不会报错，但是在查看的时候他会报镜像信息无效
root@ubuntu0:~/manifests/pods# cat 01-xiuxian.yml 
#声明api版本号，可以通过kubectl api-resources,来查看对应的些的版本号
#我们此时写的是pod
apiVersion: v1
#声明资源类型
kind: Pod
#声明元数据
metadata:
  #声明资源的名称
  name: ubuntu-xiuxian-v2
#定义用户期望的资源
spec:
  #使用宿主机网络
  hostNetwork: true
  #定义容器信息
  containers:
  #声明容器的名字
  - name: ubuntu-xiuxian-v1
  #指定镜像的信息
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/APP:v1

root@ubuntu0:~/manifests/pods# kubectl get pods -o wide
NAME                READY   STATUS             RESTARTS        AGE     IP              NODE      NOMINATED NODE   READINESS GATES
ubuntu-xiuxian-v2   0/1     InvalidImageName   0               8s      192.168.23.98   ubuntu1   <none>           <none>
xiuxian-v1          1/1     Running            4 (6m21s ago)   4d15h   10.100.1.10     ubuntu1   <none>           <none>
xiuxian-v2          1/1     Running            4 (6m20s ago)   4d15h   10.100.2.9      ubuntu2   <none>           <none>

可以用这个命令查看这个详细信息，来查看具体那里报错了  describe 后面跟类型，这个是pod类型
root@ubuntu0:~/manifests/pods# kubectl describe pod ubuntu-xiuxian-v2
  Warning  InspectFailed  1s (x9 over 77s)  kubelet            Failed to apply default image tag "registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/APP:v1": couldn't parse image reference "registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/APP:v1": invalid reference format: repository name must be lowercase


那如果pod的容器真有问题，该如何进去容器排查呢
root@ubuntu0:~/manifests/pods# cat 01-xiuxian.yml 
#声明api版本号，可以通过kubectl api-resources,来查看对应的些的版本号
#我们此时写的是pod
apiVersion: v1
#声明资源类型
kind: Pod
#声明元数据
metadata:
  #声明资源的名称
  name: ubuntu-xiuxian-v2
#定义用户期望的资源
spec:
  #使用宿主机网络
  hostNetwork: true
  #定义容器信息
  containers:
  #声明容器的名字
  - name: ubuntu-xiuxian-v1
  #指定镜像的信息
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
    #修改容器的启动命令，相当于替换了dockerfile的ENTRYPOINT指令
    command: ["sleep","3600"]

root@ubuntu0:~/manifests/pods# kubectl create -f 01-xiuxian.yml 
pod/ubuntu-xiuxian-v2 created
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide
NAME                READY   STATUS    RESTARTS      AGE     IP              NODE      NOMINATED NODE   READINESS GATES
ubuntu-xiuxian-v2   1/1     Running   0             6s      192.168.23.97   ubuntu2   <none>           <none>
xiuxian-v1          1/1     Running   4 (24m ago)   4d15h   10.100.1.10     ubuntu1   <none>           <none>
xiuxian-v2          1/1     Running   4 (24m ago)   4d15h   10.100.2.9      ubuntu2   <none>           <none>
root@ubuntu0:~/manifests/pods# curl 192.168.23.97
curl: (7) Failed to connect to 192.168.23.97 port 80 after 0 ms: Connection refused

root@ubuntu0:~/manifests/pods# kubectl exec -it ubuntu-xiuxian-v2  -- sh
/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 sleep 3600
    7 root      0:00 sh
   13 root      0:00 ps -ef
/ # nginx   #此时运行以下nginx就可以了
2025/03/14 00:26:20 [notice] 14#14: using the "epoll" event method
2025/03/14 00:26:20 [notice] 14#14: nginx/1.20.1
2025/03/14 00:26:20 [notice] 14#14: built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1) 
2025/03/14 00:26:20 [notice] 14#14: OS: Linux 5.15.0-134-generic
2025/03/14 00:26:20 [notice] 14#14: getrlimit(RLIMIT_NOFILE): 524288:524288
2025/03/14 00:26:20 [notice] 15#15: start worker processes
2025/03/14 00:26:20 [notice] 15#15: start worker process 16
/ # 2025/03/14 00:26:20 [notice] 15#15: start worker process 17
2025/03/14 00:26:20 [notice] 15#15: start worker process 18
2025/03/14 00:26:20 [notice] 15#15: start worker process 19


args相当于dockerfile的cmd指令
root@ubuntu0:~/manifests/pods# cat 02-xiuxian.yml 
apiVersion: v1
kind: Pod
metadata:
  name: argsapps-v1
spec:
  hostNetwork: true
  containers:
  - name: xiuxianapps-v1
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
    #相当于dockerfile的cmd指令
    args: ["sleep","3600"]



command和args连用，如果连用args会作为command的参数使用
root@ubuntu0:~/manifests/pods# cat 03-command-args.yml 
apiVersion: v1
kind: Pod
metadata:
  name: command-args
spec:
  hostNetwork: true
  containers:
  - name: cpm
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
    command: ["tail"]
    args: ["-f","/etc/hosts"]
root@ubuntu0:~/manifests/pods# kubectl create -f 03-command-args.yml 
pod/command-args created
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide
NAME           READY   STATUS    RESTARTS      AGE    IP              NODE      NOMINATED NODE   READINESS GATES
command-args   1/1     Running   0             9s     192.168.23.97   ubuntu2   <none>           <none>
xiuxian-v1     1/1     Running   4 (13h ago)   5d4h   10.100.1.10     ubuntu1   <none>           <none>
xiuxian-v2     1/1     Running   4 (13h ago)   5d4h   10.100.2.9      ubuntu2   <none>           <none>
root@ubuntu0:~/manifests/pods# kubectl exec -it command-args -- sh
/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 tail -f /etc/hosts
    6 root      0:00 sh
   12 root      0:00 ps -ef
	

像容器传递环境变量 env
查看现有的环境变量
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide
NAME           READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE   READINESS GATES
command-args   1/1     Running   0          13m   10.100.2.2   ubuntu2   <none>           <none>
root@ubuntu0:~/manifests/pods# kubectl exec -it command-args -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=command-args
TERM=xterm
KUBERNETES_PORT=tcp://192.168.0.1:443
KUBERNETES_PORT_443_TCP=tcp://192.168.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=192.168.0.1
KUBERNETES_SERVICE_HOST=192.168.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
NGINX_VERSION=1.20.1
NJS_VERSION=0.5.3
PKG_RELEASE=1
HOME=/root


root@ubuntu0:~/manifests/pods# cat 02-xiuxian.yml 
apiVersion: v1
kind: Pod
metadata:
  name: argsapps-v1
spec:
  hostNetwork: true
  containers:
  - name: xiuxianapps-v1
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
    #向容器传递环境变量
    env: 
    #指定环境变量的名称
    - name: SCHOOL
    #指定环境变量的值
      value: oldoy
    - name: CLASS
      value: linux94
root@ubuntu0:~/manifests/pods# kubectl create -f 02-xiuxian.yml 
pod/argsapps-v1 created
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide
NAME           READY   STATUS    RESTARTS   AGE   IP              NODE      NOMINATED NODE   READINESS GATES
argsapps-v1    1/1     Running   0          11s   192.168.23.98   ubuntu1   <none>           <none>
command-args   1/1     Running   0          17m   10.100.2.2      ubuntu2   <none>           <none>
root@ubuntu0:~/manifests/pods# kubectl exec -it argsapps-v1 -- env|grep -E 'SCHOOL|CLASS'
SCHOOL=oldoy
CLASS=linux94

root@ubuntu0: kubectl apply -f 06-pods-xiuxian-env.yaml   # apply命令可以执行多次，如果资源不存在则创建，存在则可以尝试修改。，如果资源没有发生任何改变的话，他是unchanged
pod/oldboyedu-xiuxian-env unchanged

指定节点调度，需要在nodeName字段中指定正确的节点调度， kubectl get nodes 查看节点名称
root@ubuntu0:~/manifests/pods# cat 02-xiuxian.yml 
apiVersion: v1
kind: Pod
metadata:
  name: argsapps-v1
spec:
  #指定节点调度，该节点名称必须是在etcd数据库中存在
  nodeName: ubuntu1
  containers:
  - name: xiuxianapps-v1
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
    env: 
    - name: SCHOOL
      value: oldoy
    - name: CLASS
      value: linux94
root@ubuntu0:~/manifests/pods# kubectl apply -f 02-xiuxian.yml 
pod/argsapps-v1 created
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE   READINESS GATES
argsapps-v1   1/1     Running   0          5s    10.100.1.2   ubuntu1   <none>           <none>



练习部署MySQL实战案例:
	在K8S集群部署"mysql:8.0.36-oracle"要求如下:
		- 1.部署服务在worker233节点;
		- 2.要求用户名是: linux94，密码是: oldboyedu，root密码为空;
		- 3.在windows使用图形化工具链接测试，创建一个student表，要求里面记录你们组最少3个人的爱好，最少3个字段，id,name,hobby;
worker233节点导入镜像
[root@worker233 ~]# wget http://192.168.13.253/Resources/Docker/images/WordPress/oldboyedu-mysql-v8.0.36-oracle.tar.gz
[root@worker233 ~]# docker load  -i oldboyedu-mysql-v8.0.36-oracle.tar.gz 
	2.编写资源清单
[root@master231 case-demo]# cat 01-pods-mysql.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: oldboyedu-mysql
spec:
  nodeName: worker233
  hostNetwork: true
  containers:
  - name: db
    image: mysql:8.0.36-oracle
    env:
    - name: MYSQL_ALLOW_EMPTY_PASSWORD
      value: "yes"
    - name: MYSQL_USER
      value: linux94
    - name: MYSQL_PASSWORD
      value: oldboyedu
    - name: MYSQL_DATABASE
      value: yinzhengjie


镜像的拉取策略:
root@ubuntu0:~/manifests/pods# kubectl explain po.spec.containers.imagePullPolicy
KIND:     Pod
VERSION:  v1

FIELD:    imagePullPolicy <string>

DESCRIPTION:
     Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always
     if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated.
     More info:
     https://kubernetes.io/docs/concepts/containers/images#updating-images

	- Never
		如果本地有镜像则尝试启动		
		如果本地没有镜像则不会启动。		
	- Always:
		如果本地有镜像则将本地的摘要信息和远程仓库的信息进行比对，若相同则使用本地镜像缓存，若不同，则直接去远程仓库拉取。	
		如果本地没有镜像则直接拉取镜像。			
	- IfNotPresent:
		如果本地有镜像则尝试启动。		
		如果本地没有镜像，则会去远程仓库拉取镜像。			
	默认策略:
		当镜像的TAG是latest时，则默认的镜像下载策略为Always，若镜像的TAG不是latest时，则镜像下载策略为IfNotPresent





将WordPress部署在ubuntu1节点，数据库部署在ubuntu1节点，要求windows能够正常访问并发表文章
	2.编写资源清单
[root@master231 case-demo]# cat 02-pods-mysql-wordpress.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: oldboyedu-mysql
spec:
  nodeName: worker233
  hostNetwork: true
  containers:
  - name: db
    image: mysql:8.0.36-oracle
    imagePullPolicy: IfNotPresent
    env:
    - name: MYSQL_ALLOW_EMPTY_PASSWORD
      value: "yes"
    - name: MYSQL_USER
      value: linux94
    - name: MYSQL_PASSWORD
      value: oldboyedu
    - name: MYSQL_DATABASE
      value: yinzhengjie

---

apiVersion: v1
kind: Pod
metadata:
  name: oldboyedu-wp
spec:
  nodeName: worker232
  hostNetwork: true
  containers:
  - name: db
    # 生成环境中尽量避免使用latest标签。
    # 如果非要用，且想要使用之前的镜像则需要配置镜像下载策略。
    image: wordpress:latest
    imagePullPolicy: IfNotPresent
    env:
    - name: WORDPRESS_DB_HOST
      value: 10.0.0.233
    - name: WORDPRESS_DB_USER
      value: linux94
    - name: WORDPRESS_DB_PASSWORD
      value: oldboyedu
    - name: WORDPRESS_DB_NAME
      value: yinzhengjie

管理K8S的API类型声明式和响应式

- 声明式：
	基于资源清单管理K8S集群资源。
root@ubuntu0:~/manifests/pods# cat 01-xiuxian.yml 
#声明api版本号，可以通过kubectl api-resources,来查看对应的些的版本号
#我们此时写的是pod
apiVersion: v1
#声明资源类型
kind: Pod
#声明元数据
metadata:
  #声明资源的名称
  name: ubuntu-xiuxian-v2
#定义用户期望的资源
spec:
  #使用宿主机网络
  hostNetwork: true
  #定义容器信息
  containers:
  #声明容器的名字
  - name: ubuntu-xiuxian-v1
  #指定镜像的信息
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
    command: ["sleep","3600"]

- 响应式：
		不通过资源清单管理K8S集群资源，而是基于命令行的方式管理K8S集群资源。
root@ubuntu0:~/manifests/pods# kubectl run --help
root@ubuntu0~/manifests/pods#:kubectl run oldboyedu-xiuxian-v1 --image=registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1 --env="SCHOOL=oldboyedu" --env="CLASS=linux94"

响应式管理标签
root@ubuntu0:~/manifests/pods# kubectl get pods --show-labels 
NAME                   READY   STATUS    RESTARTS   AGE     LABELS
oldboyedu-xiuxian-v1   1/1     Running   0          3m19s   run=oldboyedu-xiuxian-v1


给pod打标签
root@ubuntu0:~/manifests/pods# kubectl label pods oldboyedu-xiuxian-v1 school=linux94 class=linux93
pod/oldboyedu-xiuxian-v1 labeled
root@ubuntu0:~/manifests/pods# kubectl get pods -o wide --show-labels  
NAME                   READY   STATUS    RESTARTS   AGE     IP           NODE      NOMINATED NODE   READINESS GATES   LABELS
oldboyedu-xiuxian-v1   1/1     Running   0          7m14s   10.100.1.7   ubuntu1   <none>           <none>            class=linux93,run=oldboyedu-xiuxian-v1,school=linux94
root@ubuntu0:~/manifests/pods# 
如果向修改已经存在的标签，他会报错，在后面添加--overwrite即可
root@ubuntu0:~/manifests/pods# kubectl label pods oldboyedu-xiuxian-v1 school=linux92
error: 'school' already has a value (linux94), and --overwrite is false
root@ubuntu0:~/manifests/pods# kubectl label pods oldboyedu-xiuxian-v1 school=linux92 --overwrite 
pod/oldboyedu-xiuxian-v1 labeled
删除标签
root@ubuntu0:~/manifests/pods# kubectl label pods oldboyedu-xiuxian-v1 run- school-
pod/oldboyedu-xiuxian-v1 unlabeled


根据标签查找
root@ubuntu0:~/manifests/pods# kubectl get pods --show-labels 
NAME                   READY   STATUS    RESTARTS   AGE   LABELS
oldboyedu-xiuxian-v1   1/1     Running   0          16m   class=linux93,shool=laonanhai
root@ubuntu0:~/manifests/pods# kubectl get pods -l class
NAME                   READY   STATUS    RESTARTS   AGE
oldboyedu-xiuxian-v1   1/1     Running   0          16m
root@ubuntu0:~/manifests/pods# kubectl get pods -l school!=linux95
NAME                   READY   STATUS    RESTARTS   AGE
oldboyedu-xiuxian-v1   1/1     Running   0          17m


声明式
root@ubuntu0:~/manifests/pods# cat 03-command-args.yml 
apiVersion: v1
kind: Pod
metadata:
  name: command-args
  labels: 
    school: laonanhai
    class: linux94
spec:
  containers:
  - name: cpm
    image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1
root@ubuntu0:~/manifests/pods# kubectl apply -f 03-command-args.yml 
pod/command-args created
root@ubuntu0:~/manifests/pods# kubectl get pods --show-labels 
NAME           READY   STATUS    RESTARTS   AGE   LABELS
command-args   1/1     Running   0          7s    class=linux94,school=laonanhai

- 声明式和响应式的区别:
	相同点:
		都可以管理资源。
		
	不同点:
		- 声明式修改后需要使用apply指令进行应用才能生效;
		- 响应式会立即生效

harbor自定义证书集成K8S集群

1.安装docker和docker-compose环境

2.下载harbor 
root@ubuntu0:~# ls harbor-offline-installer-v2.7.4.tgz 
harbor-offline-installer-v2.7.4.tgz
root@ubuntu0:~# tar xf harbor-offline-installer-v2.7.4.tgz -C /oldboyedu/softwares/

3.创建工作目录
mkdir -pv /oldboyedu/softwares/harbor/certs/{ca,harbor-server,docker-client}

4.进入到harbor证书存放目录
root@ubuntu0:~# cd /oldboyedu/softwares/harbor/certs/
root@ubuntu0:/oldboyedu/softwares/harbor/certs# ll
total 20
drwxr-xr-x 5 root root 4096 Mar 16 20:34 ./
drwxr-xr-x 3 root root 4096 Mar 16 20:35 ../
drwxr-xr-x 2 root root 4096 Mar 16 20:34 ca/
drwxr-xr-x 2 root root 4096 Mar 16 20:34 docker-client/
drwxr-xr-x 2 root root 4096 Mar 16 20:34 harbor-server/

5.生成自建CA证书
	5.1 创建CA的私钥
root@ubuntu0:/oldboyedu/softwares/harbor/certs# openssl genrsa -out ca/ca.key 4096
   5.2 基于自建的CA私钥创建CA证书(注意，证书签发的域名范围)
root@ubuntu0:/oldboyedu/softwares/harbor/certs# openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=oldboyedu.com" \
 -key ca/ca.key \
 -out ca/ca.crt
      #CN=oldboyedu.com是指定的域名
  5.3 查看自建证书信息
   root@ubuntu0:/oldboyedu/softwares/harbor/certs# openssl  x509 -in ca/ca.crt -noout -text

6.配置harbor证书
  6.1 生成harbor服务器的私钥
  root@ubuntu0:/oldboyedu/softwares/harbor/certs# openssl genrsa -out harbor-server/harbor.oldboyedu.com.key 4096
  6.2 harbor服务器基于私钥签发证书认证请求（csr文件），让自建CA认证
  root@ubuntu0:/oldboyedu/softwares/harbor/certs# openssl req -sha512 -new \
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=harbor.oldboyedu.com" \
    -key harbor-server/harbor.oldboyedu.com.key \
    -out harbor-server/harbor.oldboyedu.com.csr
  6.3 生成 x509 v3 的扩展文件用于认证
  root@ubuntu0:/oldboyedu/softwares/harbor/certs# cat > harbor-server/v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=harbor.oldboyedu.com
EOF

6.4 基于 x509 v3 的扩展文件认证签发harbor server证书
root@ubuntu0:/oldboyedu/softwares/harbor/certs# openssl x509 -req -sha512 -days 3650 \
    -extfile harbor-server/v3.ext \
    -CA ca/ca.crt -CAkey ca/ca.key -CAcreateserial \
    -in harbor-server/harbor.oldboyedu.com.csr \
    -out harbor-server/harbor.oldboyedu.com.crt
Certificate request self-signature ok
subject=C = CN, ST = Beijing, L = Beijing, O = example, OU = Personal, CN = harbor.oldboyedu.com

6.5 修改harbor的配置文件使用自建证书
root@ubuntu0:/oldboyedu/softwares/harbor/certs# pwd
/oldboyedu/softwares/harbor/certs
root@ubuntu0:/oldboyedu/softwares/harbor/certs# cp ../harbor.yml{.tmpl,}
root@ubuntu0:/oldboyedu/softwares/harbor/certs# vim ../harbor.yml

root@ubuntu0:/oldboyedu/softwares/harbor/certs# vim ../harbor.yml
...
hostname: harbor.oldboyedu.com
...
https:
  ...
  certificate: /oldboyedu/softwares/harbor/certs/harbor-server/harbor.oldboyedu.com.crt
  private_key: /oldboyedu/softwares/harbor/certs/harbor-server/harbor.oldboyedu.com.key
...
harbor_admin_password: 1
...
data_volume: /oldboyedu/data/harbor

6.6  安装harbor
root@ubuntu0:/oldboyedu/softwares/harbor# pwd
/oldboyedu/softwares/harbor
root@ubuntu0:/oldboyedu/softwares/harbor# ls
LICENSE  certs  common.sh  harbor.v2.7.4.tar.gz  harbor.yml  harbor.yml.tmpl  install.sh  prepare
root@ubuntu0:/oldboyedu/softwares/harbor# ./install.sh --with-chartmuseum 


7.window访问harbor仓库测试
192.168.23.99 harbor.oldboyedu.com  # windows的hosts文件添加解析记录。
https://harbor.oldboyedu.com/harbor/projects

8.配置docker客户端证书
  8.1 生成docker客户端证书
  root@ubuntu0:/oldboyedu/softwares/harbor/certs# openssl x509 -inform PEM -in harbor-server/harbor.oldboyedu.com.crt -out docker-client/harbor.oldboyedu.com.cert

  8.2 拷贝docker client证书文件
root@ubuntu0:/oldboyedu/softwares/harbor/certs#  cp harbor-server/harbor.oldboyedu.com.key docker-client/
root@ubuntu0:/oldboyedu/softwares/harbor/certs# cp ca/ca.crt docker-client/

9.docker客户端使用证书
  9.1 docker客户端所有节点创建自建证书的目录结构
  root@ubuntu2:~#  mkdir -pv /etc/docker/certs.d/harbor.oldboyedu.com/
mkdir: created directory '/etc/docker/certs.d'
mkdir: created directory '/etc/docker/certs.d/harbor.oldboyedu.com/'
9.2 将客户端证书文件进行拷贝
root@ubuntu1:~# scp 192.168.23.99:/oldboyedu/softwares/harbor/certs/docker-client/* /etc/docker/certs.d/harbor.oldboyedu.com/
The authenticity of host '192.168.23.99 (192.168.23.99)' can't be established.
ED25519 key fingerprint is SHA256:Ec9l1SYz2C9036strXyYQcx+PEXj/7PXmBNz7WZwf24.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:1: [hashed name]
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.23.99' (ED25519) to the list of known hosts.
root@192.168.23.99's password: 
ca.crt                                                                                                           100% 2049     1.6MB/s   00:00    
harbor.oldboyedu.com.cert                                                                                        100% 2147     2.8MB/s   00:00    
harbor.oldboyedu.com.key                                                                                         100% 3272     5.5MB/s   00:00  

9.3 docker客户端验证
  先做hosts解析
root@ubuntu1:~# grep harbor.oldboyedu.com /etc/hosts
192.168.23.99 ubuntu0 harbor.oldboyedu.com
成功即可
root@ubuntu1:~# docker login -u admin -p 1 harbor.oldboyedu.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded


参考链接:
	https://goharbor.io/docs/1.10/install-config/configure-https/#generate-a-certificate-authority-certificate
	https://www.cnblogs.com/yinzhengjie/p/17153673.html

在这里插入图片描述

 
root@ubuntu1:~# docker tag mysqlsb:v1 harbor.oldboyedu.com/linux94/mysql
root@ubuntu1:~# docker push harbor.oldboyedu.com/linux94/mysql
Using default tag: latest
The push refers to repository [harbor.oldboyedu.com/linux94/mysql]
ccde1388b67e: Pushed 
f183d27a3d65: Pushed 

K8S使用harbor的镜像仓库
cat 09-pods-xiuxian-imagePullPolicy.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: oldboyedu-xiuxian-imagepullpolicy
spec:
  nodeName: worker233
  containers:
    # 表示给容器分配一个标准输入，因为alpine镜像的启动命令是sh，此处相当于docker run -i ...
  - stdin: true
    name: c1
    image: harbor.oldboyedu.com/oldboyedu-linux/alpine:latest
    # 指的镜像的拉取策略
    # imagePullPolicy: Never
    # imagePullPolicy: Always
    imagePullPolicy: IfNotPresent