基础配置
Host | Liunx version | Specs | Roles |
---|---|---|---|
82.61.xxx.xxx | CentOS7.6 | 8 GB/2 CPUs/100 GB/8M | master |
182.61.xxx.xxx | CentOS7.6 | 8 GB/2 CPUs/100 GB/8M | node |
106.12.xxx.xxx | CentOS7.6 | 8 GB/2 CPUs/100 GB/8M | node |
调整字符颜色(防止眼瞎)
vim ~/.bashrc
PS1='\[\e[32;40m\]\u@\h \W ➤ \e[m '
source ~/.bashrc
配置免密登陆 (配置 WireGuard 了的可忽略)
-
生成密钥对(注意文件名需区分)
ssh-keygen -t rsa
-
将公钥发送至其它服务器(追加)
-
82.61.xxx.xxx
ssh root@182.61.xxx.xxx ‘cat >> .ssh/authorized_keys’ < ~/.ssh/id_rsa.pub
ssh root@106.12.xxx.xxx ‘cat >> .ssh/authorized_keys’ < ~/.ssh/id_rsa.pub -
182.61.xxx.xxx
ssh root@82.61.xxx.xxx ‘cat >> .ssh/authorized_keys’ < ~/.ssh/id_rsa.pub
ssh root@106.12.xxx.xxx ‘cat >> .ssh/authorized_keys’ < ~/.ssh/id_rsa.pub -
106.12.xxx.xxx
ssh root@82.61.xxx.xxx ‘cat >> .ssh/authorized_keys’ < ~/.ssh/id_rsa.pub
ssh root@182.61.xxx.xxx ‘cat >> .ssh/authorized_keys’ < ~/.ssh/id_rsa.pub
-
额外配置(WireGuard)
因是三个账号购买的轻量云服务器,所以内网是ping不通的,这边最简单的方式就是配置一下iptables
配置 指定内网转发至 公网
iptables -t nat -A OUTPUT -d 192.168.0.4 -j DNAT --to-destination 182.61.xxx.xxx
可惜,这边三台机器内网IP都是相同的(轻量云服务器无法修改内网)。。 方案B:WireGuard
- 升级软件包
yum update -y
- 开启节点转发
echo “net.ipv4.ip_forward = 1” >> /etc/sysctl.conf
echo “net.ipv4.conf.all.proxy_arp = 1” >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf
- 添加 iptables 规则,允许本机的 NAT 转换
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i wg0 -o wg0 -m conntrack --ctstate NEW -j ACCEPT
iptables -t nat -A POSTROUTING -s 192.168.1.1/24 -o eth0 -j MASQUERADE //此IP为自定义内网, eth0为本地网卡
- 升级内核(安装WireGuard需要)
wget http://ftp.sjtu.edu.cn/sites/elrepo.org/linux/kernel/el7/x86_64/RPMS/kernel-ml-5.15.11-1.el7.elrepo.x86_64.rpm
rpm -ivh kernel-ml-5.15.11-1.el7.elrepo.x86_64.rpm
- 安装WireGuard
yum install epel-release https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
yum install yum-plugin-elrepo kmod-wireguard wireguard-tools -y
- 生成密钥
wg genkey | tee privatekey | wg pubkey > publickey
cat privatekey publickey
- 配置密钥
vim /etc/wireguard/wg0.conf
[Interface]
PrivateKey = 本机私钥
Address = 本机自定义的内网ip
ListenPort = 5418
[Peer]
PublicKey = Node公钥
EndPoint = Node公网IP:5418
AllowedIPs = Node自定义的内网IP/32
- x1
[Interface]
PrivateKey = YJckvUVQZHRQJ95C6O6Akab3fbNkeItd25rhB4eOYmg=
Address = 192.168.1.1
ListenPort = 5418
[Peer]
PublicKey = YqgbNGSNtiRWMKZbNgkEK6T2qTHPr+Jlb1gnZJEvty4=
EndPoint = 182.61.xxx.xxx:5418
AllowedIPs = 192.168.1.2/32
[Peer]
PublicKey = 5wDeWexIorW7a+/QfX+wm6H6eKnN4XKj47zWnLi0VV0=
EndPoint = 106.12.xxx.xxx:5418
AllowedIPs = 192.168.1.3/32
- x2
[Interface]
PrivateKey = aAy//YqWfx3X1ZB+T0CKVRi1Z25eV7WyHdJAwmpql1Y=
Address = 192.168.1.2
ListenPort = 5418
[Peer]
PublicKey = opIJLt4XSZjN0+yoTcFSepyl6cPJ9fZpQTjeVAzsLEg=
EndPoint = 82.61.xxx.xxx:5418
AllowedIPs = 192.168.1.1/32
[Peer]
PublicKey = 5wDeWexIorW7a+/QfX+wm6H6eKnN4XKj47zWnLi0VV0=
EndPoint = 106.12.xxx.xxx:5418
AllowedIPs = 192.168.1.3/32
- x3
[Interface]
PrivateKey = KIvNn0moJfmtppg8fPiXooGSu/rOdEgAesIdMixllmU=
Address = 192.168.1.3
ListenPort = 5418
[Peer]
PublicKey = opIJLt4XSZjN0+yoTcFSepyl6cPJ9fZpQTjeVAzsLEg=
EndPoint = 82.61.xxx.xxx:5418
AllowedIPs = 192.168.1.1/32
[Peer]
PublicKey = YqgbNGSNtiRWMKZbNgkEK6T2qTHPr+Jlb1gnZJEvty4=
EndPoint = 182.61.xxx.xxx:5418
AllowedIPs = 192.168.1.2/32
- 测试
ping 192.168.1.1 -c 3
ping 192.168.1.2 -c 3
ping 192.168.1.3 -c 3
1、Docker
安装
curl -sSL https://get.daocloud.io/docker | sh //默认最新版本
systemctl start docker
systemctl status docker
设置镜像源
自己阿里云加速地址
vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://gs41u9fe.mirror.aliyuncs.com"]
}
systemctl daemon-reload && systemctl restart docker
其它
docker rmi -f `docker images -q` //强制删除所有镜像
docker rm -f `docker ps -a -q` //强制删除所有容器
docker logs -f [CONTAINER ID or NAME] //实时查看容器镜像日志
docker images | grep [NAME] //搜索指定镜像
部署服务
- 官方镜像网址 https://hub.docker.com/search?type=image
docker search zookeeper //搜索镜像
docker run --privileged=true -d --name zookeeper -p 2181:2181 zookeeper:3.4.12 //启动容器,本地没有该镜像时会去公共镜像仓库拉取
2、Rancher (暂时忽略)
安装
docker run -d --restart=unless-stopped \
-p 80:80 -p 443:443 \
--privileged \
--restart=unless-stopped \
--name rancher \
rancher/rancher:v2.4.8 v2.5.7 v2.6.0
run 启动容器
-d 后台运行
-p 将容器的80端口映射到主机的80端口
–name 容器命名
–privileged 赋于容器真正的root权限
–network=host 使用宿主机网络(因为轻量云服务器,此直接使用公网地址)
–restart=unless-stopped 容器服务崩溃时docker将自动重启
其它
- 清理残留文件(N次重装的痛苦)
docker rm -f `docker ps -a -q`
docker rmi -f `docker images -q`
docker volume rm $(docker volume ls -q)
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
rm -rf /etc/ceph \
/etc/etcd \
/etc/kubernetes \
/etc/cni \
/opt/cni \
/run/secrets/kubernetes.io \
/run/calico \
/run/flannel \
/var/lib/calico \
/var/lib/cni \
/var/lib/kubelet \
/var/lib/etcd \
/var/lib/docker \
/var/log/containers \
/var/log/pods \
/var/run/calico
systemctl restart docker
3、K3S
Master(x1)
- 安装
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - \
--node-external-ip 82.61.xxx.xxx \
--advertise-address 82.61.xxx.xxx \
--node-ip 192.168.1.1 \
--flannel-iface wg0 \
--docker
node-external-ip //为节点设置外部IP
advertise-address //用于设置kubectl工具以及子节点进行通讯使用的地址
node-ip //指定内网IP
flannel-iface //指定节点通讯的网卡
–docker //使用docker作为容器环境
- 查看启动状态
systemctl status k3s
Node(x2、x3)
- 安装
- x2
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn \
K3S_URL=https://192.168.1.1:6443 \
K3S_TOKEN=K104c543f5acff4735a1fa9a63931cbf5ddb750be62cd0621b546b1c91ee66a1bcb::server:7c1e4005022e4576d6d4c55495bb6933 \
sh -s - \
--node-external-ip 182.61.xxx.xxx \
--node-ip 192.168.1.2 \
--flannel-iface wg0 \
--docker
- x3
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn \
K3S_URL=https://192.168.1.1:6443 \
K3S_TOKEN=K104c543f5acff4735a1fa9a63931cbf5ddb750be62cd0621b546b1c91ee66a1bcb::server:7c1e4005022e4576d6d4c55495bb6933 \
sh -s - \
--node-external-ip 106.12.xxx.xxx \
--node-ip 192.168.1.3 \
--flannel-iface wg0 \
--docker
K3S_URL //master通信地址
K3S_TOKEN //master节点token #cat /var/lib/rancher/k3s/server/node-token
- 查看启动状态
systemctl status k3s-agent
卸载
/usr/local/bin/k3s-uninstall.sh //卸载master节点
/usr/local/bin/k3s-agent-uninstall.sh //卸载node节点
测试(部署Nginx)
编写nginx的deployment文件
vim nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata: # 元信息
name: nginx-deployment
labels:
app: nginx # 标签
namespace: default # 放到默认的命名空间
spec: # 详细参数配置
replicas: 1 # pod的副本数
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers: # 容器信息
- name: nginx
image: nginx:latest # 指定拉取的镜像
ports:
- containerPort: 80 # 指定容器的端口
启动nginx
kubectl apply -f nginx.yaml
查看启动状态
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-585449566-nbfhs 1/1 Running 0 19m 10.42.2.6 x-2 <none> <none>
测试内网访问
curl 10.42.2.6
[root@x-1 ~]# curl 10.42.2.6
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
创建nginx的service提供外网访问
vim nginx-service.ymal
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx-deployment
namespace: default
spec:
ports:
- port: 9000 # Service绑定的端口
name: nginx-service80
protocol: TCP #示Service转发请求到容器的协议是TCP
targetPort: 80 #表示Service转发外部请求到容器的目标端口80
nodePort: 30080 #表示Service对外开放的节点端口
selector:
app: nginx # 转发到指定标签的pods
type: NodePort #节点端口转发类型
启动nginx-service
kubectl apply -f nginx-servcie.yaml
查看启动状态
kubectl get service -o wide
[root@x-1 ~]# kubectl get service -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 12h <none>
nginx-deployment NodePort 10.43.146.168 <none> 9000:30080/TCP 26m app=nginx
测试公网访问
curl 82.61.xxx.xxx:30080
[root@x-1 ~]# curl 82.61.xxx.xxx:30080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
- 删除deployment
kubectl delete deployment [deployment-name] -n [namespace]
- 删除service
kubectl delete services [service-name] -n [namespace]
导入rancher(暂时忽略)
- 查看全部节点
kubectl get nodes -o wide
[root@x-1 ~]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
x-1 Ready control-plane,master 11m v1.22.5+k3s1 192.168.1.1 82.61.xxx.xxx CentOS Linux 7 (Core) 3.10.0-1160.49.1.el7.x86_64 docker://20.10.12
x-3 Ready <none> 18s v1.22.5+k3s1 192.168.1.3 106.12.xxx.xxx CentOS Linux 7 (Core) 3.10.0-1160.49.1.el7.x86_64 docker://20.10.12
x-2 NotReady <none> 6s v1.22.5+k3s1 192.168.1.2 182.61.xxx.xxx CentOS Linux 7 (Core) 3.10.0-1160.49.1.el7.x86_64 docker://20.10.12
- 导入k3s
- 在rancher页面新建集群,选择导入
- 无ssl时使用第三条命令
curl --insecure -sfL https://82.61.xxx.xxx/v3/import/fkn66kg9c8kw8m24d4jr545tzhx5n6cjsc5r7xsrn57s75j25rx262.yaml | kubectl apply -f -
报错:error: no objects passed to apply
本来想直接下载该文件运行,但是因ssl原因访问不通,直接在网页段打开该文件,复制到 新建rancher.yaml文件中
kubectl apply -f rancher.yaml
Pending了半天。。。
- 查看全部空间
kubectl get ns
[root@x-1 ~]# kubectl get ns
NAME STATUS AGE
default Active 71m
kube-system Active 71m
kube-public Active 71m
kube-node-lease Active 71m
cattle-system Active 32m
- 查看rancher空间cattle-system
kubectl get pod -n cattle-system
[root@x-1 ~]# kubectl get pod -n cattle-system
NAME READY STATUS RESTARTS AGE
cattle-node-agent-h5rmb 0/1 CrashLoopBackOff 11 (76s ago) 33m
cattle-cluster-agent-5c8bdc8b77-gtjfk 0/1 CrashLoopBackOff 11 (51s ago) 33m
cattle-node-agent-b8vzm 0/1 CrashLoopBackOff 11 (49s ago) 33m
cattle-node-agent-q9d5j 0/1 CrashLoopBackOff 11 (45s ago) 33m
好家伙,状态全是CrashLoopBackOff
- 查看pod日志
kubectl logs --tail=20 cattle-node-agent-q9d5j -n cattle-system
INFO: Environment: CATTLE_ADDRESS=192.168.0.4 CATTLE_AGENT_CONNECT=true CATTLE_CA_CHECKSUM=2a25bed5d2114993b63178b5381a858cf920a3807844d772fc90911ad51e2936 CATTLE_CLUSTER=false CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=x-2 CATTLE_SERVER=https://82.61.xxx.xxx
INFO: Using resolv.conf: nameserver 192.168.0.2 nameserver 192.168.0.3 search gz.baidu.internal options rotate timeout:1
ERROR: https://82.61.xxx.xxx/ping is not accessible (The requested URL returned error: 404)
kubectl logs --tail=20 cattle-cluster-agent-5587b66dd7-f4d6v -n cattle-system
INFO: Environment: CATTLE_ADDRESS=10.42.2.4 CATTLE_CA_CHECKSUM=2a25bed5d2114993b63178b5381a858cf920a3807844d772fc90911ad51e2936 CATTLE_CLUSTER=true CATTLE_FEATURES= CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-5c8bdc8b77-2vprh CATTLE_SERVER=https://82.61.xxx.xxx
INFO: Using resolv.conf: nameserver 10.43.0.10 search cattle-system.svc.cluster.local svc.cluster.local cluster.local gz.baidu.internal options ndots:5
ERROR: https://82.61.xxx.xxx/ping is not accessible (The requested URL returned error: 404)
初步怀疑是rancher跟k3s网络不通
- 删除空间
kubectl delete ns cattle-system
sh ./create_self-signed-cert.sh --ssl-trusted-ip=82.61.xxx.xxx,106.12.xxx.xxx,182.61.xxx.xxx
helm install rancher rancher-stable/rancher --versioin=v2.4.8 -n cattle-system --set ingress.tls.source=secret --set privateCA=true
helm install rancher rancher-2.4.8/rancher --namespace cattle-system --set ingress.tls.source=secret --set privateCA=true
helm install rancher rancher-stable/rancher
–namespace cattle-system
–set ingress.tls.source=secret
–set privateCA=true
helm template rancher ./rancher-2.4.8.tgz
–namespace cattle-system --output-dir .
–set privateCA=true
–set additionalTrustedCAs=true
–set ingress.tls.source=secret
–set hostname=www.rancher.local
–set useBundledSystemChart=true
最终决定先不用rancher了。。
kubectl也能用,后续考虑再买三台真正的云服务器再进行尝试