kubernetes的二进制安装

最新推荐文章于 2024-09-05 19:51:15 发布

IA有梦

最新推荐文章于 2024-09-05 19:51:15 发布

阅读量731

点赞数

分类专栏： Kubernetes Linux 文章标签： kubernetes docker etcd

本文链接：https://blog.csdn.net/baidu_38366464/article/details/109989090

版权

Kubernetes 同时被 2 个专栏收录

6 篇文章 0 订阅

订阅专栏

Linux

2 篇文章 0 订阅

订阅专栏

文章目录

服务器环境

服务器IP	节点名称	组件
192.168.10.42	k8s-master	etcd、kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy、CNI插件
192.168.10.43	k8s-node01	etcd、kubelet、kube-proxy、CNI插件
192.168.10.44	k8s-node02	etcd、kubelet、kube-proxy、CNI插件

`二进制安装kubernetes集群准备工作`

系统初始化

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

关闭swap

swapoff -a     # 临时关闭
vi /etc/fstab   # 永久关闭

添加hosts

vi /etc/hosts

192.168.10.42  k8s-master
192.168.10.43  k8s-node01
192.168.10.44  k8s-node02

同步系统时间

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
yum install -y ntpdate ntp
vim /etc/ntp.conf

server ntp1.aliyun.com

开启内核参数

cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl -p

开启ipvs

yum -y install ipvsadm  ipset

# 临时生效
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4

# 永久生效
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

生成 ETCD, K8S 的TLS证书

使用cfssl生成SSL证书

创建etcd.conf

#[Member]
ETCD_NAME=etcd-1
ETCD_DATA_DIR=/var/lib/etcd/default.etcd
ETCD_LISTEN_PEER_URLS=https://192.168.10.42:2380
ETCD_LISTEN_CLIENT_URLS=https://192.168.10.42:2379

#[Clustering]
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.10.42:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.10.42:2380
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.10.42:2380,etcd-2=https://192.168.10.43:2380,etcd-3=https://192.168.10.44:2380"
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_INITIAL_CLUSTER_STATE=new

# [security]
ETCD_CERT_FILE="/opt/kubernetes/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/etcd/ssl/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/kubernetes/etcd/ssl/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/opt/kubernetes/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/etcd/ssl/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/kubernetes/etcd/ssl/ca.pem"
ETCD_PEER_AUTO_TLS="true"

创建etcd.service启动文件，将文件移动至 /usr/lib/systemd/system 文件夹下

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/kubernetes/etcd/conf/etcd.conf
WorkingDirectory=/opt/kubernetes/etcd
PermissionsStartOnly=true
ExecStart=/opt/kubernetes/etcd/bin/etcd
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动etcd

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

检查etcd集群健康状态

etcdctl --ca-file=ca.pem --cert-file=etcd.pem --key-file=etcd-key.pem --endpoints=https://192.168.10.42:2379,https://192.168.10.43:2379,https://192.168.10.44:2379 cluster-health

`安装Master组件kube-apiserver、kube-controller-manager、kube-scheduler`

仿照ETCD生成SSL证书

安装kube-apiserver

创建kube-apiserver.service

[Unit]
Description=Kubernetes API Server

[Service]
EnvironmentFile=/opt/kubernetes/kube-apiserver/conf/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

创建kube-apiserver.conf

KUBE_APISERVER_OPTS="--etcd-servers=https://192.168.10.42:2379,https://192.168.10.43:2379,https://192.168.10.44:2379 --bind-address=192.168.10.42 --secure-port=6443 --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --allow-privileged=true --advertise-address=192.168.10.42 --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth=true --token-auth-file=/opt/kubernetes/kube-apiserver/conf/token.csv --service-node-port-range=30000-32767 --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem --tls-cert-file=/opt/kubernetes/ssl/server.pem --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/kubernetes/ssl/etcd/ca.pem --etcd-certfile=/opt/kubernetes/ssl/etcd/etcd.pem --etcd-keyfile=/opt/kubernetes/ssl/etcd/etcd-key.pem --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

为kubelet TLS Bootstrapping授权

token可自行替换生成

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

# cat token.csv
cf4df8c2ebdcbf6246057eb96b50c98a,kubelet-bootstrap,10001,"system:node-bootstrapper"

格式: token,用户,UID,用户组

给kubelet-bootstrap授权

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

安装kube-controller-manager

创建kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager Server

[Service]
EnvironmentFile=/opt/kubernetes/kube-controller-manager/conf/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

创建kube-controller-manager.conf

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect=true --master=127.0.0.1:8080 --address=127.0.0.1 --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16 --service-cluster-ip-range=10.0.0.0/24 --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem --root-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem --experimental-cluster-signing-duration=87600h0m0s"

安装kube-scheduler

创建kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler Server

[Service]
EnvironmentFile=/opt/kubernetes/kube-scheduler/conf/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

创建kube-scheduler.conf

KUBE_SCHEDULER_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect --master=127.0.0.1:8080 --address=127.0.0.1"

启动并检查kubernetes Master节点运行状态

systemctl start kube-apiserver
systemctl start kube-scheduler
systemctl start kube-controller-manager

# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}

`安装Node节点应用`

node节点配置文件根据后缀名不同，作用不同

conf : 基本配置文件
kubeconfig : 连接apiserver的配置文件
yml : 常规主要配置文件(用于动态配置更新)

安装kubelet

创建kubelet.conf

KUBELET_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --hostname-override=k8s-02 --network-plugin=cni --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig --bootstrap-kubeconfig=/opt/kubernetes/conf/bootstrap.kubeconfig --config=/opt/kubernetes/conf/kubelet-config.yml --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/k8sxio/pause-amd64:3.2"

创建bootstrap.kubeconfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://192.168.10.42:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet-bootstrap
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: cf4df8c2ebdcbf6246057eb96b50c98a

创建kubelet-config.yml

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110

创建kubelet.service

[Unit]
Description=Kubernetes Kubelet
After=docker.service
Before=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/conf/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

安装kube-proxy

创建kube-proxy.conf

KUBE_PROXY_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --config=/opt/kubernetes/conf/kube-proxy-config.yml"

创建kube-proxy.kubeconfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://192.168.10.42:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kube-proxy
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
  user:
    client-certificate: /opt/kubernetes/ssl/kube-proxy.pem
    client-key: /opt/kubernetes/ssl/kube-proxy-key.pem

创建kube-proxy-config.yml

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
address: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/conf/kube-proxy.kubeconfig
hostnameOverride: k8s-02
clusterCIDR: 10.0.0.0/24
mode: ipvs
ipvs:
  scheduler: "rr"
iptables:
  masqueradeAll: true

创建kube-proxy.service

[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/conf/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

`将node节点加入master`

启动kubelet

systemctl start kubelet

master节点查看csr

kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-FcpbcWFOuED8FlAIBPUJcxS_6XctWdBy4ljBlboXwSU   52s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

kubectl certificate approve node-csr-FcpbcWFOuED8FlAIBPUJcxS_6XctWdBy4ljBlboXwSU

`安装CNI插件`

下载cni插件

wget https://github.com/containernetworking/plugins/releases/download/v0.8.7/cni-plugins-linux-amd64-v0.8.7.tgz

创建cni工作目录及配置目录

mkdir -p /opt/cni/bin
mkdir -p /opt/cni/net.d

`安装kube-flannel`

下载kube-flannel.yml

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

修改kube-flannel.yml中net-conf字段，Network网段与kube-controller-manager中配置的cluster-cidr网段一致

net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

`对kubernetes用户进行授权(示例)`

创建rbac授权文件
apiserver-to-kubelet-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/status
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserer
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes

kubectl工具远程连接集群

生成管理员证书

vim admin-csr.json

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "JiangSu",
      "ST": "JiangSu",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

创建kubeconfig文件

USERNAME="admin"
APISERVER="https://192.168.10.42:6443"
CA_FILE="/opt/kubernetes/ssl/ca.pem"
CA_KEY_FILE="/opt/kubernetes/ssl/ca-key.pem"

kubectl config  set-cluster ${USERNAME} --certificate-authority=${CA_FILE} --embed-certs=true --server=${APISERVER} --kubeconfig=${USERNAME}.conf
kubectl config set-credentials ${USERNAME} --client-certificate=${USERNAME}.crt --client-key=${USERNAME}.key --embed-certs=true --kubeconfig=${USERNAME}.conf
kubectl config set-context ${USERNAME}-context@${USERNAME} --cluster=${USERNAME} --user=${USERNAME} --kubeconfig=${USERNAME}.conf
kubectl config use-context ${USERNAME}-context@${USERNAME} --kubeconfig=${USERNAME}.conf
kubectl create clusterrolebinding ${USERNAME} --clusterrole=cluster-admin --user=${USERNAME}