kubernetes简介
1、kubernetes核心概念
关于k8s的概念,这里有一篇简明扼要的文章:http://www.dockone.io/article/932,此处不再赘述。
2、kubernetes安装
2.1、环境介绍
操作系统:CentOS-7-x86_64-Minimal-1611,节点信息如下:
节点 | hostname | IP | 部署 |
---|---|---|---|
K8S Master | node-0 | 10.211.55.4 | etcd、flannel、k8s |
K8S Slave-1 | node-1 | 10.211.55.19 | flannel、k8s |
K8S Slave-2 | node-2 | 10.211.55.20 | flannel、k8s |
Docker registry | node-3 | 10.211.55.15 | docker |
Docker registry的部署与使用不在本文范围之内
2.2、软件版本
- ETCD:3.1.9
- Flannel:0.7.1
- Kubernetes:1.5.2
2.3、安装步骤
2.3.1、安装ETCD
配置防火墙,开放etcd的默认的2379端口
[chenlei@node-0 ~]$ sudo firewall-cmd --zone=public --add-port=2379/tcp --permanent [chenlei@node-0 ~]$ sudo firewall-cmd --reload [chenlei@node-0 ~]$ sudo firewall-cmd --list-all
yum安装etcd
[chenlei@node-0 ~]$ sudo yum install -y etcd
配置etcd,修改/etc/etcd/etcd.conf中的
ETCD_LISTEN_CLIENT_URLS
和ETCD_ADVERTISE_CLIENT_URLS
[chenlei@node-0 ~]$ sudo vi /etc/etcd/etcd.conf # [member] ETCD_NAME=default ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_WAL_DIR="" #ETCD_SNAPSHOT_COUNT="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD_LISTEN_PEER_URLS="http://localhost:2380" #ETCD_LISTEN_CLIENT_URLS="http://localhost:2379" ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" # #[cluster] #ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380" # if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." #ETCD_INITIAL_CLUSTER="default=http://localhost:2380" #ETCD_INITIAL_CLUSTER_STATE="new" #ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379" ETCD_ADVERTISE_CLIENT_URLS="http://node-0:2379" #ETCD_DISCOVERY="" #ETCD_DISCOVERY_SRV="" #ETCD_DISCOVERY_FALLBACK="proxy" #ETCD_DISCOVERY_PROXY="" #ETCD_STRICT_RECONFIG_CHECK="false" #ETCD_AUTO_COMPACTION_RETENTION="0" # #[proxy] #ETCD_PROXY="off" #ETCD_PROXY_FAILURE_WAIT="5000" #ETCD_PROXY_REFRESH_INTERVAL="30000" #ETCD_PROXY_DIAL_TIMEOUT="1000" #ETCD_PROXY_WRITE_TIMEOUT="5000" #ETCD_PROXY_READ_TIMEOUT="0" # #[security] #ETCD_CERT_FILE="" #ETCD_KEY_FILE="" #ETCD_CLIENT_CERT_AUTH="false" #ETCD_TRUSTED_CA_FILE="" #ETCD_AUTO_TLS="false" #ETCD_PEER_CERT_FILE="" #ETCD_PEER_KEY_FILE="" #ETCD_PEER_CLIENT_CERT_AUTH="false" #ETCD_PEER_TRUSTED_CA_FILE="" #ETCD_PEER_AUTO_TLS="false" # #[logging] #ETCD_DEBUG="false" # examples for -log-package-levels etcdserver=WARNING,security=DEBUG #ETCD_LOG_PACKAGE_LEVELS="" # #[profiling] #ETCD_ENABLE_PPROF="false" #ETCD_METRICS="basic" ~
启动etcd服务
[chenlei@node-0 ~]$ systemctl start etcd [chenlei@node-0 ~]$ systemctl status etcd
检查etcd是否正常
[chenlei@node-0 ~]$ etcdctl --endpoints 'http://node-0:2379' cluster-health member 8e9e05c52164694d is healthy: got healthy result from http://node-0:2379 cluster is healthy
2.3.2、安装Flannel
在node-0、node-1、node-2上安装flanneld
[chenlei@node-0 ~]$ sudo yum install flannel -y [chenlei@node-1 ~]$ sudo yum install flannel -y [chenlei@node-2 ~]$ sudo yum install flannel -y
在node-0、node-1、node-2上修改/etc/sysconfig/flanneld,使FLANNEL_ETCD_ENDPOINTS的值指向我们刚才安装的ETCD服务
[chenlei@node-0 ~]$ sudo vi /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs #FLANNEL_ETCD_ENDPOINTS="http://127.0.0.1:2379" FLANNEL_ETCD_ENDPOINTS="http://node-0:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_PREFIX="/atomic.io/network" # Any additional options that you want to pass #FLANNEL_OPTIONS="eth0" ~ ~
[chenlei@node-1 ~]$ sudo vi /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs #FLANNEL_ETCD_ENDPOINTS="http://127.0.0.1:2379" FLANNEL_ETCD_ENDPOINTS="http://node-0:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_PREFIX="/atomic.io/network" # Any additional options that you want to pass #FLANNEL_OPTIONS="eth0" ~ ~
[chenlei@node-2 ~]$ sudo vi /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs #FLANNEL_ETCD_ENDPOINTS="http://127.0.0.1:2379" FLANNEL_ETCD_ENDPOINTS="http://node-0:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_PREFIX="/atomic.io/network" # Any additional options that you want to pass #FLANNEL_OPTIONS="eth0" ~ ~
在etcd中注册flannel的网络配置,注册时的目录前缀为上一步骤编辑的文件/etc/sysconfig/flanneld中,FLANNEL_ETCD_PREFIX的值,默认为”/atomic.io/network”
[chenlei@node-0 ~]$ etcdctl --endpoints 'http://node-0:2379' mk /atomic.io/network/config '{ "Network": "10.2.0.0/16" }'
注意:此处似乎只能使用B类IP,将子网掩码设置为24时flannel无法启动,提示子网注册失败,操作系统网络无法连接,ssh连接断开。
启动node-0、node-1、node-2上flannel服务
[chenlei@node-0 ~]$ systemctl start flanneld [chenlei@node-0 ~]$ systemctl status flanneld [chenlei@node-0 ~]$ ip addr [chenlei@node-1 ~]$ systemctl start flanneld [chenlei@node-1 ~]$ systemctl status flanneld [chenlei@node-1 ~]$ ip addr [chenlei@node-2 ~]$ systemctl start flanneld [chenlei@node-2 ~]$ systemctl status flanneld [chenlei@node-2 ~]$ ip addr
flannel正常启动后,使用ip add命令可以看到出现新的网络设备:flannel0
2.3.3、安装K8S
如果使用官方的repo文件安装,需要在本地创建对应的repo文件,本人在第一次尝试的时候,直接使用默认的源,安装的版本和官方的源版本是一样的。
创建官方repo文件(在node-0、node-1、node-2三个节点上执行)
[chenlei@node-0 ~]$ sudo vi /etc/yum.repos.d/virt7-docker-common-release.repo [virt7-docker-common-release] name=virt7-docker-common-release baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/ gpgcheck=0
安装kubernetes(在node-0、node-1、node-2三个节点上执行,默认并非最新版本,若此前已经安装了docker,建议先卸载掉,k8s会自己安装一个docker)
[chenlei@node-0 ~]$ sudo yum -y install --enablerepo=virt7-docker-common-release kubernetes
配置/etc/kubernetes/apiserver(在node-0上执行,修改内容:KUBE_API_ADDRESS、KUBE_API_PORT、KUBE_ETCD_SERVERS、KUBE_ADMISSION_CONTROL)
[chenlei@node-0 ~]$ sudo vi /etc/kubernetes/apiserver ### # kubernetes system config # # The following values are used to configure the kube-apiserver # # The address on the local server to listen to. #KUBE_API_ADDRESS="--insecure-bind-address=127.0.0.1" KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0" # The port on the local server to listen on. # KUBE_API_PORT="--port=8080" KUBE_API_PORT="--port=8080" # Port minions listen on # KUBELET_PORT="--kubelet-port=10250" # Comma separated list of nodes in the etcd cluster #KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379" KUBE_ETCD_SERVERS="--etcd-servers=http://node-0:2379" # Address range to use for services KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" # default admission control policies #KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota" KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota" # Add your own! KUBE_API_ARGS="" ~ ~
配置/etc/kubernetes/config(在node-0、node-1、node-2三个节点上执行,修改内容:KUBE_MASTER,三个节点全部指定同一个master)
[chenlei@node-0 ~]$ sudo vi /etc/kubernetes/config ### # kubernetes system config # # The following values are used to configure various aspects of all # kubernetes services, including # # kube-apiserver.service # kube-controller-manager.service # kube-scheduler.service # kubelet.service # kube-proxy.service # logging to stderr means we get it in the systemd journal KUBE_LOGTOSTDERR="--logtostderr=true" # journal message level, 0 is debug KUBE_LOG_LEVEL="--v=0" # Should this cluster be allowed to run privileged docker containers KUBE_ALLOW_PRIV="--allow-privileged=false" # How the controller-manager, scheduler, and proxy find the apiserver #KUBE_MASTER="--master=http://127.0.0.1:8080" KUBE_MASTER="--master=http://node-0:8080" ~ ~
防火墙配置,放开node-0(master)上的8080端口(在node-0上执行)
[chenlei@node-0 ~]$ sudo firewall-cmd --zone=public --add-port=8080/tcp --permanent [chenlei@node-0 ~]$ sudo firewall-cmd --zone=public --add-port=2379/tcp --permanent [chenlei@node-0 ~]$ sudo firewall-cmd --reload [chenlei@node-0 ~]$ sudo firewall-cmd --list-all
启动kube-apiserver、kube-controller-manager、kube-scheduler服务(在node-0上执行)
[chenlei@node-0 ~]$ systemctl start kube-apiserver [chenlei@node-0 ~]$ systemctl status kube-apiserver [chenlei@node-0 ~]$ systemctl start kube-controller-manager [chenlei@node-0 ~]$ systemctl status kube-controller-manager [chenlei@node-0 ~]$ systemctl start kube-scheduler [chenlei@node-0 ~]$ systemctl status kube-scheduler
配置/etc/kubernetes/kubelet(在node-1、node-2两个slave节点上执行,修改内容:KUBELET_ADDRESS、KUBELET_HOSTNAME、KUBELET_API_SERVER、KUBELET_POD_INFRA_CONTAINER)
[chenlei@node-1 ~]$ sudo vi /etc/kubernetes/kubelet ### # kubernetes kubelet (minion) config # The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces) #KUBELET_ADDRESS="--address=127.0.0.1" KUBELET_ADDRESS="--address=0.0.0.0" # The port for the info server to serve on # KUBELET_PORT="--port=10250" # You may leave this blank to use the actual hostname #KUBELET_HOSTNAME="--hostname-override=127.0.0.1" KUBELET_HOSTNAME="--hostname-override=node-1" # location of the api-server #KUBELET_API_SERVER="--api-servers=http://127.0.0.1:8080" KUBELET_API_SERVER="--api-servers=http://node-0:8080" # pod infrastructure container #KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest" KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=10.211.55.15:5000/rhel7/pod-infrastructure:latest" # Add your own! KUBELET_ARGS="" ~ ~
[chenlei@node-2 ~]$ sudo vi /etc/kubernetes/kubelet ### # kubernetes kubelet (minion) config # The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces) #KUBELET_ADDRESS="--address=127.0.0.1" KUBELET_ADDRESS="--address=0.0.0.0" # The port for the info server to serve on # KUBELET_PORT="--port=10250" # You may leave this blank to use the actual hostname #KUBELET_HOSTNAME="--hostname-override=127.0.0.1" KUBELET_HOSTNAME="--hostname-override=node-2" # location of the api-server #KUBELET_API_SERVER="--api-servers=http://127.0.0.1:8080" KUBELET_API_SERVER="--api-servers=http://node-0:8080" # pod infrastructure container #KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest" KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=10.211.55.15:5000/rhel7/pod-infrastructure:latest" # Add your own! KUBELET_ARGS=""
注意:
KUBELET_POD_INFRA_CONTAINER
项,由k8s创建的所有容器都会附带一个容器pod-infrastructure,而这个附带的容器的镜像就是由KUBELET_POD_INFRA_CONTAINER
指定的,默认的registry.access.redhat.com/rhel7/pod-infrastructure:latest可能下载慢或者下载不下来,可以想办法下载下来后上传到私库中以便下次使用。k8s如何使用私库在下面的内容中会介绍,这些镜像的网盘地址见文章末尾。启动kubelet、kube-proxy服务(在node-1、node-2两个slave节点上执行)
[chenlei@node-1 ~]$ systemctl start kubelet [chenlei@node-1 ~]$ systemctl status kubelet [chenlei@node-1 ~]$ systemctl start kube-proxy [chenlei@node-1 ~]$ systemctl status kube-proxy
[chenlei@node-2 ~]$ systemctl start kubelet [chenlei@node-2 ~]$ systemctl status kubelet [chenlei@node-2 ~]$ systemctl start kube-proxy [chenlei@node-2 ~]$ systemctl status kube-proxy
检查k8s集群节点状态(在node-0上执行,后续的主要操作都在此master上执行)
[chenlei@node-0 ~]$ kubectl get nodes NAME STATUS AGE VERSION node-1 Ready 21d v1.5.2 node-2 Ready 21d v1.5.2
至此,k8s的集群已经搭建完成,下面的内容介绍应用=部署
3、kubernetes部署应用
3.1、部署pod
编写yaml文件:single-config-file.yaml
apiVersion: v1 kind: Pod metadata: name: pod-containers spec: restartPolicy: Never containers: - name: tomcat image: 10.211.55.15:5000/tomcat:9.0.0.M26 ports: - containerPort: 8080 hostPort: 8080 command: ["/bin/sh"] args: ["-c", "/usr/local/tomcat/bin/startup.sh && tail -f /usr/local/tomcat/logs/catalina.out"] ~ ~
创建/部署pod
[chenlei@node-0 ~]$ kubectl create -f single-config-file.yaml
查看pod部署信息
[chenlei@node-0 ~]$ kubectl get pods [chenlei@node-0 ~]$ kubectl get pod pod-containers -o wide
访问pod中的服务,注意防火墙放开hostPort
浏览器访问:http://node-1:8080(http://node-ip:hostPort)
如果页面打开极慢,上网搜索:Tomcat启动慢。
删除pod
[chenlei@node-0 ~]$ kubectl delete pod pod-containers
3.2、部署RC(ReplicationController)
编写yaml文件(replication-controller-config-file.yaml)
apiVersion: v1 kind: ReplicationController metadata: name: my-replication labels: name: my-replication spec: replicas: 4 selector: name: my-replication-pod template: metadata: labels: name: my-replication-pod spec: containers: - name: my-containers image: 10.211.55.15:5000/tomcat:9.0.0.M26 ports: - containerPort: 8080 command: ["/bin/sh"] args: ["-c", "/usr/local/tomcat/bin/startup.sh && tail -f /usr/local/tomcat/logs/catalina.out"]
创建/部署RC
[chenlei@node-0 ~]$ kubectl create -f replication-controller-config-file.yaml replicationcontroller "my-replication" created
查看RC部署信息
[chenlei@node-0 ~]$ kubectl get rc
查看pods
[chenlei@node-0 ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE my-replication-3n1qs 1/1 Running 0 3m my-replication-cv26z 1/1 Running 0 3m my-replication-g5w5q 1/1 Running 0 3m my-replication-n6zmn 1/1 Running 0 3m
查看RC详细信息
[chenlei@node-0 ~]$ kubectl describe rc my-replication Name: my-replication Namespace: default Selector: name=my-replication-pod Labels: name=my-replication Annotations: <none> Replicas: 4 current / 4 desired Pods Status: 4 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: name=my-replication-pod Containers: my-containers: Image: 10.211.55.15:5000/tomcat:9.0.0.M26 Port: 8080/TCP Command: /bin/sh Args: -c /usr/local/tomcat/bin/startup.sh && tail -f /usr/local/tomcat/logs/catalina.out Environment: <none> Mounts: <none> Volumes: <none> Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 replication-controller Normal SuccessfulCreate Created pod: my-replication-3n1qs 3m 3m 1 replication-controller Normal SuccessfulCreate Created pod: my-replication-n6zmn 3m 3m 1 replication-controller Normal SuccessfulCreate Created pod: my-replication-cv26z 3m 3m 1 replication-controller Normal SuccessfulCreate Created pod: my-replication-g5w5q
调整RC的集群大小
[chenlei@node-0 ~]$ kubectl scale rc my-replication --replicas=3 replicationcontroller "my-replication" scaled [chenlei@node-0 Kubernetes-test]$ kubectl get pods NAME READY STATUS RESTARTS AGE my-replication-3n1qs 1/1 Running 0 55m my-replication-cv26z 1/1 Running 0 55m my-replication-g5w5q 1/1 Terminating 0 55m my-replication-n6zmn 1/1 Running 0 55m
访问RC中的应用
RC中的应用需要通过下方的service访问
删除RC
[chenlei@node-0 ~]$ kubectl delete -f replication-controller-config-file.yaml
3.3、部署SERVICE
在上一部RC运行的基础上创建service,所以需要先“创建/部署RC”(上一步删除了)
[chenlei@node-0 ~]$ kubectl create -f replication-controller-config-file.yaml replicationcontroller "my-replication" created
编写yaml文件(service-config-file.yaml,此处使用NodePort方式暴露服务)
apiVersion: v1 kind: Service metadata: name: my-service labels: name: my-service spec: type: NodePort ports: - port: 80 targetPort: 8080 protocol: TCP nodePort: 30002 selector: name: my-replication-pod ~ ~
创建/部署Service
[chenlei@node-0 ~]$ kubectl create -f service-config-file.yaml service "my-service" created
查看service
[chenlei@node-0 ~]$ kubectl get services [chenlei@node-0 ~]$ kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.254.0.1 <none> 443/TCP 21d my-service 10.254.160.164 <nodes> 80:30002/TCP 11m
查看service的endpoints
[chenlei@node-0 ~]$ kubectl get endpoints NAME ENDPOINTS AGE kubernetes 10.211.55.4:6443 21d my-service 10.2.58.4:8080,10.2.58.5:8080,10.2.9.2:8080 + 1 more... 47s
查看service详细信息
[chenlei@node-0 ~]$ kubectl describe svc my-service Name: my-service Namespace: default Labels: name=my-service Annotations: <none> Selector: name=my-replication-pod Type: NodePort IP: 10.254.110.8 Port: <unset> 80/TCP NodePort: <unset> 30002/TCP Endpoints: 10.2.58.4:8080,10.2.58.5:8080,10.2.9.2:8080 + 1 more... Session Affinity: None Events: <none>
查看node-1或者node-2上是否监听30002端口
[chenlei@node-1 ~]$ ss -ant | grep 30002 LISTEN 0 128 :::30002 :::*
访问service服务,注意防火墙放开30002
浏览器访问:http://node-1:30002(http://node-ip:nodePort)。
如果你的浏览器显示如下:
尝试关掉防火墙,然后在访问试试。这是因为iptables拒绝了所有转发请求,下面的坑里有进一步的说明与处理方法。
正确的页面如下:
删除service
[chenlei@node-0 ~]$ kubectl delete service my-service service "my-service" deleted
删除service并不会删除对应的RC以及RC对于的Pod
4、部署Dashboard
Dashboard是k8s的一个web界面,提供一些直观的操作。
4.1、准备镜像文件
部署Dashboard需要准备kubernetes-dashboard-amd64镜像,因为我的k8s版本号是1.5.X,所以镜像的版本也只能是1.5.x,git上的yaml给的镜像仓库地址是:gcr.io/google_containers/kubernetes-dashboard-amd64,所以完整的地址是gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1,因为一些众所周知的原因,你可能无法下载到这个镜像文件,所以本文最后提供网盘链接,你需要在下载后导入并且上传到你自己的私库中。
4.2、创建yaml文件
kubernetes-dashboard-deployment.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata: # Keep the name in sync with image version and # gce/coreos/kube-manifests/addons/dashboard counterparts name: kubernetes-dashboard-latest namespace: kube-system spec: replicas: 1 template: metadata: labels: k8s-app: kubernetes-dashboard version: latest kubernetes.io/cluster-service: "true" spec: containers: - name: kubernetes-dashboard image: 10.211.55.15:5000/google_containers/kubernetes-dashboard-amd64:v1.5.1 ports: - containerPort: 9090 args: - --apiserver-host=http://10.211.55.4:8080 livenessProbe: httpGet: path: / port: 9090 initialDelaySeconds: 30 timeoutSeconds: 30
kubernetes-dashboard-service.yaml
apiVersion: v1 kind: Service metadata: name: kubernetes-dashboard namespace: kube-system labels: k8s-app: kubernetes-dashboard kubernetes.io/cluster-service: "true" spec: type: NodePort selector: k8s-app: kubernetes-dashboard ports: - port: 80 targetPort: 9090 nodePort: 30001
4.3、创建服务
创建Deployment
[chenlei@node-0 ~]$ kubectl create -f kubernetes-dashboard-deployment.yaml
创建Service
[chenlei@node-0 ~]$ kubectl create -f kubernetes-dashboard-service.yaml
检查node是否件套30001端口
[chenlei@node-1 ~]$ ss -ant | grep 30001 LISTEN 0 128 :::30001 :::*
[chenlei@node-2 ~]$ ss -ant | grep 30001 LISTEN 0 128 :::30001 :::*
4.4、通过nodePort访问服务
- 如果你访问有问题,往上翻!
- dashboard默认的Namespace是default,而我们部署时yaml文件中指定了“namespace: kube-system”,注意切换。
5、kubernetes暴露服务
kubernetes有三种方式对外暴露服务:NodePort Service、LoadBlancer Service、Ingress。
NodePort上面的service已经提及,NodePort会在每一个slave node上打开一个相同的端口,通过nodeIP:NodePort的方式访问服务,服务一旦多起来,NodePort 在每个节点上开启的端口会及其庞大,而且难以维护,不推荐需要部署大量服务的场景使用。
LoadBlancer是使用外部负载均衡器,目前尚未部署,后续在补充!
Ingress间的理解就是使用代理服务,并配置服务转发规则,由代理服务全权受理所有的访问请求,然后根据转发规则将请求转发到集群内部的Service中。这样只需要对外暴露代理服务的端口即可,代理本身在集群内部,可以自由访问集群中的S service。 所有,实际Ingress的实现是包括两部分的规则制定和代理服务(也就是Ingress和Ingress Controller):
5.1、部署Ingress
5.1.1、准备镜像文件
需要的镜像文件如下:
- gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.13
- gcr.io/google_containers/defaultbackend:1.0
defaultbackend的作用是在代理规则匹配失败时提供默认访问(其实就是返回404)。
5.1.2、创建yaml文件
default-backend.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: default-http-backend labels: k8s-app: default-http-backend namespace: kube-system spec: replicas: 1 template: metadata: labels: k8s-app: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend # Any image is permissable as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint #image: gcr.io/google_containers/defaultbackend:1.0 image: 10.211.55.15:5000/google_containers/defaultbackend:1.0 livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi <hr /> apiVersion: v1 kind: Service metadata: name: default-http-backend namespace: kube-system labels: k8s-app: default-http-backend spec: ports: - port: 80 targetPort: 8080 selector: k8s-app: default-http-backend
nginx-ingress-daemonset.yaml
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: nginx-ingress-lb labels: name: nginx-ingress-lb namespace: kube-system spec: template: metadata: labels: name: nginx-ingress-lb annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: hostNetwork: true terminationGracePeriodSeconds: 60 containers: #- image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.13 - image: 10.211.55.15:5000/google_containers/nginx-ingress-controller:0.9.0-beta.13 name: nginx-ingress-lb readinessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP livenessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 timeoutSeconds: 1 ports: - containerPort: 80 hostPort: 80 - containerPort: 443 hostPort: 443 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --apiserver-host=http://node-0:8080
dashboard-ingress.yaml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: dashboard-ingress annotations: ingress.kubernetes.io/force-ssl-redirect: "false" ingress.kubernetes.io/ssl-redirect: "false" # ingress.kubernetes.io/rewrite-target: / namespace: kube-system spec: rules: # - host: chenlei.com - http: paths: - path: / backend: serviceName: kubernetes-dashboard servicePort: 80
注意dashboard-ingress.yaml中空格的数量
5.1.3、创建服务
事先按照上面的内容正确部署好kubernetes-dashboard-service.yaml
部署default-backend
[chenlei@node-0 ~]$ kubectl create -f default-backend.yaml
部署nginx-ingress-daemonset
[chenlei@node-0 ~]$ kubectl create -f nginx-ingress-daemonset.yaml
部署dashboard-ingress
[chenlei@node-0 ~]$ kubectl create -f dashboard-ingress.yml
5.1.4、查看部署信息
查看服务
[chenlei@node-0 ~]$ kubectl -n kube-system get all NAME READY STATUS RESTARTS AGE po/default-http-backend-3495647973-x838r 1/1 Running 2 1d po/kubernetes-dashboard-latest-217549839-flpgp 1/1 Running 3 1d po/nginx-ingress-lb-vrvrv 1/1 Running 1 1d po/nginx-ingress-lb-wnm1s 1/1 Running 1 1d NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/default-http-backend 10.254.155.251 <none> 80/TCP 1d svc/kubernetes-dashboard 10.254.175.156 <nodes> 80:30001/TCP 1d NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deploy/default-http-backend 1 1 1 1 1d deploy/kubernetes-dashboard-latest 1 1 1 1 1d NAME DESIRED CURRENT READY AGE rs/default-http-backend-3495647973 1 1 1 1d rs/kubernetes-dashboard-latest-217549839 1 1 1 1d
查看ingress
[chenlei@node-0 ~]$ kubectl -n kube-system get ing NAME HOSTS ADDRESS PORTS AGE dashboard-ingress * 10.211.55.19,... 80 1d [chenlei@node-0 ~]$ kubectl -n kube-system describe ing dashboard-ingress Name: dashboard-ingress Namespace: kube-system Address: 10.211.55.19,10.211.55.20 Default backend: default-http-backend:80 (10.2.87.3:8080) Rules: Host Path Backends ---- ---- -------- * / kubernetes-dashboard:80 (10.2.87.2:9090) Annotations: force-ssl-redirect: false ssl-redirect: false Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 12m 12m 1 ingress-controller Normal CREATE Ingress kube-system/dashboard-ingress 11m 11m 1 ingress-controller Normal CREATE Ingress kube-system/dashboard-ingress
5.1.5、访问服务
注意防火墙,放开80及443端口
注意查看时选择namespace,yaml中均设置namespace: kube-system
6、kubernetes常用命令
最简单的方式部署应用
[chenlei@node-0 ~]$ kubectl run my-alpine --image=alpine --replicas=6 ping www.baidu.com
这种方式部署的类型是deployment,单纯的删除pod会立即启动一个新的pod,需要删除deployment。
根据yaml文件(single-config-file.yaml)创建pod
[chenlei@node-0 ~]$ kubectl create -f single-config-file.yaml
这种方式可用于创建各类k8s对象,比如:deployment、deployment、service……
查看默认命名空间下的所有pod
[chenlei@node-0 ~]$ kubectl get pods
查看pod所在的节点
[chenlei@node-0 ~]$ kubectl -o wide get pods
查看指定命名空间(kube-system)下的所有pod
[chenlei@node-0 ~]$ kubectl -n kube-system get pods
根据名称(pod-containers)查看指定的pod
[chenlei@node-0 ~]$ kubectl get pod pod-containers
查看指定pod(pod-containers)的详细信息
[chenlei@node-0 ~]$ kubectl describe pod pod-containers
删除指定的pod(pod-containers)
[chenlei@node-0 ~]$ kubectl delete pod pod-containers
根据创建pod的yaml文件进行删除
[chenlei@node-0 ~]$ kubectl delete -f single-config-file.yaml
这种删除方式适用于各类k8s对象,比如:deployment、deployment、service……
删除默认命名空间下的所有pod
[chenlei@node-0 ~]$ kubectl delete pods --all
查看deployment
[chenlei@node-0 ~]$ kubectl get deployment
根据名称(my-alpine)删除deployment
[chenlei@node-0 ~]$ kubectl delete deployment my-alpine
查看events
[chenlei@node-0 ~]$ kubectl get events
删除所有events
[chenlei@node-0 ~]$ kubectl delete events --all
也可根据名称单独删除
查看所有ReplicationController(rc)
[chenlei@node-0 ~]$ kubectl get rc
修改rc的实例数量
[chenlei@node-0 ~]$ kubectl scale rc my-replication --replicas=3
查看k8s版本
[chenlei@node-0 ~]$ kubectl version
查看k8s集群信息
[chenlei@node-0 ~]$ kubectl cluster-info
查看service
[chenlei@node-0 ~]$ kubectl get services
查看endpoint
[chenlei@node-0 ~]$ kubectl get endpoints
上面的命令只是简单的列举了部分,k8s对各类对象的操作命令规律都是相同的,默认情况下(没有用-n参数指定命名空间时)都针对的是默认命名空间:default。要一次查看所有命名空间使用参数
--all-namespaces
,例如:[chenlei@node-0 ~]$ kubectl get pods --all-namespaces
。
7、这些都是坑
通过k8s创建pod之后,在slave节点上指向
docker ps
命令查看容器时提示:Cannot connect to the Docker daemon. Is the docker daemon running on this host原因:systemctl启动的服务属于root,slave节点上的docker服务是跟随kubelet启动的,所以要想执行docker命令,必须以root身份执行,正确的命令
sudo docker ps
。Service通过nodePort方式暴露服务时,浏览器访问提示:“拒绝链接请求”,检查防火墙端口已经放开,但是彻底关闭防火墙时可以正常访问
原因(以上面的30001端口为例):
查看防火墙端口:
[chenlei@node-1 ~]$ sudo firewall-cmd --list-all [sudo] password for chenlei: public (active) target: default icmp-block-inversion: no interfaces: eth0 sources: services: dhcpv6-client ssh ports: 2379/tcp 443/tcp 30001/tcp 80/tcp 8080/tcp 30002/tcp protocols: masquerade: no forward-ports: sourceports: icmp-blocks: rich rules:
查看iptables规则(k8s是通过iptables配置转发规则)
[chenlei@node-1 ~]$ sudo iptables -L -n --line-numbers Chain INPUT (policy ACCEPT) num target prot opt source destination 1 KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 4 INPUT_direct all -- 0.0.0.0/0 0.0.0.0/0 5 INPUT_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0 6 INPUT_ZONES all -- 0.0.0.0/0 0.0.0.0/0 7 DROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID 8 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) num target prot opt source destination 1 DOCKER-ISOLATION all -- 0.0.0.0/0 0.0.0.0/0 2 DOCKER all -- 0.0.0.0/0 0.0.0.0/0 3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 4 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 5 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 6 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 7 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 8 FORWARD_direct all -- 0.0.0.0/0 0.0.0.0/0 9 FORWARD_IN_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0 10 FORWARD_IN_ZONES all -- 0.0.0.0/0 0.0.0.0/0 11 FORWARD_OUT_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0 12 FORWARD_OUT_ZONES all -- 0.0.0.0/0 0.0.0.0/0 13 DROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID 14 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) num target prot opt source destination 1 KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */ 2 KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 3 OUTPUT_direct all -- 0.0.0.0/0 0.0.0.0/0 Chain DOCKER (1 references) num target prot opt source destination Chain DOCKER-ISOLATION (1 references) num target prot opt source destination 1 RETURN all -- 0.0.0.0/0 0.0.0.0/0 Chain FORWARD_IN_ZONES (1 references) num target prot opt source destination 1 FWDI_public all -- 0.0.0.0/0 0.0.0.0/0 [goto] 2 FWDI_public all -- 0.0.0.0/0 0.0.0.0/0 [goto] Chain FORWARD_IN_ZONES_SOURCE (1 references) num target prot opt source destination Chain FORWARD_OUT_ZONES (1 references) num target prot opt source destination 1 FWDO_public all -- 0.0.0.0/0 0.0.0.0/0 [goto] 2 FWDO_public all -- 0.0.0.0/0 0.0.0.0/0 [goto] Chain FORWARD_OUT_ZONES_SOURCE (1 references) num target prot opt source destination Chain FORWARD_direct (1 references) num target prot opt source destination Chain FWDI_public (2 references) num target prot opt source destination 1 FWDI_public_log all -- 0.0.0.0/0 0.0.0.0/0 2 FWDI_public_deny all -- 0.0.0.0/0 0.0.0.0/0 3 FWDI_public_allow all -- 0.0.0.0/0 0.0.0.0/0 4 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 Chain FWDI_public_allow (1 references) num target prot opt source destination Chain FWDI_public_deny (1 references) num target prot opt source destination Chain FWDI_public_log (1 references) num target prot opt source destination Chain FWDO_public (2 references) num target prot opt source destination 1 FWDO_public_log all -- 0.0.0.0/0 0.0.0.0/0 2 FWDO_public_deny all -- 0.0.0.0/0 0.0.0.0/0 3 FWDO_public_allow all -- 0.0.0.0/0 0.0.0.0/0 Chain FWDO_public_allow (1 references) num target prot opt source destination Chain FWDO_public_deny (1 references) num target prot opt source destination Chain FWDO_public_log (1 references) num target prot opt source destination Chain INPUT_ZONES (1 references) num target prot opt source destination 1 IN_public all -- 0.0.0.0/0 0.0.0.0/0 [goto] 2 IN_public all -- 0.0.0.0/0 0.0.0.0/0 [goto] Chain INPUT_ZONES_SOURCE (1 references) num target prot opt source destination Chain INPUT_direct (1 references) num target prot opt source destination Chain IN_public (2 references) num target prot opt source destination 1 IN_public_log all -- 0.0.0.0/0 0.0.0.0/0 2 IN_public_deny all -- 0.0.0.0/0 0.0.0.0/0 3 IN_public_allow all -- 0.0.0.0/0 0.0.0.0/0 4 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 Chain IN_public_allow (1 references) num target prot opt source destination 1 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate NEW 2 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:2379 ctstate NEW 3 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 ctstate NEW 4 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30001 ctstate NEW 5 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 ctstate NEW 6 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 ctstate NEW 7 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30002 ctstate NEW Chain IN_public_deny (1 references) num target prot opt source destination Chain IN_public_log (1 references) num target prot opt source destination Chain KUBE-FIREWALL (2 references) num target prot opt source destination 1 DROP all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000 Chain KUBE-SERVICES (1 references) num target prot opt source destination Chain OUTPUT_direct (1 references) num target prot opt source destination
发现FORWARD表中第14行REJECT了所有的请求,尝试删除此规则并刷新页面
[chenlei@node-1 ~]$ sudo iptables -D FORWARD 14 [chenlei@node-2 ~]$ sudo iptables -D FORWARD 14
刷新后页面能正常打开,但是重启firewalld服务后,iptables规则又恢复了,这种方式只能临时解决问题。因为后续的工作中并不打算使用nodePort方式,所以不再深入研究。
进入容器内部之后,尝试ping其他k8s service对应的CLUSTER-IP(VIP),发现无法ping通
原因:这是正常现象,因为从vip到容器内部到访问路径是通过iptables的dnat转发规则配置的,明确指定了端口,从上面iptables的Chain IN_public_allow可以看到。如果想测试服务是否正常访问可以使用curl。
K8S通过iptables的转发过程参考:http://www.jianshu.com/p/bbb673e79c3e
8、附录
8.1、使用阿里的镜像加速以及使用自己搭建的私库
编辑node-1和node-2上的/etc/sysconfig/docker文件,修改OPTIONS和INSECURE_REGISTRY的值,文件完整内容如下(registry-mirror使用你自己的加速地址,nsecure-registry使用你本地的镜像仓库):
# /etc/sysconfig/docker
# Modify these options if you want to change the way the docker daemon runs
#OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false --registry-mirror https://xxxxxxxx.mirror.aliyuncs.com'
if [ -z "${DOCKER_CERT_PATH}" ]; then
DOCKER_CERT_PATH=/etc/docker
fi
# If you want to add your own registry to be used for docker search and docker
# pull use the ADD_REGISTRY option to list a set of registries, each prepended
# with --add-registry flag. The first registry added will be the first registry
# searched.
#ADD_REGISTRY='--add-registry registry.access.redhat.com'
# If you want to block registries from being used, uncomment the BLOCK_REGISTRY
# option and give it a set of registries, each prepended with --block-registry
# flag. For example adding docker.io will stop users from downloading images
# from docker.io
# BLOCK_REGISTRY='--block-registry'
# If you have a registry secured with https but do not have proper certs
# distributed, you can tell docker to not look for full authorization by
# adding the registry to the INSECURE_REGISTRY line and uncommenting it.
# INSECURE_REGISTRY='--insecure-registry'
INSECURE_REGISTRY='--insecure-registry 10.211.55.15:5000'
# On an SELinux system, if you remove the --selinux-enabled option, you
# also need to turn on the docker_transition_unconfined boolean.
# setsebool -P docker_transition_unconfined 1
# Location used for temporary files, such as those created by
# docker load and build operations. Default is /var/lib/docker/tmp
# Can be overriden by setting the following environment variable.
# DOCKER_TMPDIR=/var/tmp
# Controls the /etc/cron.daily/docker-logrotate cron job status.
# To disable, uncomment the line below.
# LOGROTATE=false
#
# docker-latest daemon can be used by starting the docker-latest unitfile.
# To use docker-latest client, uncomment below lines
#DOCKERBINARY=/usr/bin/docker-latest
#DOCKERDBINARY=/usr/bin/dockerd-latest
#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest
~
~
8.2、镜像网盘下载地址
下载地址
pod-infrastructure.tar.gz:http://pan.baidu.com/s/1bWWMWA
nginx-ingress-controller-0.9.0-beta.13.tar.gz:http://pan.baidu.com/s/1nvn5rLn
kubernetes-dashboard-amd64-v1.5.1.tar.gz:http://pan.baidu.com/s/1bBYgIq
defaultbackend-1.0.tar.gz:http://pan.baidu.com/s/1jHDdFuu
导入方法(以kubernetes-dashboard-amd64-v1.5.1.tar.gz为例)
--- gzip解压得到tar文件 --- [chenlei@node-1 ~]$ gzip -d kubernetes-dashboard-amd64-v1.5.1.tar.gz --- docker load --- [chenlei@node-1 ~]$ sudo docker load -i ./kubernetes-dashboard-amd64-v1.5.1.tar --- 查看本地images --- [chenlei@node-1 ~]$ sudo docker images --- tag --- [chenlei@node-1 ~]$ sudo docker tag gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1 10.211.55.15:5000/google_containers/kubernetes-dashboard-amd64:v1.5.1 --- push --- [chenlei@node-1 ~]$ sudo docker push 10.211.55.15:5000/google_containers/kubernetes-dashboard-amd64:v1.5.1