文章目录
Service
Service是Pod的逻辑分组,Service使用Label selector来匹配Pod的Labels(Pod用Labels来确定自己在哪个Service分组)
Service能够提供负载均衡的能力,但只有4层负载均衡能力
Service的类型:
- ClusterIP:默认类型,自动分配一个仅Cluster内部可以访问的虚拟IP
- NodePort:在ClusterIP基础上为Service在每台机器上绑定一个端口,外部可以通过NodeIP:NodePort来访问该服务
- LoadBalancer:在NodePort的基础上,借助cloud provider创建一个外部负载均衡器,并将请求转发到NodeIP:NodePort
- ExternalName:把集群外的服务引入到集群内部,在集群内部直接使用
ClusterIP
当请求发送到对应ClusterIP和端口,
将被转发(iptables或ipvs)到kube-proxy中,
kube-proxy内部对请求进行负载均衡,
并查询该Service下对应Pod的地址和端口,
再将请求转发到Pod的地址和端口
# 先创建一个控制器
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-deployment
spec:
replicas: 3
selector:
matchLabels:
app: myapp
release: stabel
template:
metadata:
labels:
app: myapp
release: stabel
env: test
spec:
containers:
- name: myapp
image: nginx:1.8
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
---
# 再创建控制器下Pod的Service
apiVersion: v1
kind: Service
metadata:
name: myapp
spec:
# Service类型为ClusterIP
type: ClusterIP
# 如果Pod存在以下标签,则Pod属于该Service逻辑分组
selector:
app: myapp
release: stabel
# Pod要暴露到ClusterIP的端口组
ports:
- name: http
# ClusterIP要暴露的端口
port: 9999
# 目标Pod上的端口
targetPort: 80
kubectl get deployment,rs,pod,service -o wide
# NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
# deployment.apps/myapp-deployment 3/3 3 3 65s myapp nginx:1.8 app=myapp,release=stabel
# NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
# replicaset.apps/myapp-deployment-69696c78f7 3 3 3 65s myapp nginx:1.8 app=myapp,pod-template-hash=69696c78f7,release=stabel
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# pod/myapp-deployment-69696c78f7-8p6xb 1/1 Running 0 65s 172.19.1.49 k8s-node01 <none> <none>
# pod/myapp-deployment-69696c78f7-spg8v 1/1 Running 0 65s 172.19.1.48 k8s-node01 <none> <none>
# pod/myapp-deployment-69696c78f7-wmrpj 1/1 Running 0 65s 172.19.2.21 k8s-node02 <none> <none>
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
# service/kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 3d19h <none>
# service/myapp ClusterIP 172.18.0.204 <none> 9999/TCP 65s app=myapp,release=stabel
kubectl run -it testservice --image=busybox --image-pull-policy=IfNotPresent --restart=Never --rm=true -- wget http://myapp:9999 -O - | cat -
# Connecting to myapp:9999 (172.18.0.204:9999)
# writing to stdout
# <!DOCTYPE html>
# <html>
# <head>
# <title>Welcome to nginx!</title>
# <style>
# body {
# width: 35em;
# margin: 0 auto;
# font-family: Tahoma, Verdana, Arial, sans-serif;
# }
# </style>
# </head>
# <body>
# <h1>Welcome to nginx!</h1>
# <p>If you see this page, the nginx web server is successfully installed and
# working. Further configuration is required.</p>
# <p>For online documentation and support please refer to
# <a href="http://nginx.org/">nginx.org</a>.<br/>
# Commercial support is available at
# <a href="http://nginx.com/">nginx.com</a>.</p>
# <p><em>Thank you for using nginx.</em></p>
# </body>
# </html>
# - 100% |********************************| 612 0:00:00 ETA
# written to stdout
# pod "testservice" deleted
# 在Pod中可以直接通过Pod的名字访问(没有service的情况下,Pod只能通过IP访问目标Pod)
wget http://myapp:9999 -O - | cat -
# --2020-11-30 11:24:44-- http://myapp:9999/
# Resolving myapp (myapp)... failed: Name or service not known.
# wget: unable to resolve host address ‘myapp’
# 在集群外则不行
Headless Service
apiVersion: v1
kind: Service
metadata:
name: myapp-headless
spec:
selector:
app: myapp
# 显式声明clusterIP
clusterIP: "None"
ports:
- port: 9992
targetPort: 80
kubectl get svc -o wide
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
# kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 3d23h <none>
# myapp ClusterIP 172.18.0.204 <none> 9999/TCP 3h56m app=myapp,release=stabel
# myapp-headless ClusterIP None <none> 9992/TCP 17s app=myapp
kubectl run -it testservice --image=busybox --image-pull-policy=IfNotPresent --restart=Never --rm=true -- ping myapp-headless
# If you don't see a command prompt, try pressing enter.
# 64 bytes from 172.19.1.48: seq=1 ttl=62 time=0.634 ms
# 64 bytes from 172.19.1.48: seq=2 ttl=62 time=0.620 ms
# ^C
# --- myapp-headless ping statistics ---
# 3 packets transmitted, 3 packets received, 0% packet loss
# round-trip min/avg/max = 0.595/0.616/0.634 ms
# 集群内访问myapp-headless获得Pod地址
NodePort
在Node上开个端口,使集群外部可以访问内部服务
apiVersion: v1
kind: Service
metadata:
name: myapp-nodepod
spec:
type: NodePort
selector:
app: myapp
release: stabel
ports:
- name: http
# 集群中暴露的端口
port: 9999
# 目标Pod上的端口
targetPort: 80
kubectl get svc -o wide
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
# kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 9m6s <none>
# myapp-nodepod NodePort 172.18.0.90 <none> 9999:31231/TCP 4m43s app=myapp,release=stabel
# 随机生成集群外部可访问端口:31231
wget http://k8s-master01:31231 -O - | cat -
# --2020-11-30 15:56:15-- http://k8s-master01:31231/
# Resolving k8s-master01 (k8s-master01)... 172.16.22.101
# Connecting to k8s-master01 (k8s-master01)|172.16.22.101|:31231... connected.
# HTTP request sent, awaiting response... 200 OK
# Length: 612 [text/html]
# Saving to: ‘STDOUT’
# 100%[============================================================================>] 612 --.-K/s in 0s
# 2020-11-30 15:56:15 (34.6 MB/s) - written to stdout [612/612]
# <!DOCTYPE html>
# <html>
# <head>
# <title>Welcome to nginx!</title>
# <style>
# body {
# width: 35em;
# margin: 0 auto;
# font-family: Tahoma, Verdana, Arial, sans-serif;
# }
# </style>
# </head>
# <body>
# <h1>Welcome to nginx!</h1>
# <p>If you see this page, the nginx web server is successfully installed and
# working. Further configuration is required.</p>
# <p>For online documentation and support please refer to
# <a href="http://nginx.org/">nginx.org</a>.<br/>
# Commercial support is available at
# <a href="http://nginx.com/">nginx.com</a>.</p>
# <p><em>Thank you for using nginx.</em></p>
# </body>
# </html>
# 集群外访问k8s-master01:31231
kubectl run -it testservice --image=busybox --image-pull-policy=IfNotPresent --restart=Never --rm=true -- wget http://myapp-nodepod:9999 -O - | cat -
# Connecting to myapp-nodepod:9999 (172.18.0.90:9999)
# writing to stdout
# <!DOCTYPE html>
# <html>
# <head>
# <title>Welcome to nginx!</title>
# <style>
# body {
# width: 35em;
# margin: 0 auto;
# font-family: Tahoma, Verdana, Arial, sans-serif;
# }
# </style>
# </head>
# <body>
# <h1>Welcome to nginx!</h1>
# <p>If you see this page, the nginx web server is successfully installed and
# working. Further configuration is required.</p>
# <p>For online documentation and support please refer to
# <a href="http://nginx.org/">nginx.org</a>.<br/>
# Commercial support is available at
# <a href="http://nginx.com/">nginx.com</a>.</p>
# <p><em>Thank you for using nginx.</em></p>
# </body>
# </html>
# - 100% |********************************| 612 0:00:00 ETA
# written to stdout
# pod "testservice" deleted
# 集群内访问myapp-nodepod:9999
LoadBalancer
需要云服务提供商支持
ExternalName
映射集群外部服务到集群内,通过定义外部服务的别名来对集群内提供服务
apiVersion: v1
kind: Service
metadata:
name: my-baidu-service
spec:
type: ExternalName
# my-baidu-service.dufault.svc.cluster.local
externalName: www.baidu.com
kubectl get svc -o wide
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
# kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 45m <none>
# my-baidu-service ExternalName <none> www.baidu.com <none> 3m58s <none>
Ingress
官网:https://kubernetes.github.io/ingress-nginx/
nginx的修改版
根据不同的域名或路径访问不同的端口服务
Pod和Ingress通过Service关联
Ingress作为统一入口,由Service关联一组Pod
该组件非kubernetes内置组件,需要单独安装部署进kubernetes
安装Ingress
# 只支持helm3部署
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm search repo ingress-nginx
helm install cs-ingress ingress-nginx/ingress-nginx
# NAME: cs-ingress
# LAST DEPLOYED: Sat Dec 5 13:32:23 2020
# NAMESPACE: default
# STATUS: deployed
# REVISION: 1
# TEST SUITE: None
# NOTES:
# The ingress-nginx controller has been installed.
# It may take a few minutes for the LoadBalancer IP to be available.
# You can watch the status by running 'kubectl --namespace default get services -o wide -w cs-ingress-ingress-nginx-controller'
# An example Ingress that makes use of the controller:
# apiVersion: networking.k8s.io/v1beta1
# kind: Ingress
# metadata:
# annotations:
# kubernetes.io/ingress.class: nginx
# name: example
# namespace: foo
# spec:
# rules:
# - host: www.example.com
# http:
# paths:
# - backend:
# serviceName: exampleService
# servicePort: 80
# path: /
# # This section is only required if TLS is to be enabled for the Ingress
# tls:
# - hosts:
# - www.example.com
# secretName: example-tls
# If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
# apiVersion: v1
# kind: Secret
# metadata:
# name: example-tls
# namespace: foo
# data:
# tls.crt: <base64 encoded cert>
# tls.key: <base64 encoded key>
# type: kubernetes.io/tls
kubectl get all
# NAME READY STATUS RESTARTS AGE
# pod/cs-ingress-ingress-nginx-controller-6878568799-9spzb 1/1 Running 0 60s
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# service/cs-ingress-ingress-nginx-controller LoadBalancer 172.18.0.125 <pending> 80:30730/TCP,443:31025/TCP 36m
# service/cs-ingress-ingress-nginx-controller-admission ClusterIP 172.18.0.142 <none> 443/TCP 36m
# service/kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 4d3h
# NAME READY UP-TO-DATE AVAILABLE AGE
# deployment.apps/cs-ingress-ingress-nginx-controller 1/1 1 1 36m
# NAME DESIRED CURRENT READY AGE
# replicaset.apps/cs-ingress-ingress-nginx-controller-6878568799 1 1 1 36m
# 检测已安装版本
POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version
# -------------------------------------------------------------------------------
# NGINX Ingress controller
# Release: v0.41.2
# Build: d8a93551e6e5798fc4af3eb910cef62ecddc8938
# Repository: https://github.com/kubernetes/ingress-nginx
# nginx version: nginx/1.19.4
# -------------------------------------------------------------------------------
使用Ingress
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: nginx:1.19-alpine
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: web
spec:
type: NodePort
selector:
app: myapp
ports:
- name: web
port: 60000
targetPort: 80
---
# 创建Ingress规则
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
# 访问后端应用时使用的域名
- host: example.ingredemo.com
http:
paths:
# 路径
- path: /
backend:
# 后端应用的Service名字
serviceName: web
# 后端应用的Service端口
servicePort: 60000
kubectl get all -o wide
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# pod/cs-ingress-ingress-nginx-controller-6878568799-9spzb 1/1 Running 0 59m 172.19.1.97 k8s-node01 <none> <none>
# pod/web-84c44bf89d-mzcgh 1/1 Running 0 7s 172.19.1.100 k8s-node01 <none> <none>
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
# service/cs-ingress-ingress-nginx-controller LoadBalancer 172.18.0.125 <pending> 80:30730/TCP,443:31025/TCP 95m app.kubernetes.io/component=controller,app.kubernetes.io/instance=cs-ingress,app.kubernetes.io/name=ingress-nginx
# service/cs-ingress-ingress-nginx-controller-admission ClusterIP 172.18.0.142 <none> 443/TCP 95m app.kubernetes.io/component=controller,app.kubernetes.io/instance=cs-ingress,app.kubernetes.io/name=ingress-nginx
# service/kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 4d4h <none>
# service/web NodePort 172.18.0.146 <none> 60000:31346/TCP 7s app=myapp
# NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
# deployment.apps/cs-ingress-ingress-nginx-controller 1/1 1 1 95m controller k8s.gcr.io/ingress-nginx/controller:v0.41.2@sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de app.kubernetes.io/component=controller,app.kubernetes.io/instance=cs-ingress,app.kubernetes.io/name=ingress-nginx
# deployment.apps/web 1/1 1 1 7s myapp-container nginx:1.19-alpine app=myapp
# NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
# replicaset.apps/cs-ingress-ingress-nginx-controller-6878568799 1 1 1 95m controller k8s.gcr.io/ingress-nginx/controller:v0.41.2@sha256:1f4f402b9c14f3ae92b11ada1dfe9893a88f0faeb0b2f4b903e2c67a0c3bf0de app.kubernetes.io/component=controller,app.kubernetes.io/instance=cs-ingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6878568799
# replicaset.apps/web-84c44bf89d 1 1 1 7s myapp-container nginx:1.19-alpine app=myapp,pod-template-hash=84c44bf89d
kubectl get ingress
# Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
# NAME CLASS HOSTS ADDRESS PORTS AGE
# example-ingress <none> example.ingredemo.com 80 43s
kubectl get svc
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# cs-ingress-ingress-nginx-controller LoadBalancer 172.18.0.125 <pending> 80:30730/TCP,443:31025/TCP 106m
# cs-ingress-ingress-nginx-controller-admission ClusterIP 172.18.0.142 <none> 443/TCP 106m
# kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 4d5h
# web NodePort 172.18.0.146 <none> 60000:31346/TCP 15s
# ----------------------------------------------------------------------------------------------------------------------------------------------------
# 直接用地址和端口访问后端对外服务(端口为31346)
curl k8s-master01:31346
# <!DOCTYPE html>
# <html>
# <head>
# <title>Welcome to nginx!</title>
# <style>
# body {
# width: 35em;
# margin: 0 auto;
# font-family: Tahoma, Verdana, Arial, sans-serif;
# }
# </style>
# </head>
# <body>
# <h1>Welcome to nginx!</h1>
# <p>If you see this page, the nginx web server is successfully installed and
# working. Further configuration is required.</p>
# <p>For online documentation and support please refer to
# <a href="http://nginx.org/">nginx.org</a>.<br/>
# Commercial support is available at
# <a href="http://nginx.com/">nginx.com</a>.</p>
# <p><em>Thank you for using nginx.</em></p>
# </body>
# </html>
# 向hosts写入域名
echo "example.ingredemo.com 172.16.22.101" >> /etc/hosts
# 测试域名(端口为ingress对外端口30730)
curl example.ingredemo.com:30730
# <!DOCTYPE html>
# <html>
# <head>
# <title>Welcome to nginx!</title>
# <style>
# body {
# width: 35em;
# margin: 0 auto;
# font-family: Tahoma, Verdana, Arial, sans-serif;
# }
# </style>
# </head>
# <body>
# <h1>Welcome to nginx!</h1>
# <p>If you see this page, the nginx web server is successfully installed and
# working. Further configuration is required.</p>
# <p>For online documentation and support please refer to
# <a href="http://nginx.org/">nginx.org</a>.<br/>
# Commercial support is available at
# <a href="http://nginx.com/">nginx.com</a>.</p>
# <p><em>Thank you for using nginx.</em></p>
# </body>
# </html>
# 使用错误的域名访问ingress端口(一个错误的示范)
curl k8s-master01:30730
# <html>
# <head><title>404 Not Found</title></head>
# <body>
# <center><h1>404 Not Found</h1></center>
# <hr><center>nginx</center>
# </body>
# </html>
# 使用IP访问ingress端口(另一个错误的示范)
curl 172.16.22.101:30730
# <html>
# <head><title>404 Not Found</title></head>
# <body>
# <center><h1>404 Not Found</h1></center>
# <hr><center>nginx</center>
# </body>
# </html>
HPA
HPA通过定期查询Pod的状态,获得Pod的监控数据
然后通过现有Pod的使用率的平均值跟目标使用率进行比较
Pod的使用率的平均值:
监控资源1分钟使用的平均值/设定的每个Pod的request资源值
扩容的Pod数计算公式:
TargetNumOfPods = ceil(sum(CurrentPodsCPUUtuilization)/Target)
Ceil函数:
返回大于或等于指定表达式的最小整数
每次扩容和缩容时都有一个窗口时间,
即每次伸缩操作后就会进入窗口时间,
在此时间段不会再进行伸缩操作。
- 扩容后3分钟窗口时间
- 缩容后5分钟窗口时间
之后如果avg/Target下降9%,会再进行缩容,
或增加10%,将进行扩容
HPA的实现条件:
- HPA不能autoscale daemonset类型control
- 要实现autoscale,Pod必须设置request
监控指标收集过程:
- kubelet收集本机的监控数据,存储与内容中
- Metrics server周期性的从kubelet的Summary API采集信息
- 通过metric-api的形式暴露出去
- HPA控制器直接同metric-api获取监控指标
安装与应用
https://github.com/kubernetes-sigs/metrics-server
# 安装metrics server
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# 在components.yaml的容器命令中加入可以使用不安全的tls(command节)
# containers:
# - args:
# - --cert-dir=/tmp
# - --secure-port=4443
# - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
# - --kubelet-use-node-status-port
# command: ["/metrics-server", "--kubelet-insecure-tls"] 加入这一行
# image: k8s.gcr.io/metrics-server/metrics-server:v0.4.1
# imagePullPolicy: IfNotPresent
# 没有上面那步,探针将无法探测成功
kubectl apply -f components.yaml
# serviceaccount/metrics-server created
# clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
# clusterrole.rbac.authorization.k8s.io/system:metrics-server created
# rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
# clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
# clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
# service/metrics-server created
# deployment.apps/metrics-server created
# apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
kubectl -n kube-system get all | grep metrics
# pod/metrics-server-56c887f446-4gxg8 1/1 Running 0 10m
# service/metrics-server ClusterIP 172.18.0.15 <none> 443/TCP 10m
# deployment.apps/metrics-server 1/1 1 1 10m
# replicaset.apps/metrics-server-56c887f446 1 1 1 10m
# 测试是否安装成功
kubectl top node
# NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
# k8s-master01 229m 5% 2883Mi 36%
# k8s-node01 110m 2% 2271Mi 28%
# k8s-node02 42m 1% 2660Mi 33%
使用HPA
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-hpa-deployment
labels:
app: test-hpa
spec:
selector:
matchLabels:
app: test-hpa-pod
template:
metadata:
labels:
app: test-hpa-pod
spec:
containers:
- name: app-container
image: nginx:1.19-alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
# 要使用HPA必须要有请求资源,不同版本HPA对能支持的资源不同
resources:
requests:
cpu: 100m
---
apiVersion: v1
kind: Service
metadata:
name: web-test-hpa-service
spec:
type: NodePort
selector:
app: test-hpa-pod
ports:
- name: web
port: 60001
targetPort: 80
---
# HPA版本影响HPA能度量的资源
apiVersion: autoscaling/v2beta2
# HPA
kind: HorizontalPodAutoscaler
metadata:
name: test-hpa
spec:
# 最多增加到10个实例
maxReplicas: 10
# 最少减少到1个实例
minReplicas: 1
# 被测量目标参照
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-hpa-deployment
metrics:
- type: Resource
resource:
name: cpu
target:
# 类型:利用率
type: Utilization
averageUtilization: 80
kubectl get hpa
# NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
# test-hpa Deployment/test-hpa-deployment 0%/80% 1 10 1 3m21s
kubectl get hpa,svc,deployment,pod
# NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
# horizontalpodautoscaler.autoscaling/test-hpa Deployment/test-hpa-deployment 0%/80% 1 10 1 4m26s
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# service/web-test-hpa-service NodePort 172.18.0.193 <none> 60001:32553/TCP 4m26s
# NAME READY UP-TO-DATE AVAILABLE AGE
# deployment.apps/web 1/1 1 1 10h
# NAME READY STATUS RESTARTS AGE
# pod/test-hpa-deployment-5bfcdf778-m7rgr 1/1 Running 0 4m26s