Kubernetes集群 ingress Traefik基础

# vim traefik-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: kube-system
name: traefik-ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
- middlewaretcps
- ingressroutes
- traefikservices
- ingressroutetcps
- ingressrouteudps
- tlsoptions
- tlsstores
- serverstransports
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: kube-system

2.1.3 创建traefik配置文件

由traefik配置很多，通过CLI定义不方便，一般都通过配置文件对traefik进行参数配置，例如使用ConfigMap将配置挂载至traefik中

# vim traefik-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: traefik
namespace: kube-system
data:
traefik.yaml: |-
serversTransport:
insecureSkipVerify: true ## 略验证代理服务的 TLS 证书
api:
insecure: true ## 允许 HTTP 方式访问 API
dashboard: true ## 启用 Dashboard
debug: true ## 启用 Debug 调试模式
metrics:
prometheus: "" ## 配置 Prometheus 监控指标数据，并使用默认配置
entryPoints:
web:
address: ":80" ## 配置 80 端口，并设置入口名称为 web
websecure:
address: ":443" ## 配置 443 端口，并设置入口名称为 websecure
metrics:
address: ":8082" ## 配置 8082端口，并设置入口名称为 metrics
tcpep:
address: ":8083" ## 配置 8083端口，并设置入口名称为 tcpep,做为tcp入口
udpep:
address: ":8084/udp" ## 配置 8084端口，并设置入口名称为 udpep，做为udp入口
providers:
kubernetesCRD: "" ## 启用 Kubernetes CRD 方式来配置路由规则
kubernetesingress: "" ## 启用 Kubernetes Ingress 方式来配置路由规则
kubernetesGateway: "" ## 启用 Kubernetes Gateway API
experimental:
kubernetesGateway: true ## 允许使用 Kubernetes Gateway API
log:
filePath: "" ## 设置调试日志文件存储路径，如果为空则输出到控制台
level: error ## 设置调试日志级别
format: json ## 设置调试日志格式
accessLog:
filePath: "" ## 设置访问日志文件存储路径，如果为空则输出到控制台
format: json ## 设置访问调试日志格式
bufferingSize: 0 ## 设置访问日志缓存行数
filters:
retryAttempts: true ## 设置代理访问重试失败时，保留访问日志
minDuration: 20 ## 设置保留请求时间超过指定持续时间的访问日志
fields: ## 设置访问日志中的字段是否保留（keep 保留、drop 不保留）
defaultMode: keep ## 设置默认保留访问日志字段
names:
ClientUsername: drop
headers:
defaultMode: keep ## 设置 Header 中字段是否保留,设置默认保留 Header 中字段
names: ## 针对 Header 中特别字段特别配置保留模式
User-Agent: redact
Authorization: drop
Content-Type: keep

2.1.4 应用上述资源清单文件

kubectl apply -f traefik-crd.yaml

kubectl apply -f traefik-rbac.yaml

kubectl apply -f traefik-config.yaml

2.1.5 设置节点Label

由于使用DaemonSet方式部署Traefik，所以需要为节点设置label，当应用部署时会根据节点Label进行选择。

设置节点标签
# kubectl label nodes --all IngressProxy=true

查看节点标签
# kubectl get nodes --show-labels

如需要取消时，可执行下述命令
# kubectl label nodes --all IngressProxy-

2.2 部署Traefik

2.2.1 deploy资源清单文件准备

本次将用Daemonset方式部署traefik,便于后期扩展

本次部署通过hostport方式把Pod中容器内的80、443映射到物理机，方便集群外访问。当然你把service换Nodeport或者LB也行。

# vim traefik-deploy.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
namespace: kube-system
name: traefik
labels:
app: traefik
spec:
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.5.7
args:
- --configfile=/config/traefik.yaml
volumeMounts:
- mountPath: /config
name: config
ports:
- name: web
containerPort: 80
hostPort: 80 ## 将容器端口绑定所在服务器的 80 端口
- name: websecure
containerPort: 443
hostPort: 443 ## 将容器端口绑定所在服务器的 443 端口
- name: admin
containerPort: 8080 ## Traefik Dashboard 端口
- name: tcpep
containerPort: 8083
hostPort: 8083 ## 将容器端口绑定所在服务器的 8083 端口
- name: udpep
containerPort: 8084
hostPort: 8084 ## 将容器端口绑定所在服务器的 8084 端口
protocol: UDP
volumes:
- name: config
configMap:
name: traefik
tolerations: ## 设置容忍所有污点，防止节点被设置污点
- operator: "Exists"
nodeSelector: ## 设置node筛选器，在特定label的节点上启动
IngressProxy: "true"

验证端口配置是否正确
# kubectl get daemonset traefik -n kube-system -o yaml

2.2.2 应用deploy资源清单文件

kubectl apply -f traefix-deploy.yaml

2.2.3 service资源清单文件准备

# vim traefix-service.yaml
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: kube-system
spec:
ports:
- protocol: TCP
name: web
port: 80
- protocol: TCP
name: admin
port: 8080
- protocol: TCP
name: websecure
port: 443
- protocol: TCP
name: tcpep
port: 8083
- protocol: UDP
name: udpep
port: 8084
selector:
app: traefik

2.2.4 应用service资源清单文件

kubectl apply -f traefik-service.yaml

2.3 配置访问traefik dashboard路由规则

Traefik 应用已经部署完成，但是想让外部访问 Kubernetes 内部服务，还需要配置路由规则，上面部署 Traefik 时开启了 Traefik Dashboard，这是 Traefik 提供的视图看板，所以，首先配置基于 HTTP 的 Traefik Dashboard 路由规则，使外部能够访问 Traefik Dashboard。这里使用 IngressRoute方式进行演示。

2.3.1 Traefik创建路由规则方法

原生ingress
CRD IngressRoute
Gateway API

2.3.2 通过原生ingress方式暴露traefik dashboard

# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
traefik ClusterIP 10.96.174.88 <none> 80/TCP,8080/TCP,443/TCP,8083/TCP,8084/UDP 6h44m

# kubectl get endpoints -n kube-system
NAME ENDPOINTS AGE
traefik 10.244.135.204:80,10.244.159.141:80,10.244.194.92:80 + 17 more... 6h44m

# cat traefik-dashboard-native-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: traefik-dashboard-ingress
namespace: kube-system
annotations: #根据annotations来选择使用traefik
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
rules:
- host: tfni.huaweicce.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: traefik
port:
number: 8080

# kubectl apply -f traefik-dashboard-native-ingress.yaml

# kubectl get ingress -n kube-system
NAME CLASS HOSTS ADDRESS PORTS AGE
traefik-dashboard-ingress <none> tfni.huaweicce.com 80 56m

在集群之外主机访问

# vim /etc/hosts
......
192.168.10.12 tfni.huaweicce.com

2.3.3 创建dashboard ingress route资源清单文件

ingress route是CRD,ingress 是k8s 标准，可以把ingress route看成ingress强化版，推荐使用route

# vim traefik-dashboard-ingress-route.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik
namespace: kube-system
spec:
entryPoints:
- web
routes:
- match: Host(`traefik.huaweicce.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: traefik
port: 8080

2.3.4 应用资源清单文件

kubectl apply -f traefik-dashboard-ingress-route.yaml

2.3.5 在集群内或集群外主机配置域名解析

把域名解析为kubernetes集群任意节点IP既可。
# vim /etc/hosts
192.168.10.12 traefik.huaweicce.com

2.3.6 通过域名访问traefik dashboard

三、traefik基础应用

3.1 配置kubernetes dashboard路由规则

必须使用SSL证书创建secret密钥

3.1.1 查看kubernetes dashboard service状态

# kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.96.100.4 <none> 8000/TCP 21d
kubernetes-dashboard NodePort 10.96.19.1 <none> 443:30000/TCP 21d

3.1.2 编写访问kubernetes dashboard 路由规则

# ls
6864844_kubecce.com_nginx.zip kubecce.key kubecce.pem

# kubectl create secret tls kubecce-tls --cert=kubecce.pem --key=kubecce.key
secret/kubecce-tls created

# kubectl get secret
NAME TYPE DATA AGE
default-token-x4qbc kubernetes.io/service-account-token 3 24d
kubecce-tls kubernetes.io/tls 2 11s

# vim kubernetes-dashboard-ir.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
entryPoints:
- websecure
routes:
- match: Host(`dashboard.kubecce.com`)
kind: Rule
services:
- name: kubernetes-dashboard
port: 443
tls:
secretName: kubecce-tls

3.1.3 应用上述路由规则文件

kubectl apply -f kubernetes-dashboard-ir.yaml

3.1.4 配置域名解析及访问

# vim /etc/hosts
192.168.10.12 dashboard.kubecce.com

然后浏览器打开

四、traefik中间件 MiddleWare

4.1 traefik中间件介绍 MiddleWare

中间件是 Traefik2.0 中一个非常有特色的功能，我们可以根据自己的各种需求去选择不同的中间件来满足服务，Traefik 官方已经内置了许多不同功能的中间件，其中包括修改请求头信息；重定向；身份验证等等，而且中间件还可以通过链式组合的方式来适用各种情况。例如：强制跳转https、去除访问前缀、访问白名单等。

4.2 traefik 中间件应用案例 ipWhiteList

在工作中，有一些URL并不希望对外暴露，比如prometheus、grafana等，我们就可以通过白名单IP来过到要求，可以使用Traefix中的ipWhiteList中间件来完成。

# vim deploy-service.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: nginx-web-middleware
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: middle
template:
metadata:
labels:
app: middle
spec:
containers:
- name: nginx-web-c
image: nginx:latest
ports:
- containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
name: service-middle
namespace: default

spec:
ports:
- name: http
port: 80
selector:
app: middle

# kubectl apply -f deploy-service.yaml

# vim middleware-ipwhitelist.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: gs-ipwhitelist
spec:
ipWhiteList:
sourceRange:
- 127.0.0.1
- 10.244.0.0/16
- 10.96.0.0/12
- 192.168.10.0/24

# kubectl apply -f middleware-ipwhitelist.yaml

# vim deploy-service-middle.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingressroutemiddle
namespace: default

spec:
entryPoints:
- web
routes:
- match: Host(`middleware.kubecce.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: service-middle
port: 80
namespace: default
middlewares:
- name: gs-ipwhitelist

# kubectl apply -f deploy-service-middle.yaml

在集群之外的主机上访问

curl http://middleware.kubecce.com

把集群外主机所在的网段从白名单中删除，发现无法访问。

# cat middleware-ipwhitelist.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: gs-ipwhitelist
spec:
ipWhiteList:
sourceRange:
- 127.0.0.1
- 10.244.0.0/16
- 10.96.0.0/12

在集群外主机访问

# curl http://middleware.kubecce.com
Forbidden

五、traefik高级应用

TraefikService 目前能用于以下功能
servers load balancing.（负载均衡）
services Weighted Round Robin load balancing.（权重轮询）
services mirroring.（镜像）

5.1 负载均衡

创建ingressroute

# vim loadbalancer-deploy-service-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingressrouteweblb
namespace: default

spec:
entryPoints:
- web
routes:
- match: Host(`lb.kubecce.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: svc1
port: 80
namespace: default
- name: svc2
port: 80
namespace: default

5.2 灰度发布

# vim traefikservice.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
name: wrr
namespace: default

spec:
weighted:
services:
- name: svc1
port: 80
weight: 3 # 定义权重
kind: Service # 可选，默认就是 Service
- name: svc2
port: 80
weight: 1

创建ingressroute

需要注意的是现在我们配置的 Service 不再是直接的 Kubernetes 对象了，而是上面我们定义的 TraefikService 对象

# vim traefik-wrr.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingressroutewrr
namespace: default

spec:
entryPoints:
- web
routes:
- match: Host(`lb.kubecce.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: wrr
namespace: default
kind: TraefikService

5.3 流量复制

所谓的流量复制，也称为镜像服务是指将请求的流量按规则复制一份发送给其它服务，并且会忽略这部分请求的响应，这个功能在做一些压测或者问题复现的时候很有用。

指定流量来自己于kubernetes service对象

# vim mirror_from_service.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
name: mirror-from-service
namespace: default

spec:
mirroring:
name: svc1 # 发送 100% 的请求到 K8S 的 Service "v1"
port: 80
mirrors:
- name: svc2 # 然后复制 20% 的请求到 v2
port: 80
percent: 20

# vim mirror-from-service-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingressroute-mirror
namespace: default

spec:
entryPoints:
- web
routes:
- match: Host(`lb.kubecce.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: mirror-from-service
namespace: default
kind: TraefikService

通过traefikservice导入流量

# vim mirror-from-traefikservice.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
name: mirror-from-traefikservice
namespace: default

spec:
mirroring:
name: mirror-from-service #流量入口从TraefikService 来
kind: TraefikService
mirrors:
- name: svc2
port: 80
percent: 20

# vim mirror-from-traefikservice-ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingressroute-mirror-traefikservice
namespace: default

spec:
entryPoints:
- web
routes:
- match: Host(`lb.kubecce.com`) && PathPrefix(`/`)
kind: Rule
services:
- name: mirror-from-traefikservice
namespace: default
kind: TraefikService