参考
https://www.cnblogs.com/heian99/p/14608414.html
创建命名空间
kubectl create ns ingress-traefik
创建CRD资源
Traefik 2.0版本后开始使用CRD来对资源进行管理配置,先创建CRD资源
traefik-crd.yaml
## IngressRoute
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
---
## IngressRouteTCP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
---
## Middleware
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: Middleware
plural: middlewares
singular: middleware
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsoptions.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSOption
plural: tlsoptions
singular: tlsoption
---
## TraefikService
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: traefikservices.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: TraefikService
plural: traefikservices
singular: traefikservice
---
## TraefikTLSStore
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsstores.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSStore
plural: tlsstores
singular: tlsstore
---
## IngressRouteUDP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressrouteudps.traefik.containo.us
spec:
scope: Namespaced
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteUDP
plural: ingressrouteudps
singular: ingressrouteudp
部署 CRD 资源
kubectl apply -f traefik-crd.yaml
创建 RBAC 权限
Kubernetes 在 1.6 以后的版本中引入了基于角色的访问控制(RBAC)策略,方便对 Kubernetes 资源和 API 进行细粒度控制。Traefik 需要一定的权限,创建 Traefik ServiceAccount 并分配一定的权限
traefik-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: ingress-traefik
name: traefik-ingress-controller
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups: [""]
resources: ["services","endpoints","secrets"]
verbs: ["get","list","watch"]
- apiGroups: ["extensions"]
resources: ["ingresses"]
verbs: ["get","list","watch"]
- apiGroups: ["extensions"]
resources: ["ingresses/status"]
verbs: ["update"]
- apiGroups: ["traefik.containo.us"]
resources: ["middlewares"]
verbs: ["get","list","watch"]
- apiGroups: ["traefik.containo.us"]
resources: ["ingressroutes","traefikservices"]
verbs: ["get","list","watch"]
- apiGroups: ["traefik.containo.us"]
resources: ["ingressroutetcps","ingressrouteudps"]
verbs: ["get","list","watch"]
- apiGroups: ["traefik.containo.us"]
resources: ["tlsoptions","tlsstores"]
verbs: ["get","list","watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: ingress-traefik
部署 Traefik RBAC 资源
kubectl apply -f traefik-rbac.yaml
创建 Traefik 配置文件
由于 Traefik 配置很多,使用 CLI 定义操作过于繁琐,尽量使用将其配置选项放到配置文件中,然后存入 ConfigMap,将其挂入 traefik 中
traefik-config.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: traefik-config
namespace: ingress-traefik
data:
traefik.yaml: |-
serversTransport:
insecureSkipVerify: true
api:
insecure: true
dashboard: true
debug: true
metrics:
prometheus: ""
entryPoints:
web:
address: ":80"
websecure:
address: ":443"
providers:
kubernetesCRD: ""
kubernetesingress: ""
log:
filePath: ""
level: error
format: json
accessLog:
filePath: ""
format: json
bufferingSize: 0
filters:
retryAttempts: true
minDuration: 20
fields:
defaultMode: keep
names:
ClientUsername: drop
headers:
defaultMode: keep
names:
User-Agent: redact
Authorization: drop
Content-Type: keep
部署 Traefik ConfigMap 资源
kubectl apply -f traefik-config.yaml
设置Label标签
由于使用的Kubernetes DeamonSet方式部署Traefik,所以需要提前给节点设置Label,当程序部署Pod会自动调度到设置 Label的node节点上
节点设置 Label 标签
kubectl label nodes k8s-node-1.novalocal IngressProxy=true
kubectl label nodes k8s-node-2.novalocal IngressProxy=true
kubectl label nodes k8s-node-3.novalocal IngressProxy=true
删除label标签
kubectl label nodes k8s-node-1.novalocal IngressProxy-
查看标签
kubectl get node --show-labels
Kubernetes 部署 Traefik
使用DaemonSet类型部署,以便于在多服务器间扩展,使用 hostport 方式占用服务器 80、443 端口,方便流量进入
traefik-deploy.yaml
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: ingress-traefik
spec:
ports:
- name: web
port: 80
- name: websecure
port: 443
- name: admin
port: 8080
selector:
app: traefik
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: traefik-ingress-controller
namespace: ingress-traefik
labels:
app: traefik
spec:
selector:
matchLabels:
app: traefik
template:
metadata:
name: traefik
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 1
containers:
- image: traefik:2.2.0
name: traefik-ingress-lb
ports:
- name: web
containerPort: 80
hostPort: 80 #hostPort方式,将端口暴露到集群节点
- name: websecure
containerPort: 443
hostPort: 443 #hostPort方式,将端口暴露到集群节点
- name: admin
containerPort: 8080
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 1000m
memory: 1024Mi
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
args:
- --configfile=/config/traefik.yaml
volumeMounts:
- mountPath: "/config"
name: "config"
volumes:
- name: config
configMap:
name: traefik-config
tolerations: #设置容忍所有污点,防止节点被设置污点
- operator: "Exists"
nodeSelector: #设置node筛选器,在特定label的节点上启动
IngressProxy: "true"
部署 Traefik
kubectl apply -f traefik-deploy.yaml
Traefik 路由规则基础配置
让外部访问 Kubernetes 内部服务,还需要配置路由规则
traefik-dashboard-route.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard-route
namespace: ingress-traefik
spec:
entryPoints:
- web
routes:
- match: Host(`traefik.wuxingge.com`)
kind: Rule
services:
- name: traefik
port: 8080
部署Traefik Dashboard 路由规则对象
kubectl apply -f traefik-dashboard-route.yaml
services里面passHostHeader = true,无论是ws还是wss,都必须有这句发送头才能实现websocket
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: safe1
spec:
entryPoints:
- web
routes:
- match: Host(`xxx.cn`) && PathPrefix(`/safe`)
kind: Rule
priority: 3
middlewares:
- name: headers
namespace: default
services:
- name: safe1
port: 8080
# scheme: http
#websocket则加passHostHeader为true,wss则也一样加passHostHeader为true和tls配置
passHostHeader: true
# sticky:
# cookie:
# httpOnly: true
# name: cookie
L7代理配置
default.conf
upstream default_backend {
server 10.10.111.159:80;
server 10.10.110.195:80;
server 10.10.111.170:80;
}
server {
listen 80 default_server;
location / {
proxy_set_header Host $host;
proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
proxy_pass http://default_backend;
}
}
host解析
解析到代理IP
10.10.110.248 traefik.wuxingge.com
访问
http://traefik.wuxingge.com
部署nacos
nacos.yaml
apiVersion: v1
kind: Service
metadata:
name: vcloud-nacos-api-sit
namespace: sit
spec:
selector:
appname: vcloud-nacos-api
clusterIP: None
ports:
- port: 8848
targetPort: 8848
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vcloud-nacos-api
namespace: sit
spec:
replicas: 1
selector:
matchLabels:
appname: vcloud-nacos-api
template:
metadata:
labels:
appname: vcloud-nacos-api
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: appname
operator: In
values:
- vcloud-nacos-api
topologyKey: "kubernetes.io/hostname"
weight: 50
containers:
- name: vcloud-nacos-api
image: harbor.wuxingge.com/uat/vcloud-nacos-api:6
---
apiVersion: v1
kind: Service
metadata:
name: vcloud-nacos-web
namespace: sit
spec:
selector:
appname: vcloud-nacos-api
type: ClusterIP
ports:
- port: 8848
targetPort: 8848
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: vcloud-nacos-web
annotations:
kubernetes.io/ingress.class: "traefik"
namespace: sit
spec:
rules:
- host: nacos.wuxingge.com
http:
paths:
- path:
backend:
serviceName: vcloud-nacos-web
servicePort: 8848
配置 HTTPS 路由规则
创建自签名证书
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=kuboard.heian.com"
将证书存储到Kubernetes Secret中,新建的k8dash-sa-tls必须与k8dash-route中的tls: secretName一致
kubectl create secret tls k8dash-sa-tls --key=tls.key --cert=tls.crt -n kube-system
k8dash-route.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: k8dash-sa-route
namespace: kube-system
spec:
entryPoints:
- websecure
tls:
secretName: k8dash-sa-tls
routes:
- match: Host(`kuboard.heian.com`)
kind: Rule
services:
# 此处的services是Kubernetes中的svc name 与 端口 可以使用kubectl get svc --namespace=kube-system获取
- name: kuboard
port: 80