443 k8s配置开启nginx_[k8s]nginx-ingress配置4/7层测试

基本原理

default-backend提供了2个功能:

1. 404报错页面

2. healthz页面

# Any image is permissable as long as:

# 1. It serves a 404 page at /

# 2. It serves 200 on a /healthz endpoint

创建svc,外面访问80 映射到容器的8080.

deploy+svc

kubectl create -f default-backend.yaml

nginx-ingress搭建

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/without-rbac.yaml \

| kubectl apply -f -

默认使用的镜像

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0

gcr.io/google_containers/defaultbackend:1.4

docker pull lanny/gcr.io_google_containers_defaultbackend_1.4:v1.4

docker tag lanny/gcr.io_google_containers_defaultbackend_1.4:v1.4 gcr.io/google_containers/defaultbackend:1.4

贴上ingress的yaml

主要修改点:

通过18080访问状态页面(ingress-controller的nginx.conf决定)

http://192.168.x,x:18080/nginx_status

ingress-controller需要开启 hostNetwork: true

便于暴漏ingress的80端口和其他ingress-controller的nginx.conf暴漏的端口

namespace.yaml

apiVersion: v1

kind: Namespace

metadata:

name: ingress-nginx

default-backend.yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: default-http-backend

labels:

app: default-http-backend

namespace: ingress-nginx

spec:

replicas: 1

template:

metadata:

labels:

app: default-http-backend

spec:

terminationGracePeriodSeconds: 60

containers:

- name: default-http-backend

# Any image is permissable as long as:

# 1. It serves a 404 page at /

# 2. It serves 200 on a /healthz endpoint

image: gcr.io/google_containers/defaultbackend:1.4

livenessProbe:

httpGet:

path: /healthz

port: 8080

scheme: HTTP

initialDelaySeconds: 30

timeoutSeconds: 5

ports:

- containerPort: 8080

resources:

limits:

cpu: 10m

memory: 20Mi

requests:

cpu: 10m

memory: 20Mi

---

apiVersion: v1

kind: Service

metadata:

name: default-http-backend

namespace: ingress-nginx

labels:

app: default-http-backend

spec:

ports:

- port: 80

targetPort: 8080

selector:

app: default-http-backend

without-rbac.yaml

这个yaml官网已经更新了, 多了个

sysctl -w net.core.somaxconn=32768; sysctl -w net.ipv4.ip_local_port_range="1024 65535"

而我这里还没更新.

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: nginx-ingress-controller

namespace: ingress-nginx

spec:

replicas: 1

selector:

matchLabels:

app: ingress-nginx

template:

metadata:

labels:

app: ingress-nginx

annotations:

prometheus.io/port: '10254'

prometheus.io/scrape: 'true'

spec:

hostNetwork: true

containers:

- name: nginx-ingress-controller

image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0

args:

- /nginx-ingress-controller

- --default-backend-service=$(POD_NAMESPACE)/default-http-backend

- --configmap=$(POD_NAMESPACE)/nginx-configuration

- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

- --udp-services-configmap=$(POD_NAMESPACE)/udp-services

- --annotations-prefix=nginx.ingress.kubernetes.io

env:

- name: POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

ports:

- name: http

containerPort: 80

- name: https

containerPort: 443

livenessProbe:

failureThreshold: 3

httpGet:

path: /healthz

port: 10254

scheme: HTTP

initialDelaySeconds: 10

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 1

readinessProbe:

failureThreshold: 3

httpGet:

path: /healthz

port: 10254

scheme: HTTP

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 1

ingress-controller的本质是:

/nginx-ingress-controller 加启动参数

args:

- /nginx-ingress-controller

- --default-backend-service=$(POD_NAMESPACE)/default-http-backend

- --configmap=$(POD_NAMESPACE)/nginx-configuration

- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

- --udp-services-configmap=$(POD_NAMESPACE)/udp-services

- --annotations-prefix=nginx.ingress.kubernetes.io

tcp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

name: tcp-services

namespace: ingress-nginx

udp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

name: udp-services

namespace: ingress-nginx

启动nginx测试ingress的http 7层负载

kubectl run --image=nginx nginx --replicas=2

kubectl expose deployment nginx --port=80 ## 这里是svc端口,默认和容器的端口一致

nginx-ingress.conf

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: app-nginx-ingress

namespace: default

spec:

rules:

- host: mynginx.maotai.com

http:

paths:

- path: /

backend:

serviceName: nginx

servicePort: 80

注意: ingress虽然调用的是svc,貌似转发是client--nginx--svc--pod; 实际上ingress监控svc 自动将svc下的podip填充到nginx.conf.转发是client--nginx--pod

测试4层负载

udp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

name: udp-services

namespace: ingress-nginx

data:

53: "kube-system/kube-dns:53"

修改后,apply即可,nginx-ingress可以热更新

kubectl apply -f udp-services-configmap.yaml

$ host -t A nginx.default.svc.cluster.local 192.168.14.132

Using domain server:

Name: 192.168.x.x

Address: 192.168.x.x#53

Aliases:

nginx.default.svc.cluster.local has address 10.254.160.155

另一个tcp的示例

kind: ConfigMap

apiVersion: v1

metadata:

name: udp-services

namespace: ingress-nginx

data:

2200: "default/gitlab:22"

3306: "kube-public/mysql:3306"

2202: "kube-public/centos:22"

2203: "kube-public/mongodb:27017"

以下是nginx-ingress镜像的dockerfile.进这个容器可以看到

Dockerfile

FROM quay.io/kubernetes-ingress-controller/nginx-amd64:0.30

RUN clean-install \

diffutils \

dumb-init

# Create symlinks to redirect nginx logs to stdout and stderr docker log collector

# This only works if nginx is started with CMD or ENTRYPOINT

RUN mkdir -p /var/log/nginx \

&& ln -sf /dev/stdout /var/log/nginx/access.log \

&& ln -sf /dev/stderr /var/log/nginx/error.log

COPY . /

ENTRYPOINT ["/usr/bin/dumb-init"]

CMD ["/nginx-ingress-controller"]

默认启动后nginx-ingres的nginx.conf

root@n1:/etc/nginx# cat nginx.conf

daemon off;

w