aws eks 配置nginx tls 和 nginx ingress controller

zhaojiew10

已于 2024-05-27 20:45:58 修改

阅读量1k

点赞数 1

分类专栏： AWS 文章标签： nginx aws 运维

于 2022-11-06 22:53:59 首次发布

本文链接：https://blog.csdn.net/sinat_41567654/article/details/127722967

版权

AWS 专栏收录该内容

72 篇文章 4 订阅

订阅专栏

参考资料

版本区分

社区版 – 社区版 Ingress Controller 以 NGINX 开源技术为基础（文档参见 Kubernetes.io），可在 GitHub 的 kubernetes/ingress-nginx 代码库中找到。它由 Kubernetes 社区维护，并且 F5 NGINX 承诺帮助管理该项目。

NGINX 版本

– NGINX Ingress Controller 由 F5 NGINX 开发和维护，可在 GitHub 的nginxinc/kubernetes-ingress代码库中找到。它有两个版本：

基于 NGINX 开源技术（开放的开源版本）
基于 NGINX Plus（商用版本）

社区版 ingress controller

安装ingress controller

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace
##或者
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/cloud/deploy.yaml

有些无法下载的镜像手动导入

# 导出镜像
# sudo nerdctl -n=k8s.io save -o temp.tar quay.io/prometheus/node-exporter:v1.5.0
# sudo ctr -n=k8s.io image  export --platform=linux/amd64  temp.tar quay.io/prometheus/node-exporter:v1.5.0
export imagename=registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0
# docker pull $imagename
docker save -o temp.tar $imagename && aws s3 cp temp.tar s3://zhaojiew-test
# 导入镜像
aws s3 cp s3://zhaojiew-test/temp.tar . && sudo ctr -n=k8s.io image import temp.tar
# nerdctl -n=k8s.io load -i temp.tar
# docker load -i temp.tar

# 查看镜像
ctr -n=k8s.io image ls

或者使用加速工具（pullimage.sh）拉镜像再传到docker hub中

docker.io (docker hub公共镜像库)

gcr.io (Google container registry)

k8s.gcr.io ( gcr.io/google-containers)

quay.io (Red Hat运营的镜像库)

#!/bin/sh
k8s_img=$1
mirror_img=$(echo ${k8s_img}|
        sed 's/quay\.io/anjia0532\/quay/g;s/ghcr\.io/anjia0532\/ghcr/g;s/registry\.k8s\.io/anjia0532\/google-containers/g;s/k8s\.gcr\.io/anjia0532\/google-containers/g;s/gcr\.io/anjia0532/g;s/\//\./g;s/ /\n/g;s/anjia0532\./anjia0532\//g' |
        uniq)
if [ -x "$(command -v docker)" ]; then
  sudo docker pull ${mirror_img}
  sudo docker tag ${mirror_img} ${k8s_img}
  exit 0
fi
if [ -x "$(command -v ctr)" ]; then
  sudo ctr -n k8s.io image pull docker.io/${mirror_img}
  sudo ctr -n k8s.io image tag docker.io/${mirror_img} ${k8s_img}
  exit 0
fi
echo "command not found:docker or ctr"

拉取镜像

bash pullimage.sh registry.k8s.io/ingress-nginx/controller:v1.4.0
bash pullimage.sh registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343

nginx ingress controller实际上是一个nginx管理器，暴露服务类型默认为LoadBalancer，为外部访问提供了统一的入口

但是由于80和443端口未公开，将service暴露端口修改为8080和8443

nginx ingress controller进程

bash-5.1$ ps
PID   USER     TIME  COMMAND
1 www-data  0:00 /usr/bin/dumb-init -- /nginx-ingress-controller --publish-service=ingress-nginx/ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingr
7 www-data  0:04 /nginx-ingress-controller --publish-service=ingress-nginx/ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --ingress-cl
26 www-data  0:00 nginx: master process /usr/bin/nginx -c /etc/nginx/nginx.conf
476 www-data  0:00 nginx: worker process
477 www-data  0:00 nginx: worker process
478 www-data  0:00 nginx: cache manager process
543 www-data  0:00 bash
560 www-data  0:00 ps

创建ingress资源，自动将controller的externalip作为address

apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: demo-ingress
   namespace: nginx-demo
 spec:
   ingressClassName: nginx
   rules:
   - http:
       paths:
       - backend:
           service:
             name: bar-service
             port:
               number: 5678
         path: /bar
         pathType: Prefix
   - http:
       paths:
       - backend:
           service:
             name: foo-service
             port:
               number: 5678
         path: /foo
         pathType: Prefix

查看配置文件

bash-5.1$ cat /etc/nginx/nginx.conf | grep 'bar'
                location /bar/ {
                        set $service_name   "bar-service";
                        set $location_path  "/bar";
                        set $proxy_upstream_name "nginx-demo-bar-service-5678";
                location = /bar {
                        set $service_name   "bar-service";
                        set $location_path  "/bar";
                        set $proxy_upstream_name "nginx-demo-bar-service-5678";
bash-5.1$ cat /etc/nginx/nginx.conf | grep 'foo'
                location /foo/ {
                        set $service_name   "foo-service";
                        set $location_path  "/foo";
                        set $proxy_upstream_name "nginx-demo-foo-service-5678";
                location = /foo {
                        set $service_name   "foo-service";
                        set $location_path  "/foo";
                        set $proxy_upstream_name "nginx-demo-foo-service-5678";

访问生效

$ curl -k https://xxxxxxxxx.cn-north-1.elb.amazonaws.com.cn:8443/foo/
foo
$ curl -k https://xxxxxxxxx.cn-north-1.elb.amazonaws.com.cn:8443/bar/
bar

Nginx版 ingress controller

安装ingress controller

helm repo add nginx-stable https://helm.nginx.com/stable
helm install my-release nginx-stable/nginx-ingress

配置文件路径有所不同

nginx@my-release-nginx-ingress-6599f5dbdf-smq2g:/etc/nginx/conf.d$ pwd
/etc/nginx/conf.d

部署示例，查看服务配置

$ grep coffee cafe-cafe-ingress.conf
upstream cafe-cafe-ingress-cafe.example.com-coffee-svc-80 {
        zone cafe-cafe-ingress-cafe.example.com-coffee-svc-80 256k;
        location /coffee {
                set $service "coffee-svc";
                proxy_pass http://cafe-cafe-ingress-cafe.example.com-coffee-svc-80;
$ grep tea cafe-cafe-ingress.conf
upstream cafe-cafe-ingress-cafe.example.com-tea-svc-80 {
        zone cafe-cafe-ingress-cafe.example.com-tea-svc-80 256k;
        location /tea {
                set $service "tea-svc";
                proxy_pass http://cafe-cafe-ingress-cafe.example.com-tea-svc-80;

访问nodeport

curl -k --resolve cafe.example.com:31850:52.81.193.82 https://cafe.example.com:31850/tea
Server address: 192.168.30.19:8080
Server name: tea-5c457db9-vtw5p
Date: 06/Nov/2022:10:55:30 +0000
URI: /tea
Request ID: 3d3cf50f18516aa433505d4eb431456f

配置nginx的tls

证书生成

# 两级签发
openssl genrsa -out ca.key 1024
openssl req -new -key ca.key -out ca.csr
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt
openssl genrsa -out server.key 1024
openssl req -new -key server.key -out server.csr
openssl x509 -req -CA ca.crt -CAkey ca.key -CAcreateserial -in server.csr -out server.crt

# 直接签发
openssl genrsa -out server.key 1024
openssl req -new -key server.key -out server.csr
openssl x509 -req -in server.csr -out server.crt -signkey server.key -days 3650

# 生成pem
openssl genrsa -out privkey.pem 2048
openssl req -new -key privkey.pem -out cert.csr
openssl req -new -x509 -key privkey.pem -out pubcert.pem -days 1000

创建tls密钥

apiVersion: v1
kind: Secret
metadata:
  name: nginx-secret
type: kubernetes.io/tls
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMakNDQWhZQ0NRREFPRjl0THNhWFdqQU5CZ2txaGtpRzl3MEJBUXNGQURCYU1Rc3dDUVlEVlFRR0V3SlYKVXpFTE1Ba0dBMVVFQ0F3Q1EwRXhJVEFmQmdOVkJBb01HRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MApaREViTUJrR0ExVUVBd3dTWTJGbVpTNWxlR0Z0Y0d4bExtTnZiU0FnTUI0WERURTRNRGt4TWpFMk1UVXpOVm9YCkRUSXpNRGt4TVRFMk1UVXpOVm93V0RFTE1Ba0dBMVVFQmhNQ1ZWTXhDekFKQmdOVkJBZ01Ba05CTVNFd0h3WUQKVlFRS0RCaEpiblJsY201bGRDQlhhV1JuYVhSeklGQjBlU0JNZEdReEdUQVhCZ05WQkFNTUVHTmhabVV1WlhoaApiWEJzWlM1amIyMHdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDcDZLbjdzeTgxCnAwanVKL2N5ayt2Q0FtbHNmanRGTTJtdVpOSzBLdGVjcUcyZmpXUWI1NXhRMVlGQTJYT1N3SEFZdlNkd0kyaloKcnVXOHFYWENMMnJiNENaQ0Z4d3BWRUNyY3hkam0zdGVWaVJYVnNZSW1tSkhQUFN5UWdwaW9iczl4N0RsTGM2SQpCQTBaalVPeWwwUHFHOVNKZXhNVjczV0lJYTVyRFZTRjJyNGtTa2JBajREY2o3TFhlRmxWWEgySTVYd1hDcHRDCm42N0pDZzQyZitrOHdnemNSVnA4WFprWldaVmp3cTlSVUtEWG1GQjJZeU4xWEVXZFowZXdSdUtZVUpsc202OTIKc2tPcktRajB2a29QbjQxRUUvK1RhVkVwcUxUUm9VWTNyemc3RGtkemZkQml6Rk8yZHNQTkZ4MkNXMGpYa05MdgpLbzI1Q1pyT2hYQUhBZ01CQUFFd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLSEZDY3lPalp2b0hzd1VCTWRMClJkSEliMzgzcFdGeW5acS9MdVVvdnNWQTU4QjBDZzdCRWZ5NXZXVlZycTVSSWt2NGxaODFOMjl4MjFkMUpINnIKalNuUXgrRFhDTy9USkVWNWxTQ1VwSUd6RVVZYVVQZ1J5anNNL05VZENKOHVIVmhaSitTNkZBK0NuT0Q5cm4yaQpaQmVQQ0k1ckh3RVh3bm5sOHl3aWozdnZRNXpISXV5QmdsV3IvUXl1aTlmalBwd1dVdlVtNG52NVNNRzl6Q1Y3ClBwdXd2dWF0cWpPMTIwOEJqZkUvY1pISWc4SHc5bXZXOXg5QytJUU1JTURFN2IvZzZPY0s3TEdUTHdsRnh2QTgKN1dqRWVxdW5heUlwaE1oS1JYVmYxTjM0OWVOOThFejM4Zk9USFRQYmRKakZBL1BjQytHeW1lK2lHdDVPUWRGaAp5UkU9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBcWVpcCs3TXZOYWRJN2lmM01wUHJ3Z0pwYkg0N1JUTnBybVRTdENyWG5LaHRuNDFrCkcrZWNVTldCUU5semtzQndHTDBuY0NObzJhN2x2S2wxd2k5cTIrQW1RaGNjS1ZSQXEzTVhZNXQ3WGxZa1YxYkcKQ0pwaVJ6ejBza0lLWXFHN1BjZXc1UzNPaUFRTkdZMURzcGRENmh2VWlYc1RGZTkxaUNHdWF3MVVoZHErSkVwRwp3SStBM0kreTEzaFpWVng5aU9WOEZ3cWJRcCt1eVFvT05uL3BQTUlNM0VWYWZGMlpHVm1WWThLdlVWQ2cxNWhRCmRtTWpkVnhGbldkSHNFYmltRkNaYkp1dmRySkRxeWtJOUw1S0Q1K05SQlAvazJsUkthaTAwYUZHTjY4NE93NUgKYzMzUVlzeFR0bmJEelJjZGdsdEkxNURTN3lxTnVRbWF6b1Z3QndJREFRQUJBb0lCQVFDUFNkU1luUXRTUHlxbApGZlZGcFRPc29PWVJoZjhzSStpYkZ4SU91UmF1V2VoaEp4ZG01Uk9ScEF6bUNMeUw1VmhqdEptZTIyM2dMcncyCk45OUVqVUtiL1ZPbVp1RHNCYzZvQ0Y2UU5SNThkejhjbk9SVGV3Y290c0pSMXBuMWhobG5SNUhxSkpCSmFzazEKWkVuVVFmY1hackw5NGxvOUpIM0UrVXFqbzFGRnM4eHhFOHdvUEJxalpzVjdwUlVaZ0MzTGh4bndMU0V4eUZvNApjeGI5U09HNU9tQUpvelN0Rm9RMkdKT2VzOHJKNXFmZHZ5dGdnOXhiTGFRTC94MGtwUTYyQm9GTUJEZHFPZVBXCktmUDV6WjYvMDcvdnBqNDh5QTFRMzJQem9idWJzQkxkM0tjbjMyamZtMUU3cHJ0V2wrSmVPRmlPem5CUUZKYk4KNHFQVlJ6NWhBb0dCQU50V3l4aE5DU0x1NFArWGdLeWNrbGpKNkY1NjY4Zk5qNUN6Z0ZScUowOXpuMFRsc05ybwpGVExaY3hEcW5SM0hQWU00MkpFUmgySi9xREZaeW5SUW8zY2czb2VpdlVkQlZHWTgrRkkxVzBxZHViL0w5K3l1CmVkT1pUUTVYbUdHcDZyNmpleHltY0ppbS9Pc0IzWm5ZT3BPcmxEN1NQbUJ2ek5MazRNRjZneGJYQW9HQkFNWk8KMHA2SGJCbWNQMHRqRlhmY0tFNzdJbUxtMHNBRzR1SG9VeDBlUGovMnFyblRuT0JCTkU0TXZnRHVUSnp5K2NhVQprOFJxbWRIQ2JIelRlNmZ6WXEvOWl0OHNaNzdLVk4xcWtiSWN1YytSVHhBOW5OaDFUanNSbmU3NFowajFGQ0xrCmhIY3FIMHJpN1BZU0tIVEU4RnZGQ3haWWRidUI4NENtWmlodnhicFJBb0dBSWJqcWFNWVBUWXVrbENkYTVTNzkKWVNGSjFKelplMUtqYS8vdER3MXpGY2dWQ0thMzFqQXdjaXowZi9sU1JxM0hTMUdHR21lemhQVlRpcUxmZVpxYwpSMGlLYmhnYk9jVlZrSkozSzB5QXlLd1BUdW14S0haNnpJbVpTMGMwYW0rUlk5WUdxNVQ3WXJ6cHpjZnZwaU9VCmZmZTNSeUZUN2NmQ21mb09oREN0enVrQ2dZQjMwb0xDMVJMRk9ycW40M3ZDUzUxemM1em9ZNDR1QnpzcHd3WU4KVHd2UC9FeFdNZjNWSnJEakJDSCtULzZzeXNlUGJKRUltbHpNK0l3eXRGcEFOZmlJWEV0LzQ4WGY2ME54OGdXTQp1SHl4Wlp4L05LdER3MFY4dlgxUE9ucTJBNWVpS2ErOGpSQVJZS0pMWU5kZkR1d29seHZHNmJaaGtQaS80RXRUCjNZMThzUUtCZ0h0S2JrKzdsTkpWZXN3WEU1Y1VHNkVEVXNEZS8yVWE3ZlhwN0ZjanFCRW9hcDFMU3crNlRYcDAKWmdybUtFOEFSek00NytFSkhVdmlpcS9udXBFMTVnMGtKVzNzeWhwVTl6WkxPN2x0QjBLSWtPOVpSY21Vam84UQpjcExsSE1BcWJMSjhXWUdKQ2toaVd4eWFsNmhZVHlXWTRjVmtDMHh0VGwvaFVFOUllTktvCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

创建configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  labels:
    app: nginx
data:
  my-nginx-confing.conf: |
    server {
          listen       80;
          listen       [::]:80;
          listen       443 ssl;
          listen       [::]:443 ssl;
          ssl_certificate     certs/tls.crt;
          ssl_certificate_key certs/tls.key;
          server_name  _;
          ssl_ciphers  HIGH:!aNULL:!MD5;
          location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
          }   
          include /etc/nginx/default.d/*.conf;
    }

创建nginx部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dep
  namespace: default
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
          - name: config
            mountPath: /etc/nginx/conf.d
            readOnly: true
          - name: certs
            mountPath: /etc/nginx/certs
            readOnly: true
      volumes:
        - name: config
          configMap:
            name: nginx-config
            items:
            - key: my-nginx-confing.conf
              path: https.conf
        - name: certs
          secret:
            secretName: nginx-secret

访问443

# curl -k -v --resolve cafe.example.com:443:192.168.27.80 https://cafe.example.com:443
* Added cafe.example.com:443:192.168.27.80 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 192.168.27.80:443...
* Connected to cafe.example.com (192.168.27.80) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=CA; O=Internet Widgits Pty Ltd; CN=cafe.example.com
*  start date: Sep 12 16:15:35 2018 GMT
*  expire date: Sep 11 16:15:35 2023 GMT
*  issuer: C=US; ST=CA; O=Internet Widgits Pty Ltd; CN=cafe.example.com  
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> Host: cafe.example.com
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.23.2
< Date: Sun, 06 Nov 2022 14:46:40 GMT
< Content-Type: text/html
< Content-Length: 615
< Last-Modified: Wed, 19 Oct 2022 07:56:21 GMT
< Connection: keep-alive
< ETag: "634fada5-267"
< Accept-Ranges: bytes
< 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>