在k8s集群上部署一个电商demo

在这篇文章里面，将用一个k8s集群搭建一个电商demo。

文章使用了如下github项目https://github.com/RondoChen/k8s-demo.git

由于服务器数量限制，这里使用的k8s集群采用了一主二从的结构，同时由于服务器性能的限制，电商demo的MySQL数据库部署在master主机，redis缓存及其他前端、后端程序部署分别部署在2个节点中。

这套电商demo的逻辑结构如下

通过部署这套电商demo，可以熟悉日常运维工作的流程和相关技术：

1. 部署k8s集群的持久存储卷，本文采用了基于NFS搭建一个StorageClass，方便动态分配PV
2. 创建configMap和Secret，并通过存储卷引用方式被k8s集群中的程序使用
3. 熟悉有状态程序的部署
4. 熟悉通过k8s部署MySQL，并指定定制的my.cnf
5. 熟悉k8s的Job控制器的使用，并通过Job为数据库导入数据
6. 熟悉k8s的调度策略，通过配置污点容忍度及节点清和性将Pod部署到特定Node
7. 熟悉nginx的反向代理

创建NFS StorageClass

首先在master及两台node主机安装nfs，相关过程资料很多，这里就不细述了，安装好以后，执行showmount -e检查

 随后安装NFS存储分配器，这个过程并不复杂，直接下载相应的deployment.yaml，并根据自身主机的情况修改配置即可 

wget https://raw.githubusercontent.com/Kubernetes-incubator/external-storage/master/nfs-client/deploy/deployment.yaml

主要修改的是ip和目录，改为自己nfs服务器的ip和目录

执行这个yaml文件，创建NFS存储分配器 

随后创建storageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs
parameters:
  archiveOnDelete: "false"

 kubectl apply -f nfs-client-class.yaml

创建以后

部署MySQL

首先创建一个configmap和secret，作为MySQL的配置文件和登录密码

[root@VM-12-8-centos 1-mysql]# cat mysql-config.yml 
apiVersion: v1
data:
  my.cnf: "[mysqld]\nlog-bin\ncollation-server=utf8mb4_unicode_ci\ncharacter-set-server=utf8mb4\n\n[client]\ndefault-character-set=utf8mb4\n\n[mysql]\ndefault-character-set=utf8mb4 "
kind: ConfigMap
metadata:
  name: mysql
  namespace: default
[root@VM-12-8-centos 1-mysql]# cat mysql-pass.yml 
apiVersion: v1
kind: Secret
metadata:
  name: mysql-pass
type: Opaque
data:
  password: UEBzc3dvcmQxMjM0Xw==

随后创建pvc和service，考虑到生产部署MySQL均采用主从部署，所以这里配置了2个service，一个指向主库，一个指向从库，但我们实际只部署了一个数据库，所以这2个service都指向同一个MySQL即可

[root@VM-12-8-centos 1-mysql]# cat mysql-svc-pvc.yml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-11-27T09:13:52Z"
  labels:
    app: mysql
  name: mysql
  namespace: default
  resourceVersion: "3998797"
  selfLink: /api/v1/namespaces/default/services/mysql
  uid: 4946b02c-49c3-4993-844e-4f254cc31229
spec:
  clusterIP: 10.1.125.65
  ports:
  - name: tcp-mysql
    port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    app: mysql
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
  labels:
    app: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "managed-nfs-storage"
  resources:
    requests:
      storage: 2Gi
[root@VM-12-8-centos 1-mysql]# cat mysql-svc-read.yml
apiVersion: v1
kind: Service
metadata:
  name: mysql-read
  labels:
    app: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
    tier: mysql
  type: ClusterIP

最后部署mysql，这里采用ratel这个k8s界面管理系统来部署，指定了污点容忍和节点亲和性

它会自动生成相应的yaml文件，使部署更为方便，相应的代码如下

      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - vm-12-8-centos

 同样通过界面注入之前创建好的configmap和secret，生成的代码如下

      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
        - name: config-map
          mountPath: /etc/mysql/conf.d
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pvc
      - name: config-map
        configMap:
          name: mysql

最后创建，成功在master节点部署好了MySQL

通过k8s Job向MySQL导入数据

执行以下yaml文件即可

[root@VM-12-8-centos 1-mysql]# cat mysql-restore-job.yml 
apiVersion: batch/v1
kind: Job
metadata:
  name: mysql-restore
spec:
  template:
    spec:
      containers:
      - name: mysql-client
        image: rondochen/mysql-data:1.0
        command: 
          - bash
          - "-c"
          - |
            mysql -uroot -pP@ssword1234_ -hmysql --default-character-set=utf8mb4 < /data/dump.sql
      restartPolicy: Never
  backoffLimit: 2
  ttlSecondsAfterFinished: 100

进入数据库查看是否建库建表导入数据
root@mysqldb 14:46:  [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| xyzshop            |
+--------------------+
4 rows in set (0.01 sec)

root@mysqldb 14:46:  [(none)]> use mysql;
Database changed
root@mysqldb 14:47:  [mysql]> select user,host,authentication_string from user;
+--------------+-----------+-----------------------+
| user         | host      | authentication_string |
+--------------+-----------+-----------------------+
| root         | localhost |                       |
| root         | %         |                       |
| xyzshop_user | %         |                       |
+--------------+-----------+-----------------------+
3 rows in set (0.00 sec)

root@mysqldb 14:47:  [mysql]> use xyzshop;
Database changed
root@mysqldb 14:47:  [xyzshop]> show tables;
+--------------------+
| Tables_in_xyzshop  |
+--------------------+
| account            |
| hibernate_sequence |
| order              |
| order_item         |
| product            |
| product_category   |
| product_image      |
| review             |
+--------------------+

root@mysqldb 14:48:  [xyzshop]> select *  from product_category;
+-------------+--------------------+--------------------------------------------------------------------------------------------------------+
| category_id | category_title     | category_description                                                                                   |
+-------------+--------------------+--------------------------------------------------------------------------------------------------------+
|           1 | 腾讯叮当           | 能听、能看、能陪伴，腾讯叮当智能屏                                                                     |
|           2 | 腾讯极光           | 腾讯视频海量高清片源，全方位满足租房、旅行、移动办公等场景的视听需求                                   |
|           3 | 腾讯潮流手办       | 腾讯正版QQfamily手办系列，QQ2O周年纪念版手办全新推出                                                   |
|           4 | 腾讯QQfamily       | 腾讯QQfamily，用最好的方式，生产最有趣的商品                                                           |
+-------------+--------------------+--------------------------------------------------------------------------------------------------------+
==============================================================================
==============================================================================

 创建电商demo使用的configmap和secret

电商demo需要访问到指定的服务及访问数据库，从容器化部署的思想来说，这些配置不应该写死在程序中，而是应该写入configmap和secret，程序通过读取这些配置来对特定的资源进行访问，而且也方便后续程序版本的升级，如果需要读取更多的资源，只需要修改configmap和secret即可。

[root@VM-12-8-centos 2-conf]# cat app-configmap.yml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  annotations:
    strategy.spinnaker.io/versioned: "false"
data: 
  DB_HOST: mysql
  DB_HOST_READ: mysql-read
  DB_USERNAME: xyzshop_user
  DB_DATABASE: xyzshop
  REDIS_ADDRESS: redis:6379
  REDIS_DB: '0'
  SERVICE_PASSPORT: http://passport:5000
  SERVICE_PRODUCT: http://product:3000
  SERVICE_SHOPCART: http://shopcart:6000
  SERVICE_ORDER: http://order:7000
  mall-nginx-config: |-
    server { 
      listen       8000;
      index   index.html;

      location = /favicon.ico { access_log off; log_not_found off; }
      location = /robots.txt  { access_log off; log_not_found off; }

      location / {
        proxy_http_version 1.1;
        proxy_pass  http://10.0.12.8:801;
        rewrite ^/(.*)$ /mall/index.html break;
      }

      location /healthz/ready {
        access_log off;
        return 200 "ok\n";
      }

      location ^~/api/review/  {
        root /app/public;
        rewrite ^/api/review/(.*)$ /$1 break;
        try_files $uri $uri/ /index.php?$args;
      }

      location ~ \.php$ {
        set $newurl $request_uri;
        if ($newurl ~ ^/api/review(.*)$) {
          set $newurl $1;
          root /app/public;
        }

        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass review:9000;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param REQUEST_URI $newurl;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_intercept_errors off;
        fastcgi_buffer_size 16k;
        fastcgi_buffers 4 16k;
      }

      location ^~/api/passport/ {
        proxy_http_version 1.1;
        proxy_pass      http://passport:5000/;
      }
      location ^~/api/product/ {
        proxy_http_version 1.1;
        proxy_pass      http://product:3000/;
      }
      location ^~/api/shopcart/ {
        proxy_http_version 1.1;
        proxy_pass      http://shopcart:6000/;
      }
      location ^~/api/order/ {
        proxy_http_version 1.1;
        proxy_pass      http://order:7000/;
      }
    }
  mall-nginx-config-demo: |-
    server { 
      listen       8000;
      index   index.html;

      location = /favicon.ico { access_log off; log_not_found off; }
      location = /robots.txt  { access_log off; log_not_found off; }

      location / {
        proxy_http_version 1.1;
        proxy_pass  https://training-1251707795.file.myqcloud.com;
        rewrite ^/(.*)$ /mall/index.html break;
      }

      location /healthz/ready {
        access_log off;
        return 200 "ok\n";
      }

      location ^~/api/product/ {
        proxy_http_version 1.1;
        proxy_pass      http://product:3000/;
      }
    }
[root@VM-12-8-centos 2-conf]# cat app-secret.yml 
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: app-config
  annotations:
    strategy.spinnaker.io/versioned: "false"
data: 
  DB_PASSWORD: UEBzc3dvcmQxMjM=
  REDIS_PASSWORD: cmVkMXNQQHNz

部署redis

直接部署即可

apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
    - name: tcp-redis
      port: 6379
  selector:
    app: redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:latest
        imagePullPolicy: Always
        env:
          - name: REDIS_PASSWORD
            valueFrom:
              secretKeyRef:
                name: app-config
                key: REDIS_PASSWORD
        command:
          - sh 
          - -c
          - |
            redis-server --requirepass $REDIS_PASSWORD
        ports:
        - containerPort: 6379
          protocol: TCP
        resources:
          limits:
            cpu: 200m
            memory: 128Mi
          requests:
            cpu: 200m
            memory: 128Mi

部署前端应用与后端应用

这里直接使用了陈同学创建好的镜像进行部署，部署好以后的deploy和service如下

 进入容器测试服务健康情况

前端
# curl http://mall:8000/healthz/ready
ok
后端
# curl http://passport:5000/healthz/ready
{"lang":"zh-CN","service":"passport","apiVersion":"0.0.1","timeStamp":1669625731678,"success":true,"data":"I am ok."}# 
# curl http://shopcart:6000/
{"service":"shopcart","data":"hello shopcart.","error":null,"success":true,"timestamp":1669625796000}
# curl http://order:7000/healthz/ready
ok
# curl http://product:3000/healthz/ready
ok

这里需要注意的是，前端mall只是一个反向代理，真正的代码部署到了COS中，所以这里我们还要额外做多一项工作，编译mall的代码，并将编译出来的文件部署到nginx中，使k8s集群中的nginx反向代理到这个url

通过git下载代码https://github.com/RondoChen/k8s-demo.git

通过容器进行编译

docker run --rm --env NODE_OPTIONS=--openssl-legacy-provider -v /data/k8s-demo-main/src/mall:/mall  node:19.1 sh -c 'cd /mall && npm install --force && npm run build && echo "finish!"'

编译完成后，将build里的文件通过nginx部署，方便起见，直接用docker run起来

docker run -p 801:80 -v /data/k8s-demo-main/mall/build:/usr/share/nginx/html -d nginx

成果检查

通过svc映射的node ip，通过浏览器进行访问，图片由于url过期没有显示出来，但可以看到相关栏目都可以正常展示

 

 通过以上操作，我们熟悉了通过k8s集群来部署一套相对复杂的电商网站，且这也更接近于工作的实际内容。

后续，我们还将通过jenkins进行CICD持续化集成和部署，使工作更加自动化。