部署java项目进k8s

要加油呀

已于 2022-11-16 14:32:24 修改

阅读量1.3k

点赞数

分类专栏： kubernetes 文章标签： java redis docker

于 2022-06-17 17:14:24 首次发布

本文链接：https://blog.csdn.net/weixin_45112997/article/details/125336858

版权

kubernetes 专栏收录该内容

22 篇文章 1 订阅

订阅专栏

1、确定应用在各个环境配置文件的值

依赖的中间件地址
- 数据库相关的：地址/用户名/密码/端口/DB
- Redis相关的：地址/用户名/密码/端口/DB
- kafka相关的：地址/用户名/密码/端口/Topic
依赖的其他应用访问地址
- (推荐)k8s内部访问地址：应用service名字.命名空间.svc
- 外网域名
- (不推荐)需映射hosts的域名
- (不推荐)IP地址加端口
日志组件

2、应用需要的持久化volume挂载路径

询问开发，应用有哪些需要持久化的目录和需要与其他应用程序共享的持久化目录

日志存储目录 /var/www/storage/log

3、应用依赖的系统命令

询问开发人员应用中是否有业务代码调用了系统命令，防止Docker基础镜像中没有安装

4、应用是否添加了健康检查Endpoint

为了在k8s中通过探针监测应用的存活状态，需要开发人员在应用代码中添加/health的Endpoint URL的代码。

5、应用Session是否已改造使用Redis

为了让应用能在k8s进行多实例部署、达到高可用。需要解决多实例间的session问题。需要开发人员将session存到Redis中。

确定存到Redis中的key的前缀唯一（CACHE_PREFIX=应用名_）
相关的配置
k8s configmap env

BROADCAST_DRIVER=log
CACHE_DRIVER=redis
SESSION_DRIVER=redis
SESSION_LIFETIME=120
QUEUE_DRIVER=sync
CACHE_PREFIX=应用名_

REDIS_HOST=redis地址
REDIS_PASSWORD=*****
REDIS_PORT=6379

6、访问域名

应用是否需要外网访问域名
域名对应的IP地址是否被其他地方限制

6.1、镜像制作

基础镜像

FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive

ENV JAVA_HOME=/opt/jdk-11.0.14\
    CLASSPATH=.:/opt/jdk-11.0.14/lib \
    LANG=zh_CN.UTF-8 \
    LANGUAGE=zh_CN:zh:en_US:en \
    TZ=Asia/Shanghai
ENV PATH=$PATH:/opt/jdk-11.0.14/bin
RUN sed -i "s/archive.ubuntu.com/mirrors.ustc.edu.cn/g" /etc/apt/sources.list && \
    sed -i "s/security.ubuntu.com/mirrors.ustc.edu.cn/g" /etc/apt/sources.list && \
    apt-get update &&  apt-get -y install --no-install-recommends \
    tzdata \
    language-pack-zh-hans \
    wget \
    dumb-init && \
    wget -q jdk-11.0.14_linux-x64_bin.tar.gz -P /opt/ && \
    cd /opt/ && tar xf jdk-11.0.14_linux-x64_bin.tar.gz && \
    rm -f jdk-11.0.14_linux-x64_bin.tar.gz && \
    cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone && \
    echo "en_US.UTF-8 UTF-8" >> /var/lib/locales/supported.d/local && \
    echo "zh_CN.UTF-8 UTF-8" >> /var/lib/locales/supported.d/local && \
    echo "zh_CN.GBK GBK" >> /var/lib/locales/supported.d/local && \
    echo "zh_CN GB2312" >> /var/lib/locales/supported.d/local && \
    locale-gen

ENTRYPOINT  ["dumb-init", "--"]

maven-Dockerfile

FROM **/java11-ubuntu2004:v1

LABEL Author=""


ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Asia/Shanghai\
    M2_HOME=/opt/apache-maven-3.8.3
ENV PATH=$PATH:${M2_HOME}/bin

USER root

COPY settings.xml /root/.m2/settings.xml
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    curl \
    git \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common && \
    curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg  | apt-key add - && \
    add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu focal stable" && \
    apt update && \
    apt install -y --no-install-recommends docker-ce-cli && \
    wget http://**/apache-maven-3.8.3-bin.tar.gz && \
    tar xf apache-maven-3.8.3-bin.tar.gz && \
    mv apache-maven-3.8.3 /opt/apache-maven-3.8.3 && \
    rm -f apache-maven-3.8.3-bin.tar.gz

6.2、makefile

IMAGE_BASE_PUSH = **/zz-med
IMAGE_NAME = java11-maven-builder
IMAGE_VERSION = v1
all: build push
build:
		docker build --rm -f Dockerfile -t ${IMAGE_BASE_PUSH}/${IMAGE_NAME}:${IMAGE_VERSION} .
push:
		docker push ${IMAGE_BASE_PUSH}/${IMAGE_NAME}:${IMAGE_VERSION}

6.3、.gitlab-ci.yml

image: **/java11-maven-builder:v1

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_DAEMON: |
    {
      "registry-mirrors": ["阿里云镜像加速地址"],
      "insecure-registries": ["harbor域名地址"]
    }

services:
  - name: docker:18.09-dind
    command:
      - /bin/sh
      - -c
      - |
        mkdir -p /etc/docker || exit
        echo "$DOCKER_DAEMON" > /etc/docker/daemon.json || exit
        dockerd-entrypoint.sh || exit
stages:
  - scan-code
  - build
  - deploy

scan-code:
  stage: scan-code
  script:
    - mvn package sonar:sonar 
      -Dsonar.host.url=http://sonarqube地址 
      -Dsonar.login=$SONARSCANNER_TOKEN 
      -Dsonar.qualitygate.wait=true 
      -Dsonar.qualitygate.timeout=500 
      -Dsonar.projectName=$CI_PROJECT_NAME 
      -Dsonar.projectKey=$CI_PROJECT_NAME 
      -Dsonar.projectVersion=$CI_COMMIT_SHA 
      -Dsonar.gitlab.project_id=$CI_PROJECT_ID 
      -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHORT_SHA 
      -Dsonar.gitlab.ref_name=$CI_COMMIT_REF_NAME 
      -Dsonar.links.scm=$CI_PROJECT_URL 
      -Dsonar.links.ci=$CI_PIPELINE_URL 
      -Dsonar.sources=./src
      -Dsonar.test.inclusions=./*Test*/**
      -Dsonar.exclusions=./*Test*/**
      -Dsonar.java.binaries=target
      -DskipTests=true
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'release'
  tags:
    - gitlab-runner注册别名

#####################################################################
###                 构建阶段脚本                                     ##
#####################################################################
.k8s_build: &k8s_build
  - export RELEASE_NUM="$(date +%Y%m%d)-$(echo $CI_COMMIT_SHA | cut -c1-8)"
  - export IMAGE_NAME="hub.zz-med-stg.com/$CI_PROJECT_NAME/$APPENV:$RELEASE_NUM"
  - mvn package -DskipTests=true
  - wget -q -nv -np -r -l=1 -nd --http-user=gitlab-ci --http-password=$APP_K8S_CONFIG_CENTER -R "index.html*" -P docker http://***/$CI_PROJECT_NAME/$APPENV/
  - docker login harbor地址 -u gitlab-ci -p $HARBOR_TOKEN
  - docker build -t "***/$CI_PROJECT_NAME/$APPENV:$RELEASE_NUM" . -f docker/Dockerfile
  - docker push "***/$CI_PROJECT_NAME/$APPENV:$RELEASE_NUM"

###################################################################
###                 部署阶段脚本                                     ##
#####################################################################
.k8s_deploy: &k8s_deploy
  - export RELEASE_NUM="$(date +%Y%m%d)-$(echo $CI_COMMIT_SHA | cut -c1-8)"
  - export IMAGE_NAME="***/$CI_PROJECT_NAME/$APPENV:$RELEASE_NUM"
  - sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories  && apk update && apk add --no-cache wget
  - wget -q -nv -np -r -l=1 -nd --http-user=gitlab-ci --http-password=$APP_K8S_CONFIG_CENTER -R "index.html*" -P docker http://***/$CI_PROJECT_NAME/$APPENV/
  - chmod +x docker/config.sh && cd docker && ./config.sh ./k8s-application.tpl.yaml > k8s-application.yaml
  - cat k8s-application.yaml
  - kubectl apply -f k8s-application.yaml
  - kubectl -n $NAMESPACE rollout status --timeout=260s deployment/$CI_PROJECT_NAME || exit_code=$?
  - |
    if [ $exit_code -ne 0 ];then
      apk add curl jq --no-cache > /dev/null;
      ROLLBACK_ID=$(kubectl -n $NAMESPACE rollout undo deployment/$CI_PROJECT_NAME -ojson | jq -r '.status.observedGeneration');
      curl -s https://oapi.dingtalk.com/robot/send?access_token="$PIPELINE_DINGDING_ROBOT_TOKEN" -H 'Content-Type: application/json' -d '{"msgtype": "markdown","markdown": {"title": "Gitlab流水线部署失败","text": "['$CI_PROJECT_NAME']('$CI_PROJECT_URL'/-/tree/'$CI_BUILD_REF_NAME')的'$APPENV'环境第['$CI_PIPELINE_ID']('$CI_PIPELINE_URL')号流水线'$CI_JOB_STAGE'阶段失败，已回滚至最近一个稳定版本'$ROLLBACK_ID'，请检查相关错误！"},"at": {"isAtAll": true}}' > /dev/null;
      exit 1;
    fi

##############################################################################
#              Test environment build and deploy CI/CD                       #
##############################################################################
build-test:
  stage: build
  variables:
    APPENV: test
  script: *k8s_build
  only:
    - develop
  tags:
    - runner注册别名

deploy-test:
  stage: deploy
  image: ***/kubectl-stg镜像名称
  variables:
    NAMESPACE: test
    APPENV: test
    POD_NUM: 1
  script: *k8s_deploy
  only:
    - develop
  tags:
    - runner注册别名

##############################################################################
#              Stg environment build and deploy CI/CD                        #
##############################################################################
build-stg:
  stage: build
  variables:
    APPENV: stg
  script: *k8s_build
  only:
    - release
  tags:
    - runner注册别名

deploy-stg:
  stage: deploy
  image: ***/kubectl-stg:镜像名称
  variables:
    APPENV: stg
    NAMESPACE: stg
    POD_NUM: 1
  script: *k8s_deploy
  only:
    - release
  tags:
    - runner注册别名

##############################################################################
#              Production environment build and deploy CI/CD                 #
##############################################################################
build-prd:
  stage: build
  variables:
    APPENV: prod
  retry:
    max: 2
    when:
      - always
  script: *k8s_build
  only:
    - tags
  except:
    - release
    - develpo
  tags:
    - runner注册别名
  
deploy-prd:
  stage: deploy
  image: **/kubectl-prod:镜像名称
  variables:
    APPENV: prod
    NAMESPACE: prd
    POD_NUM: 2
  script: *k8s_deploy
  only:
    - tags
  except:
    - release
    - develop
  when: manual
  tags:
    - runner注册别名

6.4、deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: $CI_PROJECT_NAME
  namespace: $NAMESPACE
  labels:
    app: $CI_PROJECT_NAME
  annotations:
    kubernetes.io/change-cause: $IMAGE_NAME
spec:
  selector:
    matchLabels:
      app: $CI_PROJECT_NAME
  replicas: $POD_NUM
  template:
    metadata:
      labels:
        app: $CI_PROJECT_NAME
    spec:
      volumes:
        - name: log-local-storage
          hostPath:
            path: /log/app/$APPENV
      imagePullSecrets:
        - name: zzmed-registry
      containers:
        - name: app
          image: $IMAGE_NAME
          imagePullPolicy: Always
          args:
            - 'sh'
            - '-c'
            - 'java -jar -Duser.timezone=Asia/Shanghai -Xms1024M -Xmx1024M /application/app.jar'
          env:
            - name: K8s_POD_UID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
            - name: IDC
              value: $APPENV
            - name: APOLLO_CONFIGSERVICE
              value: http://apollo-service:8080
            - name: APP_ID
              value: $CI_PROJECT_NAME
            - name: APOLLO_NAMESPACE
              value: application
          volumeMounts:
            - name: log-local-storage
              mountPath: /logs
              subPath: $CI_PROJECT_NAME
          ports:
            - containerPort: 8080
              protocol: TCP
              name: app
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 60
            successThreshold: 1
            tcpSocket:
              port: 8080
            timeoutSeconds: 2
          readinessProbe:
            failureThreshold: 3
            periodSeconds: 60
            successThreshold: 1
            tcpSocket:
              port: 8080
            timeoutSeconds: 2
          resources:
            limits:
              cpu: "1000m"
              memory: 2000Mi
            requests:
              cpu: "100m"
              memory: 100Mi

6.5、应用Dockerfile

FROM **/java11-ubuntu2004:v1
COPY xx/target/*.jar /application/app.jar

6.6、config.sh

#!/bin/sh
if [ ! -n "$1" ];then
    echo "must input file name"
    exit 1
else
  cat $1 |
    sed 's/\$NAMESPACE'"/$NAMESPACE/g" |
    sed 's~\$IMAGE_NAME'"~$IMAGE_NAME~g" |
    sed 's~\$APPENV'"~$APPENV~g" |
    sed 's~\$POD_NUM'"~$POD_NUM~g" |
    sed 's~\$CI_PROJECT_NAME'"~$CI_PROJECT_NAME~g" |
    cat -
fi

exit 0

6.7、svc、ing文件

---
apiVersion: v1
kind: Service
metadata:
  name: ***-svc
  namespace: prd
spec:
  selector:
    app: ***
  ports:
    - name: app
      protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ***-ing
  namespace: prd
spec:
  rules:
  - host: ***.com
    http:
      paths:
      - path: /
        backend:
          serviceName: ***-svc
          servicePort: 80