K8S云原生系统架构优化（五）DevOps

xuyijing0103

已于 2023-10-20 11:21:52 修改

阅读量194

点赞数

分类专栏：云原生K8S系统架构优化文章标签：云原生 kubernetes 系统架构 devops 容器微服务

于 2023-10-13 15:57:43 首次发布

本文链接：https://blog.csdn.net/weixin_53439529/article/details/133804423

版权

云原生K8S系统架构优化专栏收录该内容

6 篇文章 0 订阅

订阅专栏

最近实在太忙，终于抽出时间继续之前的系统优化工作了，正好DevOps也是云原生里面非常重要但是我不太掌握的能力，算是一次很有意义的学习过程。

【摘要】

原生的K8S集群系统架构优化——DevOps优化，从头搭建 CI/CD 流水线。

【案例正文】

【背景】

公司在政务云环境部署了一个系统，同事自建的K8S集群V1.19.0，面板管理和展示用Kuboard

v2.0.5.1。系统分正式环境和测试环境，均为一个控制节点+两个工作节点，节点均为虚拟机部署在政务云公共服务域（与互联网隔离）。

https://blog.csdn.net/weixin_53439529/article/details/132565188https://blog.csdn.net/weixin_53439529/article/details/132565188

【问题】

微服务部署到云容器集群，系统有自动化部署和持续交付的基础，CI/CD如何实现？

【过程与结果】

【初步解决思路】

用流水线工具实现。比如常见的持续集成工具如：Jenkins、Travis CI、GitLab CI/CD等，本系统用GitLab自带的CI/CD工具runner。

【自动化】

其实云原生的自动化能力是一个包含很广的概念，强调的是所有过程自动化，在过程中 DevOps 只是基础，与容器、微服务、第三方组件一起使用，在降低分布式复杂性和提升迭代速度的同时，因为整体增大了软件技术栈的复杂度和组件规模，所以不可避免的带来了软件交付的复杂性，如果控制不当，应用是无法体会到云原生技术的优势的。所以逐渐又增加了一些概念，如 IaC（Infrastructure as Code）、GitOps（版本控制系统）、OAM（Open Application Model）、Kubernetes operator 和大量自动化交付工具。这写都可以应用在 CI/CD 流水线，一方面标准化企业内部的软件交付过程，另一方面在标准化的基础上进行自动化，通过配置数据自描述和面向终态的交付过程，让自动化工具理解交付目标和环境差异，实现整个软件交付和运维的自动化。

随着自动化工具的增多，某种程度上增加了大家的学习成本，至少在技术选型这一步就要耗费大量的时间。单拿出来流水线看，也包括版本控制系统如Git、SVN等；持续集成工具（CI）如Jenkins、Travis CI、GitLab CI/CD等；持续交付和部署（CD）工具如Spinnaker、ArgoCD等；配置管理工具如Ansible、Chef、Puppet等；容器化平台如Docker、Kubernetes等；容器镜像管理平台如Harbor、Nexus 等。工具可能整体定位是类似的，但是使用的场景和技术栈不同。要求一个运维人员完全掌握以上工具是不现实的，即便每种只掌握一个常用的，也是一个挑战。但是架构师应该尽量掌握原理和使用场景，针对不同的用户需求和业务场景设计最合适的架构和技术选型。

掌握之后确实有很多好处，最明显的就是提高交付效率，所谓敏捷开发快速迭代，微服务增加某个功能比传统方式快得多，流水线调通以后基本以天算发版周期。

【过程】

1、【源码】因为GitLab中的工程都是在正式环境上运行的功能，不合适在其上修改和测试，正好也想和微服务治理一块优化，所以拉了一个gitee上学习的demo，用的是SpringCloud框架，注册到Nacos配置中心，这块后面想试试用订阅的方式做自动化配置。

源码地址：

study: 学习记录，代码中实现nacos加负载均衡，超时时间 (gitee.com)https://gitee.com/xuyijing0103/study2、【本地测试】源码拉到本地用IDEA编译测试，过程中也遇到很多问题，简单记录一下（有些错误很低级，不过记一下算是个教训）：

A、IDEA克隆源码提示没有git：按提示安装git（需要装VPN用一个梯子才能访问gitee）

B、maven构建报错：Failure to find com.tencent.tsf:spring-cloud-tsf-dependencies:pom:1.23.0-Greenwich-RELEASE in https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/ was cached in the local repository, resolution will not be reattempted until the update interval of HuaweiCloudSDK has elapsed or updates are forced（开始测试用的是华为云的一个学习demo）

该依赖项是腾讯云tsf微服务管理平台Greenwich版本发布的，远程仓库中确实没有该依赖项的版本，只能换源码了。

C、换了TSF的Demo 工程，按教程重新安装maven 和配置私服，还是报错：com.tencent.tsf:tsf-demo:pom:1.29.4-Finchley-RELEASE was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced

这个报错还是依赖不存在，但是不应该，因为这是腾讯云官网给的教程。微服务平台 TSF 下载 Maven-开发指南-文档中心-腾讯云 (tencent.com.cn)

微服务平台 TSF Demo 工程概述-开发指南-文档中心-腾讯云 (tencent.com.cn) 依次mvn clean install -U mvn dependency:purge-local-repository mvn clean install 强制更新所有依赖和本地仓库缓存，还是报错；删掉本地缓存的镜像.m2/repository 也没用；最后找到原因是maven的配置文件C:\Program Files\Apache Maven\settings.conf没有override。

D、运行java报错：java: java.lang.ExceptionInInitializerErrorcom.sun.tools.javac.code.TypeTags

com.sun.tools.javac.code.TypeTags 实际上是 JDK 内部的一个类，它主要用于编译器的内部使用，引起异常是因为编译器的JDK版本过高，在 IntelliJ IDEA 的 “Project Structure” 中更改项目 SDK 为较低的 JDK 版本即可。

E、在运行代码连接Nacos过程中报错[.naming.updater] c.a.nacos.client.security.SecurityProxy : [SecurityProxy] login http request failed url: http://127.0.0.1:8848/nacos/v1/auth/users/login, params: {username=nacos}, bodyMap: {password=nacos}, errorMsg: errCode: 100, errMsg: Nacos serialize for class [com.alibaba.nacos.common.http.HttpRestResult] failed。

这个错误信息表示在登录 Nacos 服务器时发生了问题。具体的错误原因是 Nacos 在序列化 com.alibaba.nacos.common.http.HttpRestResult 类时出现了失败。可能是Nacos 版本不兼容，版本说明 · alibaba/spring-cloud-alibaba Wiki · GitHub查看对应的兼容版本。

所有错误解决后，是一个注册到nacos配置中心的服务，并且用ribbon实现负载均衡，本来是想在此基础上增加订阅以自动化更新配置，增加熔断限流重试微服务治理，但是出错了，后面再优化。

3、【流水线发布】本地测试通过的工程用流水线部署到集群。GitLab上有其他同事的工程在跑流水线，所以很多配置是现成的，但还是出现了很多错，记录一下：

A、【源码推送】本地测试通过的工程先推送到gitee，然后从git拉到gitlab里面跑流水线，push到gitee报错error: failed to push some refs to 'https://gitee.com/xuyijing0103/study' hint: Updates were rejected because the remote contains work that you do not

这个错误提示表明，进行 push 操作时，Gitee 远程仓库中已经存在提交记录。需要先进行更新操作，将远程仓库中的最新提交记录拉取到本地，然后合并到本地分支，最后再进行 push 操作。

git pull origin <branch-name> 远程仓库的最新提交记录拉取到本地仓库，如果拉取操作出现了冲突，按提示手动解决冲突，然后执行git add <conflicted-file>标记冲突已解决，或者git commit -m "Resolve conflicts"直接提交修改记录。最后执行 push 操作将本地的提交记录推送到远程仓库：git push origin <branch-name>。

B、git pull报错git pull study master From https://gitee.com/xuyijing0103/study * branch master -> FETCH_HEAD fatal: refusing to merge unrelated histories

这是因为本地仓库和远程仓库之间的历史记录不相关，添加参数即可：git pull --allow-unrelated-histories origin master

C、实现一个工程的流水线需要哪些基础文档？

gitlab-ci.yml是最关键的，定义了流水线的stage和不同阶段的操作，Dockerfile是容器化文件，build镜像使用。ci_settings.xml相当于本地的maven配置文件，要用来配置镜像的maven环境，而pom.xml是原来工程中就配置好的，如果是前端工程可能还需要很多前端的配置文件。

.gitlab-ci.yml模板：

# 指定流水线运行的事件，例如每次代码提交（push）或定时触发等
trigger:
  branch: 
    - master

# 定义流水线的阶段和任务
stages:
  - build
  - test
  - deploy

# 定义构建阶段的任务
build:
  stage: build
  script:
    - echo "Build your project here"

# 定义测试阶段的任务
test:
  stage: test
  script:
    - echo "Run tests here"

# 定义部署阶段的任务
deploy:
  stage: deploy
  script:
    - echo "Deploy your project here"

.gitlab-ci.yml:

stages:
  - pre-build
  - build
  - push-to-harbor
variables:
  DOCKER_AUTH_CONFIG: |
    {
      "auths": {
        "10.210.56.4": {
           "auth": ""
        },
        "nexus-docker-group.sdibd.org:20080": {
           "auth": 
        },
        "nexus-docker-group.waterwood.ga:7443": {
           "auth": 
        },
        "nexus-docker-hosted.sdibd.org:20080": {
           "auth": 
        },
        "nexus-docker-hosted.waterwood.ga:7443": {
           "auth": 
        },
        "swr.cxy-ggfw-1.cxy.sd.cegn.cn": {
           "auth": 
        }
      }
    }
  NPM_AUTH_CONFIG: |
    //nexus.sdibd.org:20080/repository/:_authToken=NpmToken.2eda5f3e-ebb4-3887-8d44-fdd58e391cb2
  BUILDKIT_CONFIG: |
    insecure-entitlements = ["network.host", "security.insecure"]
    [registry."docker.io"]
      mirrors = [
        "nexus-docker-group.sdibd.org:20080",
        "hub-mirror.c.163.com",
        "mirror.baidubce.com"
      ]
    [registry."10.210.56.4"]
      insecure = true
  NEXUS_PUBLIC_URL: 'https://nexus.sdibd.org:20080/repository/maven-public/'
  NEXUS_PUBLIC_USER: 'admin'
  NEXUS_PUBLIC_PASS: 
  NEXUS_RELEASE_URL: 'https://nexus.sdibd.org:20080/repository/maven-releases/'
  NEXUS_RELEASE_USER: 'admin'
  NEXUS_RELEASE_PASS: 
  NEXUS_SNAPSHOT_URL: 'https://nexus.sdibd.org:20080/repository/maven-snapshots/'
  NEXUS_SNAPSHOT_USER: 'admin'
  NEXUS_SNAPSHOT_PASS: 
  BUILDER_REPOSITORY: 'nexus-docker-hosted.sdibd.org:20080/library/nacos-study-builder'
  SED_BUILDER_REPOSITORY: 'nexus-docker-hosted.sdibd.org:20080\/library\/nacos-study-builder'
  APP_TARGET: 'nacos-study'
  APP_REPOSITORY: 'nexus-docker-hosted.sdibd.org:20080/library/nacos-study'
  HARBOR_APP_REPOSITORY: '10.210.56.4/devops/nacos-study'
pre-build:
  stage: pre-build
  image:
    name: "docker"
    entrypoint:
      - ""
  variables:
  # BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
  script:
    - "env"
    - |
      mkdir -p ~/.docker && cat > ~/.docker/config.json << EOF
      $DOCKER_AUTH_CONFIG
      EOF
    - |
      mkdir -p ~/.config/buildkit && cat > ~/.config/buildkit/buildkitd.toml << EOF
      $BUILDKIT_CONFIG
      EOF
    - |
      mkdir -p /etc/buildkit/ && cat > /etc/buildkit/buildkitd.toml << EOF
      $BUILDKIT_CONFIG
      EOF
    - docker buildx create --driver=remote --name=remote-builder --use tcp://10.249.37.3:1234
    - "cat ~/.docker/config.json"
    - "cat ~/.config/buildkit/buildkitd.toml"
    - "cat /etc/buildkit/buildkitd.toml"
    - "cat /etc/hosts"
    - sed "s/#{BUILDER_REPOSITORY}/${SED_BUILDER_REPOSITORY}:$CI_COMMIT_SHA/g" Dockerfile > Dockerfile.changed
    - cat Dockerfile.changed
    - cat ci_settings.xml
    - cat pom.xml
    - |
      docker buildx build \
      -f Dockerfile.changed \
      --target=builder \
      --platform=linux/arm64 \
      --add-host=nexus.sdibd.org:10.249.37.3 \
      --build-arg=BUILDER_REPOSITORY=${BUILDER_REPOSITORY}:$CI_COMMIT_SHA \
      --build-arg=NEXUS_PUBLIC_URL=$NEXUS_PUBLIC_URL \
      --build-arg=NEXUS_PUBLIC_USER=$NEXUS_PUBLIC_USER \
      --build-arg=NEXUS_PUBLIC_PASS=$NEXUS_PUBLIC_PASS \
      --build-arg=NEXUS_RELEASE_URL=$NEXUS_RELEASE_URL \
      --build-arg=NEXUS_RELEASE_USER=$NEXUS_RELEASE_USER \
      --build-arg=NEXUS_RELEASE_PASS=$NEXUS_RELEASE_PASS \
      --build-arg=NEXUS_SNAPSHOT_URL=$NEXUS_SNAPSHOT_URL \
      --build-arg=NEXUS_SNAPSHOT_USER=$NEXUS_SNAPSHOT_USER \
      --build-arg=NEXUS_SNAPSHOT_PASS=$NEXUS_SNAPSHOT_PASS \
      --build-arg=HTTPS_PROXY=http://10.249.37.3:1091 \
      --build-arg=HTTP_PROXY=http://10.249.37.3:1091 \
      --build-arg=NO_PROXY=localhost,127.0.0.0/8,10.249.37.0/24,10.96.0.0/12,10.85.0.0/16,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local,gitlab.sdibd.org,nexus-docker-hosted.waterwood.ga,nexus-docker-group.waterwood.ga,nexus-docker-hosted.sdibd.org,nexus-docker-group.sdibd.org \
      --tag=${BUILDER_REPOSITORY}:$CI_COMMIT_SHA \
      --tag=${BUILDER_REPOSITORY}:latest \
      --push .
  rules:
    - if: "$CI_COMMIT_BRANCH == 'master'"
build:
  dependencies:
    - pre-build
  stage: build
  image:
    name: "docker"
    entrypoint:
      - ""
  variables:
    HTTPS_PROXY: http://10.249.37.3:1091
    HTTP_PROXY: http://10.249.37.3:1091
    NO_PROXY: localhost,127.0.0.0/8,10.249.37.0/24,10.96.0.0/12,10.85.0.0/16,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local,gitlab.sdibd.org,nexus-docker-hosted.waterwood.ga,nexus-docker-group.waterwood.ga,nexus-docker-hosted.sdibd.org,nexus-docker-group.sdibd.org
    # BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
  script:
    - "env"
    - |
      mkdir -p ~/.docker && cat > ~/.docker/config.json << EOF
      $DOCKER_AUTH_CONFIG
      EOF
    - |
      mkdir -p ~/.config/buildkit && cat > ~/.config/buildkit/buildkitd.toml << EOF
      $BUILDKIT_CONFIG
      EOF
    - |
      mkdir -p /etc/buildkit/ && cat > /etc/buildkit/buildkitd.toml << EOF
      $BUILDKIT_CONFIG
      EOF
    - docker context create cn-i-2 --docker "host=tcp://10.249.37.2:2375" && docker context use cn-i-2
    - docker buildx create --driver=remote --name=remote-builder --use tcp://10.249.37.3:1234
    - "cat ~/.docker/config.json"
    - "cat ~/.config/buildkit/buildkitd.toml"
    - "cat /etc/buildkit/buildkitd.toml"
    - "cat /etc/hosts"
    - sed "s/#{BUILDER_REPOSITORY}/${SED_BUILDER_REPOSITORY}:$CI_COMMIT_SHA/g" Dockerfile > Dockerfile.changed
    - cat Dockerfile.changed
    - |
      docker buildx build \
      -f Dockerfile.changed \
      --target=$APP_TARGET \
      --platform=linux/amd64,linux/arm64 \
      --add-host=nexus.sdibd.org:10.249.37.3 \
      --build-arg=BUILDER_REPOSITORY=${BUILDER_REPOSITORY}:$CI_COMMIT_SHA \
      --build-arg=NEXUS_PUBLIC_URL=$NEXUS_PUBLIC_URL \
      --build-arg=NEXUS_PUBLIC_USER=$NEXUS_PUBLIC_USER \
      --build-arg=NEXUS_PUBLIC_PASS=$NEXUS_PUBLIC_PASS \
      --build-arg=NEXUS_RELEASE_URL=$NEXUS_RELEASE_URL \
      --build-arg=NEXUS_RELEASE_USER=$NEXUS_RELEASE_USER \
      --build-arg=NEXUS_RELEASE_PASS=$NEXUS_RELEASE_PASS \
      --build-arg=NEXUS_SNAPSHOT_URL=$NEXUS_SNAPSHOT_URL \
      --build-arg=NEXUS_SNAPSHOT_USER=$NEXUS_SNAPSHOT_USER \
      --build-arg=NEXUS_SNAPSHOT_PASS=$NEXUS_SNAPSHOT_PASS \
      --tag=${APP_REPOSITORY}:$CI_COMMIT_SHA \
      --tag=${APP_REPOSITORY}:latest \
      --push .
  rules:
    - if: "$CI_COMMIT_BRANCH == 'master'"
push-to-harbor:
  dependencies:
    - build
  stage: push-to-harbor
  image:
    name: "docker"
    entrypoint:
      - ""
  variables:
    HTTPS_PROXY: http://10.249.37.3:1091
    HTTP_PROXY: http://10.249.37.3:1091
    NO_PROXY: localhost,127.0.0.0/8,10.249.37.0/24,10.96.0.0/12,10.85.0.0/16,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local,gitlab.sdibd.org,nexus-docker-hosted.waterwood.ga,nexus-docker-group.waterwood.ga,nexus-docker-hosted.sdibd.org,nexus-docker-group.sdibd.org
    # BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
  script:
    - "env"
    - |
      mkdir -p ~/.docker && cat > ~/.docker/config.json << EOF
      $DOCKER_AUTH_CONFIG
      EOF
    - |
      mkdir -p ~/.config/buildkit && cat > ~/.config/buildkit/buildkitd.toml << EOF
      $BUILDKIT_CONFIG
      EOF
    - |
      mkdir -p /etc/buildkit/ && cat > /etc/buildkit/buildkitd.toml << EOF
      $BUILDKIT_CONFIG
      EOF
    - docker context create cn-i-2 --docker "host=tcp://10.249.37.2:2375" && docker context use cn-i-2
    - docker buildx create --driver=remote --name=remote-builder --use tcp://10.249.37.3:1234
    - "cat ~/.docker/config.json"
    - "cat ~/.config/buildkit/buildkitd.toml"
    - "cat /etc/buildkit/buildkitd.toml"
    - "cat /etc/hosts"
    - sed "s/#{BUILDER_REPOSITORY}/${SED_BUILDER_REPOSITORY}:$CI_COMMIT_SHA/g" Dockerfile > Dockerfile.changed
    - cat Dockerfile.changed
    - export NOW_TIME=$(date "+%Y%m%d%H%M%S")
    - docker pull --platform=linux/amd64 ${APP_REPOSITORY}:$CI_COMMIT_SHA
    - docker tag ${APP_REPOSITORY}:$CI_COMMIT_SHA ${HARBOR_APP_REPOSITORY}:latest
    - docker push ${HARBOR_APP_REPOSITORY}:latest
  rules:
    - if: "$CI_COMMIT_BRANCH == 'master' && $PUSH_TO_HARBOR == 'true'"

Dockerfile：

# build stage
FROM maven:3-eclipse-temurin-8 AS builder
ARG NEXUS_PUBLIC_URL=http://192.168.146.128:8081/repository/maven-public/
ARG NEXUS_PUBLIC_USER=admin
ARG NEXUS_PUBLIC_PASS=

ARG NEXUS_RELEASE_URL=http://192.168.146.128:8081/repository/maven-releases/
ARG NEXUS_RELEASE_USER=admin
ARG NEXUS_RELEASE_PASS=
ARG NEXUS_SNAPSHOT_URL=http://nexus.sdibd.cn/repository/maven-snapshots/
ARG NEXUS_SNAPSHOT_USER=admin
ARG NEXUS_SNAPSHOT_PASS=

ARG MAVEN_OPT="-s ci_settings.xml \
                   -Dmaven.wagon.http.ssl.insecure=true \
                   -Dmaven.wagon.http.ssl.allowall=true \
                   -Dmaven.repo.local=/usr/src/app/maven-repository \
                   -DaltSnapshotDeploymentRepository=nexus-snapshot::default::${NEXUS_SNAPSHOT_URL} \
                   -DaltReleaseDeploymentRepository=nexus-release::default::${NEXUS_RELEASE_URL} \
                   -Dnexus.public.url=${NEXUS_PUBLIC_URL} \
                   -Dnexus.public.user=${NEXUS_PUBLIC_USER} \
                   -Dnexus.public.pass=${NEXUS_PUBLIC_PASS} \
                   -Dnexus.release.url=${NEXUS_RELEASE_URL} \
                   -Dnexus.release.user=${NEXUS_RELEASE_USER} \
                   -Dnexus.release.pass=${NEXUS_RELEASE_PASS} \
                   -Dnexus.snapshot.url=${NEXUS_SNAPSHOT_URL} \
                   -Dnexus.snapshot.user=${NEXUS_SNAPSHOT_USER} \
                   -Dnexus.snapshot.pass=${NEXUS_SNAPSHOT_PASS} "

COPY ci_settings.xml /usr/share/maven/conf/settings.xml
COPY ci_settings.xml /usr/share/maven/ref/settings.xml

WORKDIR /usr/src/app

COPY pom.xml ./
COPY ci_settings.xml ./
COPY . /usr/src/app

#RUN mvn ${MAVEN_OPT} dependency:resolve

RUN mvn ${MAVEN_OPT} install && \
    mvn ${MAVEN_OPT} deploy

FROM --platform=linux/arm64 #{BUILDER_REPOSITORY} as builder-repository

FROM eclipse-temurin:11-jre AS nacos-study
COPY --from=builder-repository /usr/src/app/*/target/*.jar application.jar

ENV SERVER_PORT 5000
EXPOSE $SERVER_PORT
ENTRYPOINT ["java", "-jar", "application.jar"]

ci_settings:

<?xml version="1.0" encoding="UTF-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

<!--
 | This is the configuration file for Maven. It can be specified at two levels:
 |
 |  1. User Level. This settings.xml file provides configuration for a single user,
 |                 and is normally provided in ${user.home}/.m2/settings.xml.
 |
 |                 NOTE: This location can be overridden with the CLI option:
 |
 |                 -s /path/to/user/settings.xml
 |
 |  2. Global Level. This settings.xml file provides configuration for all Maven
 |                 users on a machine (assuming they're all using the same Maven
 |                 installation). It's normally provided in
 |                 ${maven.conf}/settings.xml.
 |
 |                 NOTE: This location can be overridden with the CLI option:
 |
 |                 -gs /path/to/global/settings.xml
 |
 | The sections in this sample file are intended to give you a running start at
 | getting the most out of your Maven installation. Where appropriate, the default
 | values (values used when the setting is not specified) are provided.
 |
 |-->
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <!-- localRepository
   | The path to the local repository maven will use to store artifacts.
   |
   | Default: ${user.home}/.m2/repository
  <localRepository>/path/to/local/repo</localRepository>
  -->

  <!-- interactiveMode
   | This will determine whether maven prompts you when it needs input. If set to false,
   | maven will use a sensible default value, perhaps based on some other setting, for
   | the parameter in question.
   |
   | Default: true
  <interactiveMode>true</interactiveMode>
  -->

  <!-- offline
   | Determines whether maven should attempt to connect to the network when executing a build.
   | This will have an effect on artifact downloads, artifact deployment, and others.
   |
   | Default: false
  <offline>false</offline>
  -->

  <!-- pluginGroups
   | This is a list of additional group identifiers that will be searched when resolving plugins by their prefix, i.e.
   | when invoking a command line like "mvn prefix:goal". Maven will automatically add the group identifiers
   | "org.apache.maven.plugins" and "org.codehaus.mojo" if these are not already contained in the list.
   |-->
  <pluginGroups>
    <!-- pluginGroup
     | Specifies a further group identifier to use for plugin lookup.
    <pluginGroup>com.your.plugins</pluginGroup>
    -->
  </pluginGroups>

  <!-- proxies
   | This is a list of proxies which can be used on this machine to connect to the network.
   | Unless otherwise specified (by system property or command-line switch), the first proxy
   | specification in this list marked as active will be used.
   |-->
  <proxies>
    <!-- proxy
     | Specification for one proxy, to be used in connecting to the network.
     |
    <proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <username>proxyuser</username>
      <password>proxypass</password>
      <host>proxy.host.net</host>
      <port>80</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
    -->
  </proxies>

  <!-- servers
   | This is a list of authentication profiles, keyed by the server-id used within the system.
   | Authentication profiles can be used whenever maven must make a connection to a remote server.
   |-->
  <servers>
    <!-- server
     | Specifies the authentication information to use when connecting to a particular server, identified by
     | a unique name within the system (referred to by the 'id' attribute below).
     |
     | NOTE: You should either specify username/password OR privateKey/passphrase, since these pairings are
     |       used together.
     |
    <server>
      <id>deploymentRepo</id>
      <username>repouser</username>
      <password>repopwd</password>
    </server>
    -->

    <!-- Another sample, using keys to authenticate.
    <server>
      <id>siteServer</id>
      <privateKey>/path/to/private/key</privateKey>
      <passphrase>optional; leave empty if not used.</passphrase>
    </server>
    -->
    <server>
      <id>nexus-release</id>
      <username>${nexus.release.user}</username>
      <password>${nexus.release.pass}</password>
    </server>
    <server>
      <id>nexus-snapshot</id>
      <username>${nexus.snapshot.user}</username>
      <password>${nexus.snapshot.pass}</password>
    </server>
    <server>
      <id>nexus-public</id>
      <username>${nexus.public.user}</username>
      <password>${nexus.public.pass}</password>
    </server>
  </servers>

  <!-- mirrors
   | This is a list of mirrors to be used in downloading artifacts from remote repositories.
   |
   | It works like this: a POM may declare a repository to use in resolving certain artifacts.
   | However, this repository may have problems with heavy traffic at times, so people have mirrored
   | it to several places.
   |
   | That repository definition will have a unique id, so we can create a mirror reference for that
   | repository, to be used as an alternate download site. The mirror site will be the preferred
   | server for that repository.
   |-->
  <mirrors>
    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |
    <mirror>
      <id>mirrorId</id>
      <mirrorOf>repositoryId</mirrorOf>
      <name>Human Readable Name for this Mirror.</name>
      <url>http://my.repository.com/repo/path</url>
    </mirror>
     -->
    <mirror>
      <id>nexus-public</id>
      <mirrorOf>central,spring</mirrorOf>
      <name>公共仓库</name>
      <url>${nexus.public.url}</url>
    </mirror>
  </mirrors>

  <!-- profiles
   | This is a list of profiles which can be activated in a variety of ways, and which can modify
   | the build process. Profiles provided in the settings.xml are intended to provide local machine-
   | specific paths and repository locations which allow the build to work in the local environment.
   |
   | For example, if you have an integration testing plugin - like cactus - that needs to know where
   | your Tomcat instance is installed, you can provide a variable here such that the variable is
   | dereferenced during the build process to configure the cactus plugin.
   |
   | As noted above, profiles can be activated in a variety of ways. One way - the activeProfiles
   | section of this document (settings.xml) - will be discussed later. Another way essentially
   | relies on the detection of a system property, either matching a particular value for the property,
   | or merely testing its existence. Profiles can also be activated by JDK version prefix, where a
   | value of '1.4' might activate a profile when the build is executed on a JDK version of '1.4.2_07'.
   | Finally, the list of active profiles can be specified directly from the command line.
   |
   | NOTE: For profiles defined in the settings.xml, you are restricted to specifying only artifact
   |       repositories, plugin repositories, and free-form properties to be used as configuration
   |       variables for plugins in the POM.
   |
   |-->
  <profiles>
    <profile>
      <id>env-ci</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
        <altSnapshotDeploymentRepository>
          nexus-snapshot::${nexus.snapshot.url}
        </altSnapshotDeploymentRepository>
        <altReleaseDeploymentRepository>
          nexus-release::${nexus.release.url}
        </altReleaseDeploymentRepository>
      </properties>
      <repositories>
        <repository>
          <id>nexus-public</id>
          <name>nexus-public</name>
          <url>${nexus.public.url}</url>
          <layout>default</layout>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>nexus-public</id>
          <name>nexus-public</name>
          <url>${nexus.public.url}</url>
          <layout>default</layout>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </pluginRepository>
      </pluginRepositories>

    </profile>
    <!-- profile
     | Specifies a set of introductions to the build process, to be activated using one or more of the
     | mechanisms described above. For inheritance purposes, and to activate profiles via <activatedProfiles/>
     | or the command line, profiles have to have an ID that is unique.
     |
     | An encouraged best practice for profile identification is to use a consistent naming convention
     | for profiles, such as 'env-dev', 'env-test', 'env-production', 'user-jdcasey', 'user-brett', etc.
     | This will make it more intuitive to understand what the set of introduced profiles is attempting
     | to accomplish, particularly when you only have a list of profile id's for debug.
     |
     | This profile example uses the JDK version to trigger activation, and provides a JDK-specific repo.
    <profile>
      <id>jdk-1.4</id>

      <activation>
        <jdk>1.4</jdk>
      </activation>

      <repositories>
        <repository>
          <id>jdk14</id>
          <name>Repository for JDK 1.4 builds</name>
          <url>http://www.myhost.com/maven/jdk14</url>
          <layout>default</layout>
          <snapshotPolicy>always</snapshotPolicy>
        </repository>
      </repositories>
    </profile>
    -->

    <!--
     | Here is another profile, activated by the system property 'target-env' with a value of 'dev',
     | which provides a specific path to the Tomcat instance. To use this, your plugin configuration
     | might hypothetically look like:
     |
     | ...
     | <plugin>
     |   <groupId>org.myco.myplugins</groupId>
     |   <artifactId>myplugin</artifactId>
     |
     |   <configuration>
     |     <tomcatLocation>${tomcatPath}</tomcatLocation>
     |   </configuration>
     | </plugin>
     | ...
     |
     | NOTE: If you just wanted to inject this configuration whenever someone set 'target-env' to
     |       anything, you could just leave off the <value/> inside the activation-property.
     |
    <profile>
      <id>env-dev</id>

      <activation>
        <property>
          <name>target-env</name>
          <value>dev</value>
        </property>
      </activation>

      <properties>
        <tomcatPath>/path/to/tomcat/instance</tomcatPath>
      </properties>
    </profile>
    -->
  </profiles>

  <!-- activeProfiles
   | List of profiles that are active for all builds.
   |
  <activeProfiles>
    <activeProfile>alwaysActiveProfile</activeProfile>
    <activeProfile>anotherAlwaysActiveProfile</activeProfile>
  </activeProfiles>
  -->
</settings>

pom：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.11.RELEASE</version>
        <relativePath/>
    </parent>

    <groupId>org.example</groupId>
    <artifactId>nacosDemo</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>
    <modules>
        <module>nacos-user</module>
        <module>nacos-order</module>
        <module>macos-base</module>
    </modules>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <spring-boot-start-web.version>2.3.11.RELEASE</spring-boot-start-web.version>
        <spring-cloud-alibaba.version>2.2.5.RELEASE</spring-cloud-alibaba.version>
        <nacos-order.version>1.0-SNAPSHOT</nacos-order.version>
        <nacos-user.version>1.0-SNAPSHOT</nacos-user.version>
        <macos-base.version>1.0-SNAPSHOT</macos-base.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.16.10</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-web</artifactId>
                <version>${spring-boot-start-web.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>com.alibaba.cloud</groupId>
                <artifactId>spring-cloud-alibaba-dependencies</artifactId>
                <version>${spring-cloud-alibaba.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.httpcomponents</groupId>
                <artifactId>httpclient</artifactId>
                <version>4.5.13</version>
            </dependency>
            <dependency>
                <groupId>org.springframework.retry</groupId>
                <artifactId>spring-retry</artifactId>
                <version>1.3.1</version>
            </dependency>
            <dependency>
                <groupId>org.example</groupId>
                <artifactId>nacos-order</artifactId>
                <version>${nacos-order.version}</version>
            </dependency>
            <dependency>
                <groupId>org.example</groupId>
                <artifactId>nacos-user</artifactId>
                <version>${nacos-user.version}</version>
            </dependency>
            <dependency>
                <groupId>org.example</groupId>
                <artifactId>macos-base</artifactId>
                <version>${macos-base.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>


</project>

以上是最终运行通过的流水线配置，流水线有三个阶段：pre-build，build和push-to-harbor，最终把构建好的镜像推送到内网的harbor上（出于安全里面的密码都删掉）。

D、配置文件中的Nexus作用是：Nexus 是一个用于软件仓库管理的工具，和 Harbor 都是软件仓库管理工具，但是定位和功能侧重点有所不同。Nexus 更适用于软件开发和依赖管理，所以会把构建好的软件包和所有的依赖项都放在Nexus；而 Harbor 则专注于容器镜像管理和容器化环境下的安全性需求，主要用于最终镜像推送和部署到K8S集群。

E、提交文档后gitlab启动流水线报错：Pipeline will not run for the selected trigger. The rules configuration prevented any jobs from being added to the pipeline.

这是因为CI配置文件里设置了触发条件，在rules的if条件： rules: - if: "$CI_COMMIT_BRANCH == 'master' && $PUSH_TO_HARBOR == 'true'设置了两种，一种是内部变量$CI_COMMIT_BRANCH指的是提交的分支，一种是外部变量 $PUSH_TO_HARBOR是在gitlab的配置中写好的。

F、#11 [build-stage 6/8] RUN pnpm install --registry=https://nexus.sdibd.org:20080/repository/npm-group/ --no-lockfile --store-dir=/app/.pnpm-store/v3 --strict-peer-dependencies=false #11 1.356 ERR_PNPM_NO_PKG_MANIFEST No package.json found in /app #11 ERROR: executor failed running [/bin/sh -c pnpm install $INSTALL_OPT]: exit code: 1

这个报错是因为在路径 /app 中未找到 package.json 文件，从而导致了错误 ERR_PNPM_NO_PKG_MANIFEST。项目根目录下缺少 package.json 文件，这是一个前端工程用的文件，所以CI配置文件里写了很多前端的配置，需要换成后端工程用的流水线配置。

G、pre-build阶段报错：48 | mvn ${MAVEN_OPT} deploy 49 | 50 | >>> FROM --platform=linux/arm64 51 | 52 | FROM eclipse-temurin:11-jre AS nacos-study -------------------- ERROR: failed to solve: failed to solve with frontend dockerfile.v0: failed to create LLB definition: dockerfile parse error line 50: FROM requires either one or three arguments

这个错误是因为dockerfile文件FROM --platform=linux/arm64写得不对，这个指令只提供了一个参数 --platform=linux/arm64，缺少了镜像名称，需要提供镜像名称和tag

H、pre-build报错：42 | COPY ci_settings.xml ./ 43 | 44 | >>> RUN mvn ${MAVEN_OPT} dependency:resolve 45 | 46 | COPY . /usr/src/app -------------------- ERROR: failed to solve: executor failed running [/bin/sh -c mvn ${MAVEN_OPT} dependency:resolve]: exit code: 1

这个报错意思是maven的依赖项解析失败，前后换过多次ci_setting.xml文件，还是解决不了，最后发现是在mvn dependency:resolve 过程中没有找到子模块[ERROR] Child module /usr/src/app/macos-base of /usr/src/app/pom.xml does not exist。最后调整了dockerfile语句的位置，COPY . /usr/src/app后再RUN mvn ${MAVEN_OPT} dependency:resolve解决。

I、pre-build阶段报错：[ERROR] Failed to execute goal on project nacos-user: Could not resolve dependencies for project org.example:nacos-user:jar:1.0-SNAPSHOT: The following artifacts could not be resolved: org.example:macos-base:jar:1.0-SNAPSHOT (absent): Could not find artifact org.example:macos-base:jar:1.0-SNAPSHOT in nexus-public (https://nexus.sdibd.org:20080/repository/maven-public/) -> [Help 1]

这个错误通常表示 Maven 无法解析项目依赖关系中所需的一个或多个依赖项。具体是 org.example:macos-base:jar:1.0-SNAPSHOT 这个依赖项。但是该jar包是前一个阶段打包形成的工程基础jar包，不应该去maven-public仓库找，pom配置是跟本地一样的，而本地编译并没有问题。同事也遇到了这个情况，把本地的jar包推到maven的公共库解决，但是本地包本就不用推到公共库，尝试pom中指定读某个路径但是还是不行

    <parent>
        <groupId>org.example</groupId>
        <artifactId>nacosDemo</artifactId>
        <version>1.0-SNAPSHOT</version>
        <relativePath>../pom.xml</relativePath>
    </parent>

最终注释了依赖解析#RUN mvn ${MAVEN_OPT} dependency:resolve解决了

J、pre-build过程中再次出现了Fatal error compiling: java.lang.ExceptionInInitializerError: com.sun.tools.javac.code.TypeTags 错误，本地是通过IDEA的设置功能修改了JDK编译器版本解决，但是流水线写的编译yaml，怎么修改java编译器版本？

尝试yaml指定编译器版本没有用：

# 定义流水线的阶段和步骤
stages:
  - build

# 定义流水线的步骤
steps:
  - name: Build
    image: maven:3.8.4-jdk-11
    script:
      - mvn clean install -Dmaven.compiler.source=11 -Dmaven.compiler.target=11

最后发现是基础镜像如果是maven:3-eclipse-temurin-11内置了Java 编译器版本 11，改成一个低版本镜像maven:3-eclipse-temurin-8就可以了。

K、换了基础镜像版本后报了新错误：[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on project nacosDemo: Invalid syntax for repository.: Invalid syntax for repository. #12 23.10 [ERROR] Invalid syntax for alternative repository. Use "id::layout::url".

这个是仓库参数的语法格式不对，因为maven 8 版本和 11 版本的语法格式不一样，所以参数写法应该改一改，改成id::layout::url格式即可。

L、build阶段报错ERROR: failed to solve: lstat /tmp/buildkit-mount709807356/usr/src/app/target: no such file or directory

这是因为前一个阶段pre-build构建好的基础镜像builder-repository中没有对应的/usr/src/app/target/*-fat.jar包，可以看前一个阶段的日志确定目录和包名，输出Building jar: /usr/src/app/macos-base/target/macos-base-1.0-SNAPSHOT.jar说明生成的jar包应该位于路径/usr/src/app/macos-base/target/macos-base-1.0-SNAPSHOT.jar，所以语句应该调整为：

COPY --from=builder-repository /usr/src/app/*/target/*.jar /app/

M、push-to-harbor阶段报错unauthorized: unauthorized to access repository: devops/nacos-study, action: push: unauthorized to access repository: devops/nacos-study, action: push

这个错误是说明指定的harbor没有权限，原因是用户名或密码没有配置。

如果push-to-harbor这一步的地址、用户名和密码都没问题那么pre-build、build和push-to-harbor全都没问题了，就可以在harbor中看到对应的镜像了。

到这里其实只是CI没问题了，从Harbor部署镜像到集群中，并创建deployment负载还不能实现，因为两边的网络不互通，另外集群中需要装gitlab的agent才能用runner跑流水线部署，所以要先解决这两个问题才能CD。目前是把镜像打上latest标签，然后集群中用latest的image部署，设置镜像拉取策略为ifnotpresent，会自动拉取最新上传的镜像（需手动重启服务）。