配置中心Apollo概念、核心组件与实战

2 篇文章 0 订阅
2 篇文章 0 订阅

配置中心概念

  • 配置的种形态
    • 程序内部hardcode,这种做法为反模式,不建议
    • 配置文件比如Spring的配置文件在 application.properties
    • 环境变量,程序启动时读取
    • 启动参数
    • 基于数据库存或配置中心,储配置运行时可以灵活调配

常用配置中心

  • XDiamond: 全剧配置中心存储应用的配置项,源于阿里开源项目diamond
  • QConf: 分布式配置管理工具,可以让配置信息和程序代码分离,同时配置变化能够实时同步到客户端,而且保证用户搞笑读取配置
  • Disconf: 专注于各种分布式系统配置管理的通用组件和平台,提供统一的配置管理服务
  • SpringCloudConfig: Spring Cloud Config为分布式系统中的外部配置提供服务器和客户端支持
  • K8S ConfigMap: K8S的标准资源,专门用来集中管理应用的配置
  • Apollo: 携程框架部门开源,分布式配置中心,本身就是一套为服务并且基于SpringCloud实现

ConfigMap

  1. 将所有dubbo-demo组件的replicas数量修改为0暂停服务提供

    [root@hdss7-21 ~]# kubectl edit deployment dubbo-demo-service -napp
    [root@hdss7-21 ~]# kubectl edit deployment dubbo-demo-consumer -napp
    [root@hdss7-21 ~]# kubectl edit deployment dubbo-monitor -napp
    [root@hdss7-21 ~]# /opt/zookeeper/bin/zkServer.sh stop
    
  2. 10.4.7.11 and 10.4.7.12 zk修改为单节点

    [root@hdss7-11 ~]# cd /data/zookeeper/
    [root@hdss7-11 zookeeper]# rm -rf data/*
    [root@hdss7-11 zookeeper]# rm -rf logs/*
    [root@hdss7-11 zookeeper]# cd /opt/zookeeper/conf/
    [root@hdss7-11 conf]# vim zoo.cfg
    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/data/zookeeper/data
    dataLogDir=/data/zookeeper/logs
    clientPort=2181
    [root@hdss7-11 conf]# bash /opt/zookeeper/bin/zkServer.sh start
    [root@hdss7-11 ~]# bash /opt/zookeeper/bin/zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /opt/zookeeper/bin/../conf/zoo.cfg
    Mode: standalone
    
  3. 准备dubbo-montior的ConfigMap,将dubbo-montior的配置文件抽象成标准资源

    [root@hdss7-200 ~]# cd /data/k8s-yaml/dubbo-monitor/
    [root@hdss7-200 dubbo-monitor]# vim cm.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: dubbo-monitor-cm
      namespace: infra
    data:
      dubbo.properties: |
        dubbo.container=log4j,spring,registry,jetty
        dubbo.application.name=simple-monitor
        dubbo.application.owner=mabowen
        dubbo.registry.address=zookeeper://zk1.od.com:2181
        dubbo.protocol.port=20880
        dubbo.jetty.port=8080
        dubbo.jetty.directory=/dubbo-monitor-simple/monitor
        dubbo.charts.directory=/dubbo-monitor-simple/charts
        dubbo.statistics.directory=/dubbo-monitor-simple/statistics
        dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
        dubbo.log4j.level=WARN
    [root@hdss7-200 dubbo-monitor]# vim dp2.yaml
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: dubbo-monitor
      namespace: infra
      labels:
        name: dubbo-monitor
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: dubbo-monitor
      template:
        metadata:
          labels:
            app: dubbo-monitor
            name: dubbo-monitor
        spec:
          containers:
          - name: dubbo-monitor
            image: harbor.od.com/infra/dubbo-monitor:latest
            ports:
            - containerPort: 8080
              protocol: TCP
            - containerPort: 20880
              protocol: TCP
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - name: configmap-volume
                mountPath: /dubbo-monitor-simple/conf
          volumes:
            - name: configmap-volume
              configMap:
                name: dubbo-monitor-cm
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext:
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/cm.yaml
    [root@hdss7-21 ~]# kubectl delete -f http://k8s-yaml.od.com/dubbo-monitor/dp.yaml
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/dp2.yaml
    

    20201105175001
    20201105175025

交付Apollo

  • Apollo与CM的使用场景区别于程序本身是否有将配置文件解藕
  • Apollo是携程框架部门研发的分布式配置中心,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景
  • 服务端基于Spring Boot和Spring Cloud开发,打包后可以直接运行,不需要额外安装Tomcat等应用容器

20201105193100

ConfigService与Eureka只是逻辑上的区分,在交付时二者合为一组app

  • ConfigService提供配置的读取、推送等功能,服务Apollo客户端
  • AdminService提供配置的修改、发布等功能,服务ApolloPortal管理页面
  • ConfigService和AdminService都是多实例、无状态部署,所以需要将自己注册到Eureka中并保持心跳
  • 在Eureka智商我们架设了一层MetaServer用于封装Eureka服务发现接口
  • Client通过域名访问MetaServer获取ConfigService服务列表(IP+port),而后直接通过IP+端口访问服务,同时在Client侧会做loadbalance、错误重试
  • Portal通过域名访问MetaServer获取AdminService服务列表,而后通过IP+Port访问服务,同时在Portal侧会做loadbalance、错误重试
部署MySQL

MySQL需要5.6以上,MariaDB需要10.1以上

  1. 安装mariadb

    [root@hdss7-11 ~]# vim /etc/yum.repos.d/mariadb.repo
    [mariadb]
    name = MariaDB
    baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.2/centos7-amd64
    gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
    gpgcheck=1
    [root@hdss7-11 ~]# yum install MariaDB-server MariaDB-client -y
    [root@hdss7-11 ~]# vim /etc/my.cnf.d/server.cnf
    [mysqld]
    character-set-server = utf8mb4
    collation-server = utf8mb4_unicode_ci
    init_connect='SET NAMES utf8mb4'
    [root@hdss7-11 ~]# vim /etc/my.cnf.d/mysql-clients.cnf
    [mysql]
    default-character-set = utf8
    [root@hdss7-11 ~]# mysqladmin -uroot password
    123456
    [root@hdss7-11 ~]# mysql -uroot -p123456
    MariaDB [(none)]> \s        # 确保utf-8生效
    --------------
    mysql  Ver 15.1 Distrib 10.2.34-MariaDB, for Linux (x86_64) using readline 5.1
    
    Connection id:  11
    Current database:
    Current user:  root@localhost
    SSL:  Not in use
    Current pager:  stdout
    Using outfile:  ''
    Using delimiter:  ;
    Server:  MariaDB
    Server version:  10.2.34-MariaDB MariaDB Server
    Protocol version: 10
    Connection:  Localhost via UNIX socket
    Server characterset: utf8mb4
    Db     characterset: utf8mb4
    Client characterset: utf8
    Conn.  characterset: utf8
    UNIX socket:  /var/lib/mysql/mysql.sock
    Uptime: 4 min 43 sec
    
    Threads: 7  Questions: 6  Slow queries: 0  Opens: 17  Flush tables: 1  Open tables: 11  Queries per second avg: 0.021
    --------------
    
  2. 初始化db,脚本位置为github上 1.5.1 apollo/scripts/db/migration/configdb/V1.0.0__initialization.sql

    [root@hdss7-11 ~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql -O apolloconfig.sql
    [root@hdss7-11 ~]# mysql -uroot -p123456 < apolloconfig.sql
    [root@hdss7-11 ~]# mysql -uroot -p123456
    MariaDB [(none)]> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | ApolloConfigDB     |
    | information_schema |
    | mysql              |
    | performance_schema |
    +--------------------+
    4 rows in set (0.01 sec)
    
  3. 授权用户与修改ServerConfig的value并添加解析,这里添加IP为NodeIP是因为pod到集群外需要做NAT

    MariaDB [ApolloConfigDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to 'apolloconfig'@'10.4.7.%' identified by '123456';
    MariaDB [ApolloConfigDB]> select user,host from mysql.user;
    +--------------+-------------------+
    | user         | host              |
    +--------------+-------------------+
    | apolloconfig | 10.4.7.%          |
    | root         | 127.0.0.1         |
    | root         | ::1               |
    |              | hdss7-11.host.com |
    | root         | hdss7-11.host.com |
    |              | localhost         |
    | root         | localhost         |
    +--------------+-------------------+
    7 rows in set (0.00 sec)
    MariaDB [ApolloConfigDB]> select * from ServerConfig where Id=1\G
    *************************** 1. row ***************************
                          Id: 1
                          Key: eureka.service.url
                      Cluster: default
                        Value: http://localhost:8080/eureka/
                      Comment: Eureka服务Url,多个service以英文逗号分隔
                    IsDeleted:
        DataChange_CreatedBy: default
      DataChange_CreatedTime: 2020-11-05 21:06:34
    DataChange_LastModifiedBy:
          DataChange_LastTime: 2020-11-05 21:06:34
    1 row in set (0.00 sec)
    MariaDB [ApolloConfigDB]> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where Id=1;
    Query OK, 1 row affected (0.00 sec)
    Rows matched: 1  Changed: 1  Warnings: 0
    [root@hdss7-11 ~]# vim /var/named/od.com.zone
    
    $ORIGIN od.com.
    $TTL 600  ; 10 minutes
    @       IN SOA  dns.od.com. dnsadmin.od.com. (
                            2020101910 ; serial
                            10800      ; refresh (3 hours)
                            900        ; retry (15 minutes)
                            604800     ; expire (1 week)
                            86400      ; minimum (1 day)
                            )
                            NS   dns.od.com.
    $TTL 60 ; 1 minute
    dns                A    10.4.7.11
    harbor             A    10.4.7.200
    k8s-yaml           A    10.4.7.200
    traefik            A    10.4.7.10
    dashboard          A    10.4.7.10
    zk1                A    10.4.7.11
    zk2                A    10.4.7.12
    zk3                A    10.4.7.21
    jenkins            A    10.4.7.10
    dubbo-monitor      A    10.4.7.10
    demo               A    10.4.7.10
    config             A    10.4.7.10
    
交付Eureka与ConfigService

Eureka可以扩容并自动组成集群

  1. 修改 apollo-configservice-1.5.1 配置文件

    [root@hdss7-200 ~]# cd /opt/src/
    [root@hdss7-200 src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-configservice-1.5.1-github.zip
    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-configservice
    [root@hdss7-200 src]# unzip apollo-configservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice/
    [root@hdss7-200 src]# cd !$
    [root@hdss7-200 apollo-configservice]# cat config/app.properties
    appId=100003171                                                       # 全局唯一IP
    jdkVersion=1.8
    [root@hdss7-200 apollo-configservice]# vim config/application-github.properties
    # DataSource
    spring.datasource.url = jdbc:mysql://10.4.7.11:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = FillInCorrectPassword
    
  2. 修改 apollo-configservice-1.5.1 启动脚本

    在github apollo/scripts/apollo-on-kubernetes/apollo-config-server/scripts/startup-kubernetes.sh 中获取

    [root@hdss7-200 apollo-configservice]# cd scripts/
    [root@hdss7-200 scripts]# vim startup.sh
    #!/bin/bash
    SERVICE_NAME=apollo-configservice
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-config-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
    SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"
    
    ## Adjust memory settings if necessary
    export JAVA_OPTS="-Xms128m -Xmx128m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:SurvivorRatio=8"
    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
    
    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi
    
    if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi
    
    printf "$(date) ==== Starting ==== \n"
    
    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start
    
    rc=$?;
    
    if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi
    
    tail -f /dev/null
    
  3. 制作dockerfile apollo/scripts/apollo-on-kubernetes/apollo-config-server/Dockerfile

    [root@hdss7-200 scripts]# vim Dockerfile
    
    FROM harbor.od.com/base/jre8:8u112
    
    ENV VERSION 1.5.1
    
    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
        echo "Asia/Shanghai" > /etc/timezone
    
    ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
    ADD config/ /apollo-configservice/config
    ADD scripts/ /apollo-configservice/scripts
    
    CMD ["/apollo-configservice/scripts/startup.sh"]
    [root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
    [root@hdss7-200 apollo-configservice]# docker push harbor.od.com/infra/apollo-configservice:v1.5.1
    
  4. 资源配置清单,他提供http则需要ingress资源

    [root@hdss7-200 apollo-configservice]# cd /data/k8s-yaml/
    [root@hdss7-200 k8s-yaml]# mkdir apollo-configservice
    [root@hdss7-200 k8s-yaml]# vim dp.yaml
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-configservice
      namespace: infra
      labels:
        name: apollo-configservice
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: apollo-configservice
      template:
        metadata:
          labels:
            app: apollo-configservice
            name: apollo-configservice
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-configservice-cm
          containers:
          - name: apollo-configservice
            image: harbor.od.com/infra/apollo-configservice:v1.5.1
            ports:
            - containerPort: 8080                                   # 启动脚本里定义的8080端口
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-configservice/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext:
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600
    [root@hdss7-200 k8s-yaml]# vim cm.yaml                        # 镜像里已经定义了配置文件,但依然可以使用cm资源挂载出来
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-configservice-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://10.4.7.11:3306/ApolloConfigDB?characterEncoding=utf8
        spring.datasource.username = apolloconfig
        spring.datasource.password = 123456
        eureka.service.url = http://config.od.com/eureka
      app.properties: |
        appId=100003171
    [root@hdss7-200 k8s-yaml]# vim svc.yaml
    kind: Service
    apiVersion: v1
    metadata:
      name: apollo-configservice
      namespace: infra
    spec:
      ports:
      - protocol: TCP
        port: 8080
        targetPort: 8080
      selector:
        app: apollo-configservice
    [root@hdss7-200 k8s-yaml]# vim ingress.yaml
    kind: Ingress
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-configservice
      namespace: infra
    spec:
      rules:
      - host: config.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: apollo-configservice
          servicePort: 8080
    [root@hdss7-200 k8s-yaml]# mv *.yaml apollo-configservice/
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/cm.yaml
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/dp.yaml
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml
    
AdminService
  1. 下载与配置脚本

    [root@hdss7-200 ~]# cd /opt/src/
    [root@hdss7-200 src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-adminservice-1.5.1-github.zip
    [root@hdss7-200 src]# mkdir /data/dockerfile/
    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-adminservice ; unzip apollo-adminservice-1.5.1-github.zip  -d /data/dockerfile/apollo-adminservice
    [root@hdss7-200 src]# cd !$
    [root@hdss7-200 apollo-adminservice]# cat config/app.properties
    appId=100003172                               # 不能与其他组件重复
    [root@hdss7-200 apollo-adminservice]# echo >  scripts/startup.sh ; vim scripts/startup.sh
    #!/bin/bash
    APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
    SERVICE_NAME=apollo-adminservice
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-admin-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    # SERVER_URL="http://localhost:${SERVER_PORT}"
    SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"
    
    ## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
    
    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
    
    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
    
    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi
    
    if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi
    
    printf "$(date) ==== Starting ==== \n"
    
    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start
    
    rc=$?;
    
    if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi
    
    tail -f /dev/null
    
  2. 制作Docker镜像

    [root@hdss7-200 apollo-adminservice]# vim Dockerfile
    FROM stanleyws/jre8:8u112
    
    ENV VERSION 1.5.1
    
    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
        echo "Asia/Shanghai" > /etc/timezone
    
    ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
    ADD config/ /apollo-adminservice/config
    ADD scripts/ /apollo-adminservice/scripts
    
    CMD ["/apollo-adminservice/scripts/startup.sh"]
    [root@hdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1
    [root@hdss7-200 apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.5.1
    
  3. 配置清单,没有提供http所以不需要svc和ingress,admin只服务portal

    [root@hdss7-200 ~]# cd /data/k8s-yaml/ ; mkdir apollo-adminservice
    [root@hdss7-200 apollo-adminservice]# vim cm.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-adminservice-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://10.4.7.11:3306/ApolloConfigDB?characterEncoding=utf8
        spring.datasource.username = apolloconfig
        spring.datasource.password = 123456
        eureka.service.url = http://config.od.com/eureka
      app.properties: |
        appId=100003172
    [root@hdss7-200 apollo-adminservice]# vim dp.yaml
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-adminservice
      namespace: infra
      labels:
        name: apollo-adminservice
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: apollo-adminservice
      template:
        metadata:
          labels:
            app: apollo-adminservice
            name: apollo-adminservice
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-adminservice-cm
          containers:
          - name: apollo-adminservice
            image: harbor.od.com/infra/apollo-adminservice:v1.5.1
            ports:
            - containerPort: 8080
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-adminservice/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext:
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600
    
  4. 健康检查

    [root@hdss7-21 ~]# kubectl get pod -A -owide|grep admin
    infra         apollo-adminservice-5cccf97c64-lvpql   1/1     Running   0          2m21s   172.7.21.4   hdss7-21.host.com   <none>           <none>
    [root@hdss7-21 ~]# curl 172.7.21.4:8080/info
    {"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}
    
JVM调优简述

JVM监控是一个锯齿状,从低谷爬升到顶峰(Heap值)过程中也会产生GC(垃圾回收),而达到顶峰并开始骤降的现象称之为full_GC(完全垃圾回收),调优的理念则是通过调整内存大小,让full_GC出现的不要过于频繁,每次full_GC会影响吞吐量,另一个方向则是full_GC的骤降耗时尽量减少

apollo_portal
  1. 创建portaldb库 apollo/scripts/db/migration/portaldb/V1.0.0__initialization.sql

    [root@hdss7-11 ~]# mysql -uroot -p123456 < portaldb.sql
    [root@hdss7-11 ~]# mysql -uroot -p123456
    Welcome to the MariaDB monitor.  Commands end with ; or \g.
    Your MariaDB connection id is 119
    Server version: 10.2.34-MariaDB MariaDB Server
    
    Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    MariaDB [(none)]>
    MariaDB [(none)]> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | ApolloConfigDB     |
    | ApolloPortalDB     |
    | information_schema |
    | mysql              |
    | performance_schema |
    +--------------------+
    5 rows in set (0.00 sec)
    MariaDB [(none)]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to 'apolloportal'@'10.4.7.%' identified by '123456';
    MariaDB [(none)]> select user,host from mysql.user;
    +--------------+-------------------+
    | user         | host              |
    +--------------+-------------------+
    | apolloconfig | 10.4.7.%          |
    | apolloportal | 10.4.7.%          |
    | root         | 127.0.0.1         |
    | root         | ::1               |
    |              | hdss7-11.host.com |
    | root         | hdss7-11.host.com |
    |              | localhost         |
    | root         | localhost         |
    +--------------+-------------------+
    
  2. 修改db数据为分场景作准备

    MariaDB [(none)]> use ApolloConfigDB;
    MariaDB [ApolloConfigDB]> update ServerConfig set Value='[{"orgId":"ops01","orgName":"ops1"},{"orgId":"ops2","orgName":"ops2"},{"orgId":"ops3","orgName":"ops3"}]' where Id=2;
    
  3. 下载与配置meta service,meta service即portal连接的环境列表

    [root@hdss7-200 ~]# cd /opt/src/
    [root@hdss7-200 src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-portal-1.5.1-github.zip
    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-portal ; unzip apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal
    [root@hdss7-200 src]# cd !$
    [root@hdss7-200 apollo-portal]# cat config/app.properties
    appId=100003173                                   # 唯一ID
    jdkVersion=1.8
    [root@hdss7-200 src]# cd /data/dockerfile/apollo-portal
    [root@hdss7-200 apollo-portal]# vim config/apollo-env.properties
    local.meta=http://localhost:8080
    dev.meta=http://fill-in-dev-meta-server:8080        # 开发环境
    fat.meta=http://fill-in-fat-meta-server:8080        # 测试环境
    uat.meta=http://fill-in-uat-meta-server:8080        # 预发环境
    lpt.meta=${lpt_meta}                                # 性能测试
    pro.meta=http://fill-in-pro-meta-server:8080        # 生产环境
    
  4. 修改启动脚本与Dockerfile

    [root@hdss7-200 apollo-portal]# vim scripts/startup.sh
    #!/bin/bash
    SERVICE_NAME=apollo-portal
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-portal-server
    ## Adjust server port if necessary
    #SERVER_PORT=8070
    SERVER_PORT=8080
    APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:$SERVER_PORT"
    SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"
    
    ## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
    
    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
    
    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
    
    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi
    
    if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi
    
    printf "$(date) ==== Starting ==== \n"
    
    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start
    
    rc=$?;
    
    if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi
    
    tail -f /dev/null
    [root@hdss7-200 apollo-portal]# vim Dockerfile
    FROM stanleyws/jre8:8u112
    
    ENV VERSION 1.5.1
    
    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
        echo "Asia/Shanghai" > /etc/timezone
    
    ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
    ADD config/ /apollo-portal/config
    ADD scripts/ /apollo-portal/scripts
    
    CMD ["/apollo-portal/scripts/startup.sh"]
    [root@hdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.5.1
    [root@hdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.5.1
    
  5. 托管k8s并添加解析

    [root@hdss7-200 apollo-porta]# cd /data/k8s-yaml/ ; mkdir apollo-portal
    [root@hdss7-200 k8s-yaml]# cd /data/k8s-yaml/apollo-portal/
    [root@hdss7-200 apollo-portal]# vi cm.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-portal-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://10.4.7.11:3306/ApolloPortalDB?characterEncoding=utf8
        spring.datasource.username = apolloportal
        spring.datasource.password = 123456
      app.properties: |
        appId=100003173
      apollo-env.properties: |
        dev.meta=http://config.od.com
    [root@hdss7-200 apollo-portal]# vi dp.yaml
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-portal
      namespace: infra
      labels:
        name: apollo-portal
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: apollo-portal
      template:
        metadata:
          labels:
            app: apollo-portal
            name: apollo-portal
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-portal-cm
          containers:
          - name: apollo-portal
            image: harbor.od.com/infra/apollo-portal:v1.5.1
            ports:
            - containerPort: 8080
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-portal/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext:
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600
    [root@hdss7-200 apollo-portal]# vim svc.yaml
    kind: Service
    apiVersion: v1
    metadata:
      name: apollo-portal
      namespace: infra
    spec:
      ports:
      - protocol: TCP
        port: 8080
        targetPort: 8080
      selector:
        app: apollo-portal
    [root@hdss7-200 apollo-portal]# vim ingress.yaml
    kind: Ingress
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-portal
      namespace: infra
    spec:
      rules:
      - host: portal.od.com
        http:
          paths:
          - path: /
            backend:
              serviceName: apollo-portal
              servicePort: 8080
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/cm.yaml
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/dp.yaml
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml
    [root@hdss7-11 ~]# vim /var/named/od.com.zone
    
    $ORIGIN od.com.
    $TTL 600  ; 10 minutes
    @       IN SOA  dns.od.com. dnsadmin.od.com. (
                            2020101911 ; serial
                            10800      ; refresh (3 hours)
                            900        ; retry (15 minutes)
                            604800     ; expire (1 week)
                            86400      ; minimum (1 day)
                            )
                            NS   dns.od.com.
    $TTL 60 ; 1 minute
    dns                A    10.4.7.11
    harbor             A    10.4.7.200
    k8s-yaml           A    10.4.7.200
    traefik            A    10.4.7.10
    dashboard          A    10.4.7.10
    zk1                A    10.4.7.11
    zk2                A    10.4.7.12
    zk3                A    10.4.7.21
    jenkins            A    10.4.7.10
    dubbo-monitor      A    10.4.7.10
    demo               A    10.4.7.10
    config             A    10.4.7.10
    portal             A    10.4.7.10
    
  6. 访问页面,用户为apllo密码为admin,登陆后修改密码为admin123,同时在系统参数里也可以配置db表
    20201107125947

  7. 为dubbo-demo-service创建项目
    20201107130657

  8. 当代码中调用zpollo时应该是类似如下的配置 dubbo-demo-service/dubbo-server/src/main/resources/config.properties

    dubbo.registry=${dubbo.registry}
    dubbo.port=${dubbo.port}
    
  9. 在apollo中新增配置
    20201107182804
    20201107182953
    20201107183113
    20201107183203

  10. 使用apollo分支重新构建一版镜像
    在这里插入图片描述

  11. 修改资源配置清单的镜像版本号,并传环境变量来让dubbo启动时加载apollo

    ... ...
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: dubbo-demo-service
      template:
        metadata:
          labels:
            app: dubbo-demo-service
            name: dubbo-demo-service
        spec:
          containers:
          - name: dubbo-demo-service
            image: harbor.od.com/app/dubbo-damo-service:apollo_20201107_1830
            ports:
            - containerPort: 20880
              protocol: TCP
            env:
            - name: C_OPTS
              value: -Denv=dev -Dapollo.meta=http://config.od.com
    ... ...
    [root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml
    [root@hdss7-21 ~]# kubectl logs  -napp  dubbo-demo-service-ccbf9d87d-xlvfv | grep apollo
    2020-11-07 20:31:15.418  INFO 1 --- [           main] c.c.f.a.i.DefaultMetaServerProvider      : Located meta services from apollo.meta configuration: http://config.od.com!
    

    这时在portal页面的项目里能看到连接进来的实例了,如果代码中配置了apollo监听器,在页面上修改配置后则可以立刻reload
    apollo创建项目的appid需要与代码中 app.properties 中的appId完全一致

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值