配置中心概念
- 配置的种形态
- 程序内部hardcode,这种做法为反模式,不建议
- 配置文件比如Spring的配置文件在
application.properties
中 - 环境变量,程序启动时读取
- 启动参数
- 基于数据库存或配置中心,储配置运行时可以灵活调配
常用配置中心
- XDiamond: 全剧配置中心存储应用的配置项,源于阿里开源项目diamond
- QConf: 分布式配置管理工具,可以让配置信息和程序代码分离,同时配置变化能够实时同步到客户端,而且保证用户搞笑读取配置
- Disconf: 专注于各种分布式系统配置管理的通用组件和平台,提供统一的配置管理服务
- SpringCloudConfig: Spring Cloud Config为分布式系统中的外部配置提供服务器和客户端支持
- K8S ConfigMap: K8S的标准资源,专门用来集中管理应用的配置
- Apollo: 携程框架部门开源,分布式配置中心,本身就是一套为服务并且基于SpringCloud实现
ConfigMap
-
将所有dubbo-demo组件的
replicas
数量修改为0暂停服务提供[root@hdss7-21 ~]# kubectl edit deployment dubbo-demo-service -napp [root@hdss7-21 ~]# kubectl edit deployment dubbo-demo-consumer -napp [root@hdss7-21 ~]# kubectl edit deployment dubbo-monitor -napp [root@hdss7-21 ~]# /opt/zookeeper/bin/zkServer.sh stop
-
将
10.4.7.11 and 10.4.7.12
zk修改为单节点[root@hdss7-11 ~]# cd /data/zookeeper/ [root@hdss7-11 zookeeper]# rm -rf data/* [root@hdss7-11 zookeeper]# rm -rf logs/* [root@hdss7-11 zookeeper]# cd /opt/zookeeper/conf/ [root@hdss7-11 conf]# vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/logs clientPort=2181 [root@hdss7-11 conf]# bash /opt/zookeeper/bin/zkServer.sh start [root@hdss7-11 ~]# bash /opt/zookeeper/bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/zookeeper/bin/../conf/zoo.cfg Mode: standalone
-
准备dubbo-montior的ConfigMap,将dubbo-montior的配置文件抽象成标准资源
[root@hdss7-200 ~]# cd /data/k8s-yaml/dubbo-monitor/ [root@hdss7-200 dubbo-monitor]# vim cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: dubbo-monitor-cm namespace: infra data: dubbo.properties: | dubbo.container=log4j,spring,registry,jetty dubbo.application.name=simple-monitor dubbo.application.owner=mabowen dubbo.registry.address=zookeeper://zk1.od.com:2181 dubbo.protocol.port=20880 dubbo.jetty.port=8080 dubbo.jetty.directory=/dubbo-monitor-simple/monitor dubbo.charts.directory=/dubbo-monitor-simple/charts dubbo.statistics.directory=/dubbo-monitor-simple/statistics dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log dubbo.log4j.level=WARN [root@hdss7-200 dubbo-monitor]# vim dp2.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: dubbo-monitor namespace: infra labels: name: dubbo-monitor spec: replicas: 1 selector: matchLabels: name: dubbo-monitor template: metadata: labels: app: dubbo-monitor name: dubbo-monitor spec: containers: - name: dubbo-monitor image: harbor.od.com/infra/dubbo-monitor:latest ports: - containerPort: 8080 protocol: TCP - containerPort: 20880 protocol: TCP imagePullPolicy: IfNotPresent volumeMounts: - name: configmap-volume mountPath: /dubbo-monitor-simple/conf volumes: - name: configmap-volume configMap: name: dubbo-monitor-cm imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600 [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/cm.yaml [root@hdss7-21 ~]# kubectl delete -f http://k8s-yaml.od.com/dubbo-monitor/dp.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/dp2.yaml
交付Apollo
- Apollo与CM的使用场景区别于程序本身是否有将配置文件解藕
- Apollo是携程框架部门研发的分布式配置中心,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景
- 服务端基于Spring Boot和Spring Cloud开发,打包后可以直接运行,不需要额外安装Tomcat等应用容器
ConfigService与Eureka只是逻辑上的区分,在交付时二者合为一组app
- ConfigService提供配置的读取、推送等功能,服务Apollo客户端
- AdminService提供配置的修改、发布等功能,服务ApolloPortal管理页面
- ConfigService和AdminService都是多实例、无状态部署,所以需要将自己注册到Eureka中并保持心跳
- 在Eureka智商我们架设了一层MetaServer用于封装Eureka服务发现接口
- Client通过域名访问MetaServer获取ConfigService服务列表(IP+port),而后直接通过IP+端口访问服务,同时在Client侧会做loadbalance、错误重试
- Portal通过域名访问MetaServer获取AdminService服务列表,而后通过IP+Port访问服务,同时在Portal侧会做loadbalance、错误重试
部署MySQL
MySQL需要5.6以上,MariaDB需要10.1以上
-
安装mariadb
[root@hdss7-11 ~]# vim /etc/yum.repos.d/mariadb.repo [mariadb] name = MariaDB baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.2/centos7-amd64 gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB gpgcheck=1 [root@hdss7-11 ~]# yum install MariaDB-server MariaDB-client -y [root@hdss7-11 ~]# vim /etc/my.cnf.d/server.cnf [mysqld] character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci init_connect='SET NAMES utf8mb4' [root@hdss7-11 ~]# vim /etc/my.cnf.d/mysql-clients.cnf [mysql] default-character-set = utf8 [root@hdss7-11 ~]# mysqladmin -uroot password 123456 [root@hdss7-11 ~]# mysql -uroot -p123456 MariaDB [(none)]> \s # 确保utf-8生效 -------------- mysql Ver 15.1 Distrib 10.2.34-MariaDB, for Linux (x86_64) using readline 5.1 Connection id: 11 Current database: Current user: root@localhost SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server: MariaDB Server version: 10.2.34-MariaDB MariaDB Server Protocol version: 10 Connection: Localhost via UNIX socket Server characterset: utf8mb4 Db characterset: utf8mb4 Client characterset: utf8 Conn. characterset: utf8 UNIX socket: /var/lib/mysql/mysql.sock Uptime: 4 min 43 sec Threads: 7 Questions: 6 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 11 Queries per second avg: 0.021 --------------
-
初始化db,脚本位置为github上
1.5.1 apollo/scripts/db/migration/configdb/V1.0.0__initialization.sql
[root@hdss7-11 ~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql -O apolloconfig.sql [root@hdss7-11 ~]# mysql -uroot -p123456 < apolloconfig.sql [root@hdss7-11 ~]# mysql -uroot -p123456 MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | ApolloConfigDB | | information_schema | | mysql | | performance_schema | +--------------------+ 4 rows in set (0.01 sec)
-
授权用户与修改ServerConfig的value并添加解析,这里添加IP为NodeIP是因为pod到集群外需要做NAT
MariaDB [ApolloConfigDB]> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to 'apolloconfig'@'10.4.7.%' identified by '123456'; MariaDB [ApolloConfigDB]> select user,host from mysql.user; +--------------+-------------------+ | user | host | +--------------+-------------------+ | apolloconfig | 10.4.7.% | | root | 127.0.0.1 | | root | ::1 | | | hdss7-11.host.com | | root | hdss7-11.host.com | | | localhost | | root | localhost | +--------------+-------------------+ 7 rows in set (0.00 sec) MariaDB [ApolloConfigDB]> select * from ServerConfig where Id=1\G *************************** 1. row *************************** Id: 1 Key: eureka.service.url Cluster: default Value: http://localhost:8080/eureka/ Comment: Eureka服务Url,多个service以英文逗号分隔 IsDeleted: DataChange_CreatedBy: default DataChange_CreatedTime: 2020-11-05 21:06:34 DataChange_LastModifiedBy: DataChange_LastTime: 2020-11-05 21:06:34 1 row in set (0.00 sec) MariaDB [ApolloConfigDB]> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where Id=1; Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 [root@hdss7-11 ~]# vim /var/named/od.com.zone $ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2020101910 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11 harbor A 10.4.7.200 k8s-yaml A 10.4.7.200 traefik A 10.4.7.10 dashboard A 10.4.7.10 zk1 A 10.4.7.11 zk2 A 10.4.7.12 zk3 A 10.4.7.21 jenkins A 10.4.7.10 dubbo-monitor A 10.4.7.10 demo A 10.4.7.10 config A 10.4.7.10
交付Eureka与ConfigService
Eureka可以扩容并自动组成集群
-
修改
apollo-configservice-1.5.1
配置文件[root@hdss7-200 ~]# cd /opt/src/ [root@hdss7-200 src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-configservice-1.5.1-github.zip [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-configservice [root@hdss7-200 src]# unzip apollo-configservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice/ [root@hdss7-200 src]# cd !$ [root@hdss7-200 apollo-configservice]# cat config/app.properties appId=100003171 # 全局唯一IP jdkVersion=1.8 [root@hdss7-200 apollo-configservice]# vim config/application-github.properties # DataSource spring.datasource.url = jdbc:mysql://10.4.7.11:3306/ApolloConfigDB?characterEncoding=utf8 spring.datasource.username = apolloconfig spring.datasource.password = FillInCorrectPassword
-
修改
apollo-configservice-1.5.1
启动脚本在github
apollo/scripts/apollo-on-kubernetes/apollo-config-server/scripts/startup-kubernetes.sh
中获取[root@hdss7-200 apollo-configservice]# cd scripts/ [root@hdss7-200 scripts]# vim startup.sh #!/bin/bash SERVICE_NAME=apollo-configservice ## Adjust log dir if necessary LOG_DIR=/opt/logs/apollo-config-server ## Adjust server port if necessary SERVER_PORT=8080 APOLLO_CONFIG_SERVICE_NAME=$(hostname -i) SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}" ## Adjust memory settings if necessary export JAVA_OPTS="-Xms128m -Xmx128m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:SurvivorRatio=8" ## Only uncomment the following when you are using server jvm #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks" ########### The following is the same for configservice, adminservice, portal ########### export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom" export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/" # Find Java if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then javaexe="$JAVA_HOME/bin/java" elif type -p java > /dev/null 2>&1; then javaexe=$(type -p java) elif [[ -x "/usr/bin/java" ]]; then javaexe="/usr/bin/java" else echo "Unable to find Java" exit 1 fi if [[ "$javaexe" ]]; then version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}') version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}') # now version is of format 009003 (9.3.x) if [ $version -ge 011000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" elif [ $version -ge 010000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" elif [ $version -ge 009000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" else JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC" JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails" JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M" fi fi printf "$(date) ==== Starting ==== \n" cd `dirname $0`/.. chmod 755 $SERVICE_NAME".jar" ./$SERVICE_NAME".jar" start rc=$?; if [[ $rc != 0 ]]; then echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc" exit $rc; fi tail -f /dev/null
-
制作dockerfile
apollo/scripts/apollo-on-kubernetes/apollo-config-server/Dockerfile
[root@hdss7-200 scripts]# vim Dockerfile FROM harbor.od.com/base/jre8:8u112 ENV VERSION 1.5.1 RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ echo "Asia/Shanghai" > /etc/timezone ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar ADD config/ /apollo-configservice/config ADD scripts/ /apollo-configservice/scripts CMD ["/apollo-configservice/scripts/startup.sh"] [root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1 [root@hdss7-200 apollo-configservice]# docker push harbor.od.com/infra/apollo-configservice:v1.5.1
-
资源配置清单,他提供http则需要ingress资源
[root@hdss7-200 apollo-configservice]# cd /data/k8s-yaml/ [root@hdss7-200 k8s-yaml]# mkdir apollo-configservice [root@hdss7-200 k8s-yaml]# vim dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: apollo-configservice namespace: infra labels: name: apollo-configservice spec: replicas: 1 selector: matchLabels: name: apollo-configservice template: metadata: labels: app: apollo-configservice name: apollo-configservice spec: volumes: - name: configmap-volume configMap: name: apollo-configservice-cm containers: - name: apollo-configservice image: harbor.od.com/infra/apollo-configservice:v1.5.1 ports: - containerPort: 8080 # 启动脚本里定义的8080端口 protocol: TCP volumeMounts: - name: configmap-volume mountPath: /apollo-configservice/config terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600 [root@hdss7-200 k8s-yaml]# vim cm.yaml # 镜像里已经定义了配置文件,但依然可以使用cm资源挂载出来 apiVersion: v1 kind: ConfigMap metadata: name: apollo-configservice-cm namespace: infra data: application-github.properties: | # DataSource spring.datasource.url = jdbc:mysql://10.4.7.11:3306/ApolloConfigDB?characterEncoding=utf8 spring.datasource.username = apolloconfig spring.datasource.password = 123456 eureka.service.url = http://config.od.com/eureka app.properties: | appId=100003171 [root@hdss7-200 k8s-yaml]# vim svc.yaml kind: Service apiVersion: v1 metadata: name: apollo-configservice namespace: infra spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: apollo-configservice [root@hdss7-200 k8s-yaml]# vim ingress.yaml kind: Ingress apiVersion: extensions/v1beta1 metadata: name: apollo-configservice namespace: infra spec: rules: - host: config.od.com http: paths: - path: / backend: serviceName: apollo-configservice servicePort: 8080 [root@hdss7-200 k8s-yaml]# mv *.yaml apollo-configservice/ [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/cm.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/dp.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml
AdminService
-
下载与配置脚本
[root@hdss7-200 ~]# cd /opt/src/ [root@hdss7-200 src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-adminservice-1.5.1-github.zip [root@hdss7-200 src]# mkdir /data/dockerfile/ [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-adminservice ; unzip apollo-adminservice-1.5.1-github.zip -d /data/dockerfile/apollo-adminservice [root@hdss7-200 src]# cd !$ [root@hdss7-200 apollo-adminservice]# cat config/app.properties appId=100003172 # 不能与其他组件重复 [root@hdss7-200 apollo-adminservice]# echo > scripts/startup.sh ; vim scripts/startup.sh #!/bin/bash APOLLO_ADMIN_SERVICE_NAME=$(hostname -i) SERVICE_NAME=apollo-adminservice ## Adjust log dir if necessary LOG_DIR=/opt/logs/apollo-admin-server ## Adjust server port if necessary SERVER_PORT=8080 # SERVER_URL="http://localhost:${SERVER_PORT}" SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}" ## Adjust memory settings if necessary #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8" ## Only uncomment the following when you are using server jvm #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks" ########### The following is the same for configservice, adminservice, portal ########### export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom" export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/" # Find Java if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then javaexe="$JAVA_HOME/bin/java" elif type -p java > /dev/null 2>&1; then javaexe=$(type -p java) elif [[ -x "/usr/bin/java" ]]; then javaexe="/usr/bin/java" else echo "Unable to find Java" exit 1 fi if [[ "$javaexe" ]]; then version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}') version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}') # now version is of format 009003 (9.3.x) if [ $version -ge 011000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" elif [ $version -ge 010000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" elif [ $version -ge 009000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" else JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC" JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails" JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M" fi fi printf "$(date) ==== Starting ==== \n" cd `dirname $0`/.. chmod 755 $SERVICE_NAME".jar" ./$SERVICE_NAME".jar" start rc=$?; if [[ $rc != 0 ]]; then echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc" exit $rc; fi tail -f /dev/null
-
制作Docker镜像
[root@hdss7-200 apollo-adminservice]# vim Dockerfile FROM stanleyws/jre8:8u112 ENV VERSION 1.5.1 RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ echo "Asia/Shanghai" > /etc/timezone ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar ADD config/ /apollo-adminservice/config ADD scripts/ /apollo-adminservice/scripts CMD ["/apollo-adminservice/scripts/startup.sh"] [root@hdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1 [root@hdss7-200 apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.5.1
-
配置清单,没有提供http所以不需要svc和ingress,admin只服务portal
[root@hdss7-200 ~]# cd /data/k8s-yaml/ ; mkdir apollo-adminservice [root@hdss7-200 apollo-adminservice]# vim cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: apollo-adminservice-cm namespace: infra data: application-github.properties: | # DataSource spring.datasource.url = jdbc:mysql://10.4.7.11:3306/ApolloConfigDB?characterEncoding=utf8 spring.datasource.username = apolloconfig spring.datasource.password = 123456 eureka.service.url = http://config.od.com/eureka app.properties: | appId=100003172 [root@hdss7-200 apollo-adminservice]# vim dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: apollo-adminservice namespace: infra labels: name: apollo-adminservice spec: replicas: 1 selector: matchLabels: name: apollo-adminservice template: metadata: labels: app: apollo-adminservice name: apollo-adminservice spec: volumes: - name: configmap-volume configMap: name: apollo-adminservice-cm containers: - name: apollo-adminservice image: harbor.od.com/infra/apollo-adminservice:v1.5.1 ports: - containerPort: 8080 protocol: TCP volumeMounts: - name: configmap-volume mountPath: /apollo-adminservice/config terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600
-
健康检查
[root@hdss7-21 ~]# kubectl get pod -A -owide|grep admin infra apollo-adminservice-5cccf97c64-lvpql 1/1 Running 0 2m21s 172.7.21.4 hdss7-21.host.com <none> <none> [root@hdss7-21 ~]# curl 172.7.21.4:8080/info {"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}
JVM调优简述
JVM监控是一个锯齿状,从低谷爬升到顶峰(Heap值)过程中也会产生GC(垃圾回收),而达到顶峰并开始骤降的现象称之为full_GC(完全垃圾回收),调优的理念则是通过调整内存大小,让full_GC出现的不要过于频繁,每次full_GC会影响吞吐量,另一个方向则是full_GC的骤降耗时尽量减少
apollo_portal
-
创建portaldb库
apollo/scripts/db/migration/portaldb/V1.0.0__initialization.sql
[root@hdss7-11 ~]# mysql -uroot -p123456 < portaldb.sql [root@hdss7-11 ~]# mysql -uroot -p123456 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 119 Server version: 10.2.34-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | ApolloConfigDB | | ApolloPortalDB | | information_schema | | mysql | | performance_schema | +--------------------+ 5 rows in set (0.00 sec) MariaDB [(none)]> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to 'apolloportal'@'10.4.7.%' identified by '123456'; MariaDB [(none)]> select user,host from mysql.user; +--------------+-------------------+ | user | host | +--------------+-------------------+ | apolloconfig | 10.4.7.% | | apolloportal | 10.4.7.% | | root | 127.0.0.1 | | root | ::1 | | | hdss7-11.host.com | | root | hdss7-11.host.com | | | localhost | | root | localhost | +--------------+-------------------+
-
修改db数据为分场景作准备
MariaDB [(none)]> use ApolloConfigDB; MariaDB [ApolloConfigDB]> update ServerConfig set Value='[{"orgId":"ops01","orgName":"ops1"},{"orgId":"ops2","orgName":"ops2"},{"orgId":"ops3","orgName":"ops3"}]' where Id=2;
-
下载与配置meta service,meta service即portal连接的环境列表
[root@hdss7-200 ~]# cd /opt/src/ [root@hdss7-200 src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-portal-1.5.1-github.zip [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-portal ; unzip apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal [root@hdss7-200 src]# cd !$ [root@hdss7-200 apollo-portal]# cat config/app.properties appId=100003173 # 唯一ID jdkVersion=1.8 [root@hdss7-200 src]# cd /data/dockerfile/apollo-portal [root@hdss7-200 apollo-portal]# vim config/apollo-env.properties local.meta=http://localhost:8080 dev.meta=http://fill-in-dev-meta-server:8080 # 开发环境 fat.meta=http://fill-in-fat-meta-server:8080 # 测试环境 uat.meta=http://fill-in-uat-meta-server:8080 # 预发环境 lpt.meta=${lpt_meta} # 性能测试 pro.meta=http://fill-in-pro-meta-server:8080 # 生产环境
-
修改启动脚本与Dockerfile
[root@hdss7-200 apollo-portal]# vim scripts/startup.sh #!/bin/bash SERVICE_NAME=apollo-portal ## Adjust log dir if necessary LOG_DIR=/opt/logs/apollo-portal-server ## Adjust server port if necessary #SERVER_PORT=8070 SERVER_PORT=8080 APOLLO_PORTAL_SERVICE_NAME=$(hostname -i) # SERVER_URL="http://localhost:$SERVER_PORT" SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}" ## Adjust memory settings if necessary #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8" ## Only uncomment the following when you are using server jvm #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks" ########### The following is the same for configservice, adminservice, portal ########### export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom" export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/" # Find Java if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then javaexe="$JAVA_HOME/bin/java" elif type -p java > /dev/null 2>&1; then javaexe=$(type -p java) elif [[ -x "/usr/bin/java" ]]; then javaexe="/usr/bin/java" else echo "Unable to find Java" exit 1 fi if [[ "$javaexe" ]]; then version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}') version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}') # now version is of format 009003 (9.3.x) if [ $version -ge 011000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" elif [ $version -ge 010000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" elif [ $version -ge 009000 ]; then JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace" else JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC" JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails" JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M" fi fi printf "$(date) ==== Starting ==== \n" cd `dirname $0`/.. chmod 755 $SERVICE_NAME".jar" ./$SERVICE_NAME".jar" start rc=$?; if [[ $rc != 0 ]]; then echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc" exit $rc; fi tail -f /dev/null [root@hdss7-200 apollo-portal]# vim Dockerfile FROM stanleyws/jre8:8u112 ENV VERSION 1.5.1 RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ echo "Asia/Shanghai" > /etc/timezone ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar ADD config/ /apollo-portal/config ADD scripts/ /apollo-portal/scripts CMD ["/apollo-portal/scripts/startup.sh"] [root@hdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.5.1 [root@hdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.5.1
-
托管k8s并添加解析
[root@hdss7-200 apollo-porta]# cd /data/k8s-yaml/ ; mkdir apollo-portal [root@hdss7-200 k8s-yaml]# cd /data/k8s-yaml/apollo-portal/ [root@hdss7-200 apollo-portal]# vi cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: apollo-portal-cm namespace: infra data: application-github.properties: | # DataSource spring.datasource.url = jdbc:mysql://10.4.7.11:3306/ApolloPortalDB?characterEncoding=utf8 spring.datasource.username = apolloportal spring.datasource.password = 123456 app.properties: | appId=100003173 apollo-env.properties: | dev.meta=http://config.od.com [root@hdss7-200 apollo-portal]# vi dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: apollo-portal namespace: infra labels: name: apollo-portal spec: replicas: 1 selector: matchLabels: name: apollo-portal template: metadata: labels: app: apollo-portal name: apollo-portal spec: volumes: - name: configmap-volume configMap: name: apollo-portal-cm containers: - name: apollo-portal image: harbor.od.com/infra/apollo-portal:v1.5.1 ports: - containerPort: 8080 protocol: TCP volumeMounts: - name: configmap-volume mountPath: /apollo-portal/config terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600 [root@hdss7-200 apollo-portal]# vim svc.yaml kind: Service apiVersion: v1 metadata: name: apollo-portal namespace: infra spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: apollo-portal [root@hdss7-200 apollo-portal]# vim ingress.yaml kind: Ingress apiVersion: extensions/v1beta1 metadata: name: apollo-portal namespace: infra spec: rules: - host: portal.od.com http: paths: - path: / backend: serviceName: apollo-portal servicePort: 8080 [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/cm.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/dp.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml [root@hdss7-11 ~]# vim /var/named/od.com.zone $ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2020101911 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11 harbor A 10.4.7.200 k8s-yaml A 10.4.7.200 traefik A 10.4.7.10 dashboard A 10.4.7.10 zk1 A 10.4.7.11 zk2 A 10.4.7.12 zk3 A 10.4.7.21 jenkins A 10.4.7.10 dubbo-monitor A 10.4.7.10 demo A 10.4.7.10 config A 10.4.7.10 portal A 10.4.7.10
-
访问页面,用户为apllo密码为admin,登陆后修改密码为admin123,同时在系统参数里也可以配置db表
-
为dubbo-demo-service创建项目
-
当代码中调用zpollo时应该是类似如下的配置
dubbo-demo-service/dubbo-server/src/main/resources/config.properties
dubbo.registry=${dubbo.registry} dubbo.port=${dubbo.port}
-
在apollo中新增配置
-
使用apollo分支重新构建一版镜像
-
修改资源配置清单的镜像版本号,并传环境变量来让dubbo启动时加载apollo
... ... spec: replicas: 1 selector: matchLabels: name: dubbo-demo-service template: metadata: labels: app: dubbo-demo-service name: dubbo-demo-service spec: containers: - name: dubbo-demo-service image: harbor.od.com/app/dubbo-damo-service:apollo_20201107_1830 ports: - containerPort: 20880 protocol: TCP env: - name: C_OPTS value: -Denv=dev -Dapollo.meta=http://config.od.com ... ... [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml [root@hdss7-21 ~]# kubectl logs -napp dubbo-demo-service-ccbf9d87d-xlvfv | grep apollo 2020-11-07 20:31:15.418 INFO 1 --- [ main] c.c.f.a.i.DefaultMetaServerProvider : Located meta services from apollo.meta configuration: http://config.od.com!
这时在portal页面的项目里能看到连接进来的实例了,如果代码中配置了apollo监听器,在页面上修改配置后则可以立刻reload
apollo创建项目的appid需要与代码中app.properties
中的appId完全一致