Skywalking8.5.0-ES7 Rancher部署及ES高可用部署过程记录

背景

近期本司内部需要用到Skywalking进行链路追踪和日志收集，故记录下部署过程，方便后期学习与查询。

ElasticSearch高可用部署

##　１.docker启动elasticsearch,并将内存设置的稍微大一些,然后进行端口和配置文件映射

docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
-e ES_JAVA_OPTS="-Xms10240m -Xmx20480m" \
-v /home/mapping/elasticsearch/data:/usr/share/elasticsearch/data \
-v /home/mapping/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-v /home/mapping/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
--restart=always -d 10.156.204.60/oher/elasticsearch:7.12.0

2. 添加配置文件

elasticsearch.yml配置

NodeA
cluster.name: ClusterDev
node.master: true
node.data: true
discovery.seed_hosts: ["10.156.204.58:9300","10.156.204.59:9300",10.156.204.60:9300]
node.max_local_storage_nodes: 3
node.name: node-a
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
http.port: 9200
transport.tcp.port: 9300
http.cors.allow-headers: Authorization,Content-Type,X-Requested-with,Content-Length
xpack.security.enabled: true
action.auto_create_index: ".watches,.triggered_watches,.watcher-history-*,-markdown"

NodeB
cluster.name: ClusterDev
node.master: true
node.data: true
discovery.seed_hosts: ["10.156.204.58:9300","10.156.204.59:9300",10.156.204.60:9300]
node.max_local_storage_nodes: 3
node.name: node-b
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
http.port: 9200
transport.tcp.port: 9300
http.cors.allow-headers: Authorization,Content-Type,X-Requested-with,Content-Length
xpack.security.enabled: true
action.auto_create_index: ".watches,.triggered_watches,.watcher-history-*,-markdown"

NodeC
cluster.name: ClusterDev
node.master: true
node.data: true
discovery.seed_hosts: ["10.156.204.58:9300","10.156.204.59:9300",10.156.204.60:9300]
node.max_local_storage_nodes: 3
node.name: node-c
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
http.port: 9200
transport.tcp.port: 9300
http.cors.allow-headers: Authorization,Content-Type,X-Requested-with,Content-Length
xpack.security.enabled: true
action.auto_create_index: ".watches,.triggered_watches,.watcher-history-*,-markdown"

3.重置三台ES的密码

1. docker exec -it elasticsearch bash进入到容器内
2. 进入/usr/share/elasticsearch/bin目录
3. ./elasticsearch-setup-passwords interactive执行重置密码指令

Skywalking-OAP-Sever

本文选定8.5.0-es7版本进行演示

1. 在rancher中新建服务,镜像地址填写

apache/skywalking-oap-server:8.5.0-es7

2. 端口映射

   • 映射容器端口12800到NodePort12800
   • 映射容器端口11800到NodePort11800

3. 环境变量配置

Skywalking-UI

1. docker镜像: apache/skywalking-ui:8.5.0
2. 端口映射:8000端口到NodePort8000
3. 环境变量

Skywalking-Agent

本文采用K8s部署,K8s部署常有两种方式:第一种是直接将Agent打包到项目镜像中,然后通过-javaagent参数镜像启动;第二种是通过K8s的SideCar模式进行启动.本文采用第二种模式,方便简单.

下载apache-skywalking-apm-8.5.0.tar.gz

https://www.apache.org/dyn/closer.cgi/skywalking/8.5.0/apache-skywalking-apm-8.5.0.tar.gz

编写DockerFile打包Agent镜像

FROM busybox:latest
LABEL maintainer="wangpx02@catl-21c.com"
ENV LANG=C.UTF-8

RUN set -eux && mkdir -p /usr/skywalking/agent/
  
ADD apache-skywalking-apm-bin/agent/ /usr/skywalking/agent/
  
WORKDIR /

SideCar接入Skywalking-Agent

前提: 配置PV和PVC

PV:绑定一个持久卷,这里采用NFS的形式

PVC: 声明使用持久卷,只能与一个PV绑定,PVC只能在当前命名空间下使用

添加SideCar

1. 选择类型为Init类型,也就是在主镜像之前先启动完成

2. 选择镜像地址为Agent的Docker地址

3. 数据卷绑定上我们声明的PVC,这里容器路径填写/skywalking/agent,也就是将这个PVC挂载到SideCar镜像的/skywalking/agent目录

4. 然后配置启动命令如下,主要实现了目录创建,还有jar包拷贝到PVC对应的NFS上

   入口: /bin/sh
   命令: -c 'set -ex; mkdir -p /skywalking/agent; cp -R /usr/skywalking/agent/* /skywalking/agent;'
   

##　修改原镜像配置

1. 配置卷映射,将PVC绑定到/skywalking/agent下,这样就实现了两个镜像之间,通过NFS镜像文件共享
2. 添加环境变量如下

SW_AGENT_NAME 指明当前服务在Skywalking下显示的名字
SKYWALKING_OPTS 指定SW启动参数
SW_AGENT_COLLECTOR_BACKEND_SERVICES 指定SW_OAP_SERVER地址

修改自己的服务的DockerFile

在自己服务中添加Skywalking启动参数SKYWALKING_OPTS,或者简单粗暴,直接加上-javaagent也行

日志整合

给自己的项目添加pom依赖

// 日志依赖
<dependency>
    <groupId>org.apache.skywalking</groupId>
    <artifactId>apm-toolkit-logback-1.x</artifactId>
    <version>8.5.0</version>
</dependency>
// 追踪id
<dependency>
    <groupId>org.apache.skywalking</groupId>
    <artifactId>apm-toolkit-trace</artifactId>
    <version>8.5.0</version>
</dependency>

修改logback-spring.xml日志配置

<configuration>
    <property name="APM_PATTERN"
              value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%X{tid}] [%thread] %-5level %logger{36} -%msg%n" />
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">　　　
        <encoder>
            <!--格式化输出：%d表示日期，%thread表示线程名，%-5level：级别从左显示5个字符宽度%msg：日志消息，%n是换行符-->
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %thread %logger %level - [%tid] %msg %n
            </pattern>
        </encoder>
    </appender>
    <appender name="sw-log" class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.log.GRPCLogClientAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.mdc.TraceIdMDCPatternLogbackLayout">
                <Pattern>${APM_PATTERN}</Pattern>
            </layout>
        </encoder>
    </appender>
    <!-- root级别   INFO -->
    <root level="INFO">
        <appender-ref ref="STDOUT"/>
    </root>

    <root level="WARN">　　　　　　
        <appender-ref ref="sw-log"/>　　　
    </root>
</configuration>