athas排查k8s的skywalking问题

• skywalking 索引id过长
  oap错误日志

2019-09-27 11:07:53,182 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker -51257996 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 598;
org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 598;
	at org.elasticsearch.action.ValidateActions.addValidationError(ValidateActions.java:26) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.action.index.IndexRequest.validate(IndexRequest.java:183) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:515) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
	at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:348) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
	at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceInsert(ElasticSearchClient.java:241) ~[library-client-6.3.0.jar:6.3.0]
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceInsert(RegisterEsDAO.java:51) ~[storage-elasticsearch-plugin-6.3.0.jar:6.3.0]
	at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:102) ~[server-core-6.3.0.jar:6.3.0]
	at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_181]
	at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:84) [server-core-6.3.0.jar:6.3.0]
	at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:35) [server-core-6.3.0.jar:6.3.0]
	at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:141) [server-core-6.3.0.jar:6.3.0]
	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:82) [apm-datacarrier-6.3.0.jar:6.3.0]
	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:53) [apm-datacarrier-6.3.0.jar:6.3.0]

• 查看pod的id

kubectl get pods -n skywalking

> kubectl  get pods -n skywalking
NAME                   READY   STATUS    RESTARTS   AGE
elasticsearch-0        1/1     Running   0          23d
oap-6dc7746876-gqxmz   1/1     Running   0          12h
ui-64d9d548dc-cvldm    1/1     Running   0          23d

• 进入容器

kubectl exec -it oap-6dc7746876-gqxmz -n skywalking --container oap – /bin/bash -c “su root”

• 下载jdk

wget --no-check-certificate --header ‘Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie’ ‘http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz’

• 在cli控制台上不能打开ftp连接，只能 wget命令下载jdk， 容器内的wget没有–no-cookies参数所以把这个参数去掉 *

• 解压jdk

tar -zxvf jdk-8u141-linux-x64.tar.gz

• 安装jdk依赖环境

wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-2.29-r0.apk && apk add glibc-2.29-r0.apk

• oap容器上的jdk是Alpine版本，做了很多简化， 而athas需要完整的jdk依赖，所以要安装jdk依赖环境 *

• 下载athas

wget https://alibaba.github.io/arthas/arthas-boot.jar

• 启动

/skywalking/jdk1.8.0_141/bin/java -jar arthas-boot.jar 1

• 进入athas控制台，观察异常信息

watch org.elasticsearch.client.RestClient$SyncResponseListener get throwExp -e -x 2
watch org.elasticsearch.action.index.IndexRequest validate ‘target’

参考:
https://alibaba.github.io/arthas/watch.html
https://yq.aliyun.com/articles/698624
https://blog.csdn.net/liumiaocn/article/details/89702529