在启动skywalking后,添加的服务达到一定数量的时候,发现skywalking的ui界面查不到数据。
尝试升级了skywalking的版本之后在短期内可以看到效果,但过了一段时间之后又查不到数据了。
报错日志:
解决方案:
Skywalking选择ElasticSearch存储中可能会遇到以下错误,错误代码429。或者ElasticSearch性能问题
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:9200], URI [/service_instance_inventory/type/6_tcc-app-gateway-77b98ff6ff-crblx.cards_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[elasticsearch-0][10.16.9.130:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@19a5cf02 on EsThreadPoolExecutor[name = elasticsearch-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@389297ad[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 147611]]"},"status":429}
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch
将一下的配置添加
elasticsearch.yml ,根据环境自定修改参数。
#在跟踪方案中,请考虑至少设置更多。
thread_pool.index.queue_size: 500
thread_pool.write.queue_size: 500
https://github.com/apache/skywalking/blob/v6.3.0/docs/en/FAQ/ES-Server-FAQ.md