中间我还尝试过用esrally进行性能测试,用的http_logs这个track,写入速度大概是15w.用graylog集群写入速度2w.中间这差距也不知道是否正常.但是http_logs的每条数据量都很小,而且字段很少.把graylog index导出来的数据拿去做esrally性能测试,但是死活有问题.那暂时也搁置,将来有机会再去解决.
随便贴两条http_logs和graylog写入的数据.
一个字符数约150 ,一个约1400,同样一条消息,字符量相差10倍,这么看起来差不多是极限了.观察es节点服务器,貌似是io性能快到极限了.
{"@timestamp": 895435223, "clientip":"26.105.4.0", "request": "GET /english/competition/format_of_tournament.htm HTTP/1.0", "status": 200, "size": 7893}
{"fields_is_tomcat":true,"agent_id":"298d905e-b9c4-41a1-831a-bb3094bdac59","agent_name":"ZJHZ-CMREAD-MGWX02-VINT-SD","agent_hostname":"ZJHZ-CMREAD-MGWX02-VINT-SD","collector_node_id":"ZJHZ-CMREAD-MGWX02-VINT-SD","log_offset":104283920,"g
l2_remote_ip":"10.158.56.50","@metadata_version":"7.8.0","gl2_remote_port":16569,"source":"ZJHZ-CMREAD-MGWX02-VINT-SD","beats_type":"filebeat","gl2_source_input":"5f06d66152e46a698ce1a3ca","ecs_version":"1.5.0","is_wxw":"true","@metadata
_beat":"filebeat","fields_is_war":true,"@metadata_type":"_doc","log_file_path":"/home/commonservice/data/tomcat73/logs/migu/migu_info.log","gl2_source_node":"10.158.56.49dafba78c-441d-4afa-a16d-bfdff9cbe720","fields_tomcat_app":"wenxuein
terface","timestamp":"2020-07-31 08:42:08.560","gl2_accounted_message_size":882,"gl2_source_collector":"97d046b6-1152-4b55-9664-778350724767","agent_ephemeral_id":"2a8ef2d1-a998-42b3-8cfc-7d7b7d45076e","streams":["00000000000000000000000
1","5f20d61fbdc0f6519b77064c"],"input_type":"log","gl2_message_id":"01EEJ0JA86X8ZPJFT814F031QW","message":"2020-07-31 16\\:42\\:08 [ http-nio-8883-exec-1\\:59251132 ] - [ ERROR ] post url=htxxtp://172.16.2.221:7280/csu/portalengine/getUs
erInfo;httxxp result code =200","tags":["tomcat","war","wxw"],"fields_is_wxw":true,"agent_type":"filebeat","agent_version":"7.8.0","@timestamp":"2020-07-31T08:42:08.560Z","host_name":"ZJHZ-CMREAD-MGWX02-VINT-SD"}