logstash es同步 部分数据丢失

错误信息

es，查询商品数据，没有查询到条码信息。mysql中条码信息是存在的
测试同学反馈的抓包信息。商品条码(code)不存在

排查过程

查看es中的数据，数据从identifies这个对象中获取，该对象中数据为空。这条数据上有tag (_jdbcstreamingfailure)，这个有点奇怪

查看对应的说明，是同步过程中出现错误添加的默认标签(https://www.elastic.co/guide/en/logstash/7.4/plugins-filters-jdbc_streaming.html#plugins-filters-jdbc_streaming-tag_on_failure)

找到对应的logstash相关时间点的日志，发现mysql连接有异常

[2019-09-23T11:40:01,523][WARN ][logstash.filters.jdbcstreaming] Exception when executing JDBC query 
{:exception=>#<Sequel::DatabaseError: Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: 
The last packet successfully received from the server was 249,761,473 milliseconds ago. 
The last packet sent successfully to the server was 249,761,473 milliseconds ago. is longer than the server configured value of 'wait_timeout'. 
You should consider either expiring and/or testing connection validity before use in your application, 
increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.>}

---

#问题原因
网上查到的资料，分析的主要原因为MySQL断开了空闲的连接。
详情可参考https://blog.csdn.net/pandajava/article/details/41946251

查看了对应的环境设置

---

解决思路

1. mysql jdbc connection增加了autoReconnect配置 (网上资料说这个只对mysql 4.X生效，5之后的版本不生效。我们的是5.7，未单独验证)
2. 增加jdbc_validate_connection配置
   

https://www.elastic.co/guide/en/logstash/7.4/plugins-filters-jdbc_streaming.html#plugins-filters-jdbc_streaming-jdbc_validate_connection

修改配置后，观察了1d运行情况，未出现之前的错误信息