iLogtail专题六：实际场景采集模拟

 公司一直使用的Filebeat进行日志采集
 由于Filebeat采集组件一些问题，现需要使用iLogtail进行代替
 现记录下iLogtail介绍和实际使用过程
 这是iLogtail系列的第六篇文章

背景：

由于平时日志采集存在堆栈信息日志情况，需要对日志进行正确分割。

前期准备：

准备日志打印程序并启动。输出日志信息如下：

2022-03-02 14:01:30.000 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值1 2022-03-02 14:01:30.000 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值2 2022-03-02 14:01:30.000 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值3 2022-03-02 14:01:30.006 ERROR 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : ERROR java.lang.ArithmeticException: / by zero         at com.ai.toptea.sysm.invilid.InvilidResource.invilidJob2(InvilidResource.java:35) ~[sysm-data-invilid-2.0.0.jar:2.0.0]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_152]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_152]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_152]         at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_152]         at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]         at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]         at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_152]         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_152]         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_152]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152] 2022-03-02 14:01:30.006 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值4 2022-03-02 14:01:30.006 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值5

采集配置：

多行日志在提交给插件部分时也需要使用 processor_split_log_regex 进行基于行首正则的切分，日期开头配置如下：

{     "detail":{         "SplitKey":"content",         "SplitRegex":"\\d+-\\d+-\\d+ \\d+:\\d+:\\d+.*"     },     "type":"processor_split_log_regex" }

 

采集到kafka中的数据，已将堆栈信息和error日志正常合并，可以满足日常采集需求。

{"Time":1646200910,"Contents":[{"Key":"content","Value":"2022-03-02 14:01:50.000 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值1\n"},{"Key":"@timestamp","Value":"@timestamp"},{"Key":"beat","Value":"beat"},{"Key":"fields","Value":"fields"},{"Key":"input_type","Value":"log"},{"Key":"offset","Value":"offset"},{"Key":"type","Value":"LOGTAIL_LOG"}]} {"Time":1646200910,"Contents":[{"Key":"content","Value":"2022-03-02 14:01:50.000 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值2\n"},{"Key":"type","Value":"LOGTAIL_LOG"},{"Key":"@timestamp","Value":"@timestamp"},{"Key":"beat","Value":"beat"},{"Key":"fields","Value":"fields"},{"Key":"input_type","Value":"log"},{"Key":"offset","Value":"offset"}]} {"Time":1646200910,"Contents":[{"Key":"content","Value":"2022-03-02 14:01:50.000 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值3\n"},{"Key":"@timestamp","Value":"@timestamp"},{"Key":"beat","Value":"beat"},{"Key":"fields","Value":"fields"},{"Key":"input_type","Value":"log"},{"Key":"offset","Value":"offset"},{"Key":"type","Value":"LOGTAIL_LOG"}]} {"Time":1646200910,"Contents":[{"Key":"content","Value":"2022-03-02 14:01:50.001 ERROR 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : ERROR\n\njava.lang.ArithmeticException: / by zero\n\tat com.ai.toptea.sysm.invilid.InvilidResource.invilidJob2(InvilidResource.java:35) ~[sysm-data-invilid-2.0.0.jar:2.0.0]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_152]\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_152]\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_152]\n\tat java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_152]\n\tat org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]\n\tat org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]\n\tat org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_152]\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_152]\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_152]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]\n\tat java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]\n\n"},{"Key":"input_type","Value":"log"},{"Key":"offset","Value":"offset"},{"Key":"type","Value":"LOGTAIL_LOG"},{"Key":"@timestamp","Value":"@timestamp"},{"Key":"beat","Value":"beat"},{"Key":"fields","Value":"fields"}]} {"Time":1646200910,"Contents":[{"Key":"content","Value":"2022-03-02 14:01:50.001 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值4\n"},{"Key":"type","Value":"LOGTAIL_LOG"},{"Key":"@timestamp","Value":"@timestamp"},{"Key":"beat","Value":"beat"},{"Key":"fields","Value":"fields"},{"Key":"input_type","Value":"log"},{"Key":"offset","Value":"offset"}]} {"Time":1646200910,"Contents":[{"Key":"content","Value":"2022-03-02 14:01:50.001 DEBUG 3049 --- [pool-5-thread-1] c.a.toptea.sysm.invilid.InvilidResource  : 输出正常值5"},{"Key":"type","Value":"LOGTAIL_LOG"},{"Key":"@timestamp","Value":"@timestamp"},{"Key":"beat","Value":"beat"},{"Key":"fields","Value":"fields"},{"Key":"input_type","Value":"log"},{"Key":"offset","Value":"offset"}]}

备注：

处理数据插件介绍可参考使用Logtail插件处理数据 - 日志服务 - 阿里云

尽量不要通过修改基础配置的方式实现，测试中修改基础设置未生效，查看源码后发现可能有所限制。