filebeat-elasticsearch(docker)+kibana 7.6.2(docker)日志采集并展现（centos7）
一、docker安装elasticsearch+kibana
1）下载ealastic search和kibana
docker pull elasticsearch:7.6.2
docker pull kibana:7.6.2
（2）配置
mkdir -p /mydata/elasticsearch/config
mkdir -p /mydata/elasticsearch/data
echo "http.host: 0.0.0.0" >/mydata/elasticsearch/config/elasticsearch.yml
chmod -R 777 /mydata/elasticsearch/
（3）启动elasticsearch
elasticsearch：
docker run --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms64m -Xmx256m" -v /mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /mydata/elasticsearch/data:/usr/share/elasticsearch/data -v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins -d elasticsearch:7.6.2
（4）启动 kibana
kibana：
docker run --name kibana -e ELASTICSEARCH_HOSTS=http://172.17.127.219:9200 -p 5601:5601 -d kibana:7.6.2
（5）汉化 kibana
Kibana中文汉化支持
Kibana从6.6.0版本开始支持中文
参考：https://github.com/anbai-inc/Kibana_Hanization
汉化方法如下：
以现行最新版本7.2.0为例，测试机器为Windows 10
打开\kibana-7.2.0-windows-x86_64\config\kibana.yml
找到i18n.locale，如果没找到自行添加如下文本
Copy
i18n.locale: "zh-CN"
启动Kibana，应该就可以查看到中文界面。如果没有正确汉化，请找到目录\kibana-7.2.0-windows-x86_64\x-pack\plugins\translations\translations，该目录下应存在汉化包，一个Json文件 zh-CN.json
二、安装filebeat
下载tar版本
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-linux-x86_64.tar.gz
tar xzvf filebeat-7.6.2-linux-x86_64.tar.gz
[root@localhost filebeat]# ls
filebeat-7.6.2-linux-x86_64 filebeat-7.6.2-linux-x86_64.tar.gz
[root@localhost filebeat]#
三、修改filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /root/filebeat/filebeat-7.6.2-linux-x86_64/apache-daily-access.log
fields_under_root: true
multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.name: "filebeat-admin"
setup.template.pattern: "filebeat-admin*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
output.elasticsearch:
hosts: ["localhost:9200"]
indices:
- index: "filebeat-admin-%{+yyyy.MM.dd}"
读取日志文件apache-daily-access.log，也可以使用其它的(在解压后的filebeat-7.6.2-linux-x86_64下自行建立)：
2021-10-11 14:48:24.244 [http-nio-8889-exec-1] [DEBUG] c.f.m.mapper.ContentUGCMapper.upDateNii.debug [traceId:0a3b5f39e76129f7] [SpanId:0a3b5f39e76129f7] ==> Preparing: update nova_invitation_info info set FKey5='1',weighting=? where info.id=?
2021-10-11 14:48:24.245 [http-nio-8889-exec-1] [DEBUG] c.f.m.mapper.ContentUGCMapper.upDateNii.debug [traceId:0a3b5f39e76129f7] [SpanId:0a3b5f39e76129f7] ==> Parameters: 78.0(Double), 391(String)
2021-10-11 14:48:24.247 [ForkJoinPool.commonPool-worker-1] [DEBUG] c.f.m.mapper.ContentUGCMapper.getDzCount.debug [traceId:] [SpanId:] <== Total: 1
2021-10-11 14:48:24.247 [ForkJoinPool.commonPool-worker-1] [DEBUG] c.f.m.mapper.ContentUGCMapper.getPlCount.debug [traceId:] [SpanId:] ==> Preparing: SELECT count(info.FID) as FCommentCnt from comment_details_info info INNER JOIN comment_info comm on comm.FID = info.FCommentInfoId INNER join nova_invitation_info ni on comm.FContentId = ni.id and ni.FIsDelete=0 and ni.bus_type='1' where info.FIsDelete=0 and info.FStatus=1 and comm.FCommentType = 8500 and ni.id=?
2021-10-11 14:48:24.248 [ForkJoinPool.commonPool-worker-1] [DEBUG] c.f.m.mapper.ContentUGCMapper.getPlCount.debug [traceId:] [SpanId:] ==> Parameters: 504(String)
2021-10-11 14:48:24.248 [ForkJoinPool.commonPool-worker-2] [DEBUG] c.f.m.m.ContentUGCMapper.getPicCount.debug [traceId:] [SpanId:] <== Total: 1
2021-10-11 14:48:24.248 [ForkJoinPool.commonPool-worker-2] [DEBUG] c.f.m.mapper.ContentUGCMapper.getDzCount.debug [traceId:] [SpanId:] ==> Preparing: SELECT IFNULL(sum(dz.fCnt),0) dz from nova_invitation_like_info dz LEFT JOIN nova_invitation_info info on dz.inv_id = info.id and dz.FTouchCode != '0' where info.id=? and info.FIsDelete=0 and info.bus_type='1'
2021-10-11 14:48:24.248 [ForkJoinPool.commonPool-worker-2] [DEBUG] c.f.m.mapper.ContentUGCMapper.getDzCount.debug [traceId:] [SpanId:] ==> Parameters: 459(String)
2021-10-11 14:48:24.249 [ForkJoinPool.commonPool-worker-4] [ERROR] druid.sql.Statement.statementLogError [traceId:] [SpanId:] {conn-10006, pstmt-20224} execute error. insert into content_ugc_info(nii_id,weighting,basic_score,subject_score,characters_score,pic_score,praise_score,comment_score) VALUES (?,?,?,?,?,?,?,?)
java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '368' for key 'nii_id'
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955)
at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:372)
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3461)
at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440)
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3459)
at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440)
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3459)
at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167)
at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497)
at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:59)
at com.sun.proxy.$Proxy348.execute(Unknown Source)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
at sun.reflect.GeneratedMethodAccessor208.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
at com.sun.proxy.$Proxy346.update(Unknown Source)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197)
at sun.reflect.GeneratedMethodAccessor207.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
at com.sun.proxy.$Proxy182.update(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:294)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:67)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58)
at com.sun.proxy.$Proxy253.insertUgc(Unknown Source)
at com.faw.membercenter.service.ContentUGCService.updateOrAdd(ContentUGCService.java:298)
at com.faw.membercenter.service.ContentUGCService.upDataJf(ContentUGCService.java:189)
at com.faw.membercenter.service.ContentUGCService.lambda$getAllComputeWeight$0(ContentUGCService.java:48)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
2021-10-11 14:48:24.250 [ForkJoinPool.commonPool-worker-4] [DEBUG] c.f.m.mapper.ContentUGCMapper.upDateNii.debug [traceId:] [SpanId:] ==> Preparing: update nova_invitation_info info set FKey5='1',weighting=? where info.id=?
2021-10-11 14:48:24.250 [ForkJoinPool.commonPool-worker-4] [DEBUG] c.f.m.mapper.ContentUGCMapper.upDateNii.debug [traceId:] [SpanId:] ==> Parameters: 78.0(Double), 368(String)
2021-10-11 14:48:24.277 [ForkJoinPool.commonPool-worker-6] [DEBUG] c.f.m.mapper.ContentUGCMapper.getPlCount.debug [traceId:] [SpanId:] <== Total: 1
2021-10-11 14:48:24.277 [ForkJoinPool.commonPool-worker-7] [ERROR] druid.sql.Statement.statementLogError [traceId:] [SpanId:] {conn-10005, pstmt-20229} execute error. insert into content_ugc_info(nii_id,weighting,basic_score,subject_score,characters_score,pic_score,praise_score,comment_score) VALUES (?,?,?,?,?,?,?,?)
java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '42' for key 'nii_id'
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955)
at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:372)
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3461)
at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440)
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3459)
at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440)
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3459)
at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167)
at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497)
at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:59)
at com.sun.proxy.$Proxy348.execute(Unknown Source)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
at sun.reflect.GeneratedMethodAccessor208.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
at com.sun.proxy.$Proxy346.update(Unknown Source)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197)
at sun.reflect.GeneratedMethodAccessor207.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
at com.sun.proxy.$Proxy182.update(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:294)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:67)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58)
at com.sun.proxy.$Proxy253.insertUgc(Unknown Source)
at com.faw.membercenter.service.ContentUGCService.updateOrAdd(ContentUGCService.java:298)
at com.faw.membercenter.service.ContentUGCService.upDataJf(ContentUGCService.java:189)
at com.faw.membercenter.service.ContentUGCService.lambda$getAllComputeWeight$0(ContentUGCService.java:48)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
2021-10-11 14:48:24.278 [ForkJoinPool.commonPool-worker-6] [DEBUG] c.f.m.mapper.ContentUGCMapper.insertUgc.debug [traceId:] [SpanId:] ==> Preparing: insert into content_ugc_info(nii_id,weighting,basic_score,subject_score,characters_score,pic_score,praise_score,comment_score) VALUES (?,?,?,?,?,?,?,?)
catalog.CatalogConsulClient.getCatalogServices(CatalogConsulClient.java:141)
四、运行filebeat
[root@localhost filebeat-7.6.2-linux-x86_64]# ./filebeat -e
五、在kibana中查看数据：
1.建立索引视图：找到filebeat中配置的filebeat-admin*，点next step完成，点击discover查看收集到的日志信息
5026
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
