日志收集和展示

1. filebeat部署

rpm安装部署

1.1 rpm -ivh filebeat-7.1.1-x86_64.rpm

1.2 配置filebeat.yml

其内容如下：

filebeat.inputs:

- type: log
  encoding: utf8 
  enabled: true
  paths:
    - /data/pangu-sixscheck-worker/logs/worker.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  reload.period: 10s

setup.template.settings:
  index.number_of_shards: 1

name: 10.15.163.11

output.elasticsearch:
  hosts: ["10.8.1.9:9200"]                --写入elasticsearch的设置
  username: "elastic"
  password: "sony@61"
  index: "6s-woker-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "6s-woker"                    --自定义索引需要设置此两项，顶格
setup.template.pattern: "6s-woker-*"
setup.template.overwrite: true
setup.template.enabled: true
setup.ilm.enabled: false     --使自定义索引生效

processors:
  - add_locale:
      format: abbreviation
      beat.timezone: Asia/Shanghai
  - add_host_metadata: ~
  - add_cloud_metadata: ~

migration.6_to_7.enabled: true

1.3 sudo systemctl restart filebeat

metricbeat部署和配置，修改metricbeat.yml

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
name: 10.15.163.11
fields_under_root: true

output.kafka:
    hosts: ["10.8.1.5:9092","10.8.1.6:9092","10.8.1.7:9092"]
    topic: 'metric'

processors:
  - add_locale:
      format: abbreviation
      beat.timezone: Asia/Shanghai
  - add_host_metadata: ~
  - add_cloud_metadata: ~
migration.6_to_7.enabled: true

 

2. elasticsearch部署配置

2.1 下载解压： tar zxvf elasticsearch-7.1.1-linux-x86_64.tar.gz

2.2 ln -s elasticsearch-7.1.1 elasticsearch

2.3 修改配置文件 elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

cluster.name: es-sany

# ------------------------------------ Node ------------------------------------
node.name: node-1
node.master: true   --主节点  （数据节点设置为false）
node.data: true        --数据节点

# ----------------------------------- Paths ------------------------------------
path.data: /mnt/es/data   --设置好索引数据存储路径
path.logs: /home/elsearch/es/logs

# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

# ---------------------------------- Network -----------------------------------
network.bind_host: 10.8.1.9
network.publish_host: 10.8.1.9
http.port: 9200
transport.tcp.port: 9300

# --------------------------------- Discovery ----------------------------------
discovery.seed_hosts: ["10.8.1.9"]
cluster.initial_master_nodes: ["10.8.1.9"] 
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

优化jvm，修改jvm.options

-Xms48g
-Xmx48g
-XX:-UseBiasedLocking
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:G1HeapRegionSize=128M
-XX:G1ReservePercent=25
-XX:InitiatingHeapOccupancyPercent=15
-XX:+ExplicitGCInvokesConcurrent
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:ReservedCodeCacheSize=512M
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.attach.allowAttachSelf=true
-Djdk.nio.maxCachedBufferSize=2000000
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=data
-XX:ErrorFile=logs/hs_err_pid%p.log

2.4 启动服务

bin/elasticsearch -d

2.5 设置用户名和密码

bin/elasticsearch-setup-passwords interactive

按照步骤执行

3. kibana部署和设置

3.1 下载解压 tar zxvf kibana-7.1.1-linux-x86_64.tar.gz

3.2 ln -s kibana-7.1.1 kibana

3.3 修改配置文件kibana.yml

server.port: 5601

server.host: "10.8.1.9"
server.name: "aistation-89"

elasticsearch.hosts: ["http://10.8.1.7:9200","http://10.8.1.8:9200","http://10.8.1.9:9200"] --ES集群
kibana.index: ".kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "sony@61"
elasticsearch.requestTimeout: 50000
i18n.locale: "zh-CN"    --中文展示

3.4 启动服务 

nohup bin/kibana serve &