docker部署ELK

已于 2022-06-09 20:46:15 修改
阅读量696 收藏 2
点赞数
分类专栏： 我的运维笔记 Docker 文章标签： ELK elasticsearch logstash kibana filebeat
于 2021-04-09 14:04:30 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wangshui898/article/details/115549227
版权

服务规划

架构: Filebeat->kafka->logstash->ES

部署服务程序路径/数据目录端口配置文件
elasticsearch/data/elasticsearch9200/data/elasticsearch/config/elasticsearch.yml
logstash/data/logstash–/data/logstash/config/logstash.yml
kibana/data/kibana5601/data/kibana/config/kibana.yml
filebeat/data/filebeat–/data/filebeat/config/filebeat.yml

索引服务-Elasticsearch

创建数据目录

mkdir -pv /data/elasticsearch/{config,data,logs}
chown 1000 /data/elasticsearch/{data,logs}

修改主机配置

vim /etc/sysctl.conf
加入
vm.max_map_count=655360
sysctl -p

vim /etc/security/limits.conf
加入
* soft memlock unlimited
* hard memlock unlimited

配置文件

cat > /data/elasticsearch/config/elasticsearch.yml << 'EOF'
cluster.name: ccms-es-cluster
node.name: ccms-es1
network.host: 172.16.20.51
http.port: 9200
bootstrap.memory_lock: true

# 允许跨域访问
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-methods: "OPTIONS, HEAD, GET, POST, PUT, DELETE"
http.cors.allow-headers: "Authorization, X-Requested-With, Content-Type, Content-Length, X-User"

# Cluster
node.master: true
node.data: true
transport.tcp.port: 9300
discovery.seed_hosts: ["172.16.20.51","172.16.20.52","172.16.20.53"]
cluster.initial_master_nodes: ["ccms-es1","ccms-es2","ccms-es3"]

cluster.routing.allocation.same_shard.host: true
cluster.routing.allocation.node_initial_primaries_recoveries: 4
cluster.routing.allocation.node_concurrent_recoveries: 4

# X-Pack
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
EOF

chown 1000 /data/elasticsearch/config/*
# 容器启动后先生成证书, 分发到各个节点的config目录下, 再重启es容器

discovery.zen.minimum_master_nodes算法: 节点数/2+1

设置ES密码:
自动设置密码命令
elasticsearch-setup-passwords auto
或者
自定义密码命令
elasticsearch-setup-passwords interactive

es-head登录
http://172.16.20.52:9200/?auth_user=elastic&auth_password=elastic123456

生成证书(证书不需要设置密码):
cd /usr/share/elasticsearch/config/
elasticsearch-certutil ca -out config/elastic-certificates.p12 -pass ""

docker-compose编排

mkdir -pv /data/docker-compose/elasticsearch/
cat > /data/docker-compose/elasticsearch/docker-compose.yml << EOF
version: "3"
services:
  es:
    container_name: es
    image: elasticsearch:7.11.1
    network_mode: host
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
      - /data/elasticsearch/config:/usr/share/elasticsearch/config
      - /data/elasticsearch/data:/usr/share/elasticsearch/data
      - /data/elasticsearch/logs:/usr/share/elasticsearch/logs
    environment:
      TZ: Asia/Shanghai
      bootstrap.memory_lock: true
      ES_JAVA_OPTS: "-Xmx8G -Xms8G"
      ELASTIC_PASSWORD: "G1T@es2022#ccms"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
           memory: 10G
EOF

1. 解决es-head跨域问题(浏览器报: Request header field Content-Type is not allowed by Access-Control-Allow-Headers)
es配置文件加入:
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-methods: "OPTIONS, HEAD, GET, POST, PUT, DELETE"
http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, X-User"

2. 解决es-head数据浏览空白(浏览器报: 406 Not Acceptable)
修改es-head代码文件vendor.js
第6886行左右
contentType: "application/x-www-form-urlencoded" --> contentType: "application/json;charset=UTF-8"

启动

docker-compose up -d

日志采集-Filebeat

创建数据目录

mkdir -pv /data/filebeat/{config,data}

配置文件

发送到kafka

cat > /data/filebeat/config/filebeat.yml << 'EOF'
###################### Filebeat Configuration Example #########################
filebeat.name: ccms-test-08
filebeat.idle_timeout: 5s
filebeat.spool_zie: 2048

#----------------------------------input form ccms servers--------------------------------#
filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /opt/ccms-auto-deploy/credit-business/*/*/target/logs/*.log
   - /opt/ccms-auto-deploy/credit-support/*/*/target/logs/*.log
  fields:
    kafka_topic: topic-ccms-dev
  fields_under_root: true

  # filebeat 多行日志的处理
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

  encoding: plain
  tail_files: false

  # 检测指定目录下文件更新时间
  scan_frequency: 3s
  # 每隔1s检测一下文件变化，如果连续检测2次之后文件还没有变化，下一次检测间隔时间变为5s
  backoff: 1s
  max_backoff: 5s
  backoff_factor: 2

#----------------------------------input form nginx access_log--------------------------------#
- type: log
  enabled: true
  paths:
   - /data/nginx/logs/ccms-access.log
  fields:
    kafka_topic: topic-nginx-access
  fields_under_root: true

  encoding: plain
  tail_files: false

  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: false

  # 检测指定目录下文件更新时间
  scan_frequency: 3s
  # 每隔1s检测一下文件变化，如果连续检测2次之后文件还没有变化，下一次检测间隔时间变为5s
  backoff: 1s
  max_backoff: 5s
  backoff_factor: 2

#----------------------------------Kafka output--------------------------------#
output.kafka:
  enabled: true
  hosts: ['3.1.101.33:9092','3.1.101.34:9092','3.1.101.35:9092']
  topic: '%{[kafka_topic]}'
EOF

docker-compose编排

mkdir -pv /data/docker-compose/filebeat
cat > /data/docker-compose/filebeat/docker-compose.yml << EOF
version: "3"
services:
  filebeat:
    container_name: filebeat
    image: elastic/filebeat:7.11.1
    user: root
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
      - /data/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /data/filebeat/data:/usr/share/filebeat/data/registry
      - /opt/ccms-auto-deploy:/opt/ccms-auto-deploy
      - /data/nginx/logs:/data/nginx/logs/
    deploy:
      resources:
        limits:
           memory: 4G
        reservations:
           memory: 1G
EOF

启动

docker-compose up -d

安装kibana仪表盘

docker-compose exec  filebeat filebeat setup --dashboards

过滤服务-Logstash

创建数据目录

mkdir -pv /data/logstash/{config,data,pipeline,logs}
chown 1000.1000 /data/logstash/{config,data,pipeline,logs}

配置文件

logstash.yml

cat > /data/logstash/config/logstash.yml << 'EOF'
node.name: logstast-node1
http.host: "0.0.0.0"
path.data: data
path.logs: /usr/share/logstash/logs
config.reload.automatic: true
config.reload.interval: 5s
config.test_and_exit: false
EOF

如果使用pipeline管道,不要配置path.config

pipelines.yml

cat > /data/logstash/config/pipelines.yml << 'EOF'
- pipeline.id: ccms-credit-java
  path.config: "/usr/share/logstash/pipeline/ccms-credit-java.conf"
- pipeline.id: ccms-credit-nginx-access
  path.config: "/usr/share/logstash/pipeline/ccms-credit-nginx-access.conf"
- pipeline.id: ccms-credit-nginx-error
  path.config: "/usr/share/logstash/pipeline/ccms-credit-nginx-error.conf"
EOF

pipeline配置文件

pipeline/ccms-credit-java.conf

cat > /data/logstash/pipeline/ccms-credit-java.conf<< 'EOF'
input {
   kafka {
    topics_pattern => "topic-ccms-credit-sit-java"
    bootstrap_servers => "172.16.20.51:9092,172.16.20.52:9092,172.16.20.53:9092"
    consumer_threads => 4
    decorate_events => true
    group_id => "kafka-ccms-credit-sit-java"
    add_field => {"logstash-server" => "172.16.20.51"}
   }
}

filter {
    json {
      source => "message"
    }

    grok {
      match => { "message" => "\[%{TIMESTAMP_ISO8601:currentDateTime}\] \[%{LOGLEVEL:level}\] \[%{DATA:traceInfo}\] \[%{NOTSPACE:class}\] \[%{DATA:hostName}\] \[%{IP:hostIp}\] \[%{DATA:applicationName}\] \[%{DATA:location}\] \[%{DATA:messageInfo}\] ## %{QUOTEDSTRING:throwable}" }
    }

    mutate{
      enable_metric => "false"
      remove_field => ["ecs","tags","input","agent","@version","log","port","host","message"]
    }

    date {
      match => [ "currentDateTime", "ISO8601" ]
    }
}

output {
        elasticsearch {
        hosts => ["172.16.20.51:9200","172.16.20.52:9200","172.16.20.53:9200"]
        user => "elastic"
        password => "G1T@es2022#ccms"
        index => "index-ccms-credit-sit-java_%{+YYY-MM-dd}"
        sniffing => true
        template_overwrite => true
    }
}
EOF

pipeline/ccms-credit-nginx-access.conf

cat > /data/logstash/pipeline.d/ccms-nginx-access.conf<< 'EOF'
input {
   kafka {
    topics_pattern => "topic-ccms-credit-sit-nginx-access"
    bootstrap_servers => "172.16.20.51:9092,172.16.20.52:9092,172.16.20.53:9092"
    codec => "json"
    consumer_threads => 4
    decorate_events => true
    group_id => "kafka-ccms-credit-sit-nginx-access"
    add_field => {"logstash-server" => "172.16.20.51"}
   }
}

filter {
  geoip {
      source => "client_ip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
      remove_field => [ "[geoip][latitude]", "[geoip][longitude]", "[geoip][country_code2]","[geoip][country_code3]", "[geoip][timezone]", "[geoip][continent_code]", "[geoip][dma_code]", "[geoip][region_code]" ]
  }

  mutate {
    convert => [ "size", "integer" ]
    convert => [ "status", "integer" ]
    convert => [ "responsetime", "float" ]
    convert => [ "upstreamtime", "float" ]
    convert => [ "[geoip][coordinates]", "float" ]
    # 过滤 filebeat 没用的字段,这里过滤的字段要考虑好输出到es的，否则过滤了就没法做判断
    remove_field => [ "ecs","agent","host","cloud","@version","input","logs_type" ]
  }


  useragent {
    source => "http_user_agent"
    target => "ua"
    # 过滤useragent没用的字段
    remove_field => [ "[ua][minor]","[ua][major]","[ua][build]","[ua][patch]","[ua][os_minor]","[ua][os_major]" ]
  }

}

output {
        elasticsearch {
        hosts => ["172.16.20.51:9200","172.16.20.52:9200","172.16.20.53:9200"]
        user => "elastic"
        password => "G1T@es2022#ccms"
        index => "logstash-ccms-credit-sit-nginx-access_%{+YYY-MM-dd}"
        sniffing => true
        template_overwrite => true
    }
}
EOF

pipeline/ccms-credit-nginx-error.conf

cat > /data/logstash/pipeline.d/ccms-nginx-error.conf<< 'EOF'
input {
   kafka {
    topics_pattern => "topic-ccms-credit-sit-nginx-error"
    bootstrap_servers => "172.16.20.51:9092,172.16.20.52:9092,172.16.20.53:9092"
    consumer_threads => 4
    decorate_events => true
    group_id => "kafka-ccms-credit-sit-nginx-error"
    add_field => {"logstash-server" => "172.16.20.51"}
    enable_metric => true
   }
}

filter {
    json {
      source => "message"
    }

    grok {
      match => [
        "message", "%{DATESTAMP:currentDateTime}\s{1,}\[%{LOGLEVEL:level}\]\s{1,}(%{NUMBER:pid:int}#%{NUMBER}:\s{1,}\*%{NUMBER})\s{1,}(%{GREEDYDATA:messageInfo})(?:,\s{1,}client:\s{1,}(?<client>%{IP}|%{HOSTNAME}))(?:,\s{1,}server:\s{1,}%{IPORHOST:server})(?:, request: %{QS:request})?(?:, upstream: \"%{URI:endpoint}\")?(?:, host: \"%{HOSTPORT:host}\")?(?:, referrer: \"%{URI:referrer}\")?",
        "message", "%{DATESTAMP:currentDateTime}\s{1,}\[%{DATA:level}\]\s{1,}%{GREEDYDATA:messageInfo}"]
    }

    date{
      match => ["currentDateTime", "yy/MM/dd HH:mm:ss", "ISO8601"]
      timezone => "+08:00"
      target => "@timestamp"
    }

    mutate{
      enable_metric => "false"
      remove_field => [ "ecs","tags","input","agent","@version","log","port","host","message" ]
    }
}

output {
        elasticsearch {
        hosts => ["172.16.20.51:9200","172.16.20.52:9200","172.16.20.53:9200"]
        user => "elastic"
        password => "G1T@es2022#ccms"
        index => "logstash-ccms-credit-sit-nginx-error_%{+YYY-MM-dd}"
        sniffing => true
        template_overwrite => true
    }
}
EOF

docker-compose编排

mkdir -pv /data/docker-compose/logstash
cat > /data/docker-compose/logstash/docker-compose.yml << EOF
version: "3"
services:
  logstash:
    container_name: logstash
    image: 172.16.20.50:8005/public/logstash:7.11.1
    user: root
    network_mode: host
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
      - /data/logstash/config:/usr/share/logstash/config
      - /data/logstash/data:/usr/share/logstash/data
      - /data/logstash/pipeline:/usr/share/logstash/pipeline
    environment:
      TZ: Asia/Shanghai
      LS_JAVA_OPTS: "-Xmx8G -Xms8G"
    deploy:
      resources:
        limits:
           memory: 10G
EOF

启动

docker-compose up -d

展示服务-Kibana

创建数据目录

mkdir -pv /data/kibana/{config,logs}
chown 1000 /data/kibana/{config,logs}

配置文件

cat > /data/kibana/config/kibana.yml << 'EOF'
# Default Kibana configuration for docker target
server.name: ccms-kibana
server.port: 5601
server.host: "0"
elasticsearch.hosts: [ "http://172.16.20.51:9200","http://172.16.20.52:9200","http://172.16.20.53:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
i18n.locale: "zh-CN"
map.tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

xpack.security.enabled: true
xpack.security.encryptionKey: "fhjskloppd678ehkdfdlliverpoolfcr"
elasticsearch.username: "elastic"
elasticsearch.password: "G1T@es2022#ccms"
EOF

docker-compose编排

mkdir -pv /data/docker-compose/kibana/
cat > /data/docker-compose/kibana/docker-compose.yml << EOF
version: "3"
services:
  kibana:
    container_name: kibana
    image: kibana:7.11.1
    restart: always
    ports:
      - "5601:5601"
    volumes:
      - /data/kibana/config/kibana.yml:/opt/kibana/config/kibana.yml
EOF

启动

docker-compose up -d
WAIT_TIME
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Filebeat读取日志推送kafka（docker-compose）
weixin_49566876的博客
01-04 912
Filebeat是本地文件的日志数据采集器，可监控日志目录或特定日志文件（tail file），并将它们转发给Elasticsearch或Logstatsh进行索引、kafka等。带有内部模块（auditd，Apache，Nginx，System和MySQL），可通过一个指定命令来简化通用日志格式的收集，解析和可视化。*** kafka常用命令。
Docker-Compose部署ELK
weixin_44359151的博客
03-04 2614
Docker-Compose部署ELK
docker-compose集成elk（基于logstash+filebeat）采集java和nginx日志
最新发布
Yangyg_0818的博客
05-09 373
访问kibana首页>stack management>索引模式>创建索引模式。
dockercompose搭建elk
qq_38899062的博客
08-02 6058
Elastic Stack简介 如果你没有听说过Elastic Stack，那你一定听说过ELK，实际上ELK是三款软件的简称，分别是Elasticsearch、 Logstash、Kibana组成，在发展的过程中，又有新成员Beats的加入，所以就形成了Elastic Stack。所以说，ELK是旧的称呼，Elastic Stack是新的名字。 全系的Elastic Stack技术栈包括： Elasticsearch ElasticSearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用
大数据日志监控filebeat+kafka+elk+docker部署
木一番的博客
12-02 2125
部署架构 日志格式说明 系统日志 {"file":"entry.go:224","level":"info","msg":"set sensive words: [呵呵呵呵 测试一下 你好]","time":"2020-04-24T11:20:32+08:00"} 访问日志，包含登录日志（请求类型值为2则为登录日志，登录日志才有额外信息） # IP - 用户名 [时间] "方法 路径 协议" 响应状态码 内容大小 "refere" "User-Agent" 请求次数 "路由名称" "后端服务地址" 花费
基于docker-compose构建filebeat + Logstash +Elasticsearch+ kibana日志系统
04-30
基于docker-compose构建filebeat + Logstash +Elasticsearch+ kibana日志系统 对nginx日志进行正则切割字段。 https://www.jianshu.com/p/f7927591d530
(亲测无坑)Cenos7.x无脑使用docker部署elk
01-09
今天给给位小伙伴带来的文章是使用docker无脑部署elk，废话不多说，直接上干货！！ 每条命令下方的文字为命令的解释注释！！ systemctl stop firewalld #关闭防火墙； systemctl disable firewalld #禁止开机自启； ...
Docker部署ELK7.3.0日志收集服务最佳实践
01-08
本文仅包含ELK7.3.0部署！ 部署环境： 系统 CentOS 7 Docker Docker version 19.03.5 CPU 2核 内存 2.5G 磁盘 30G（推荐设置，磁盘不足可能会引发es报错） Filebeat v7.3.0，单节点 ElasticSearch v...
Docker构建ELK Docker集群日志收集系统
09-30
为了在Docker集群中更好的管理查看日志 我们使用Docker 来搭建集群的ELK日志收集系统，这篇文章介绍了Docker构建ELK Docker集群日志收集系统的相关资料,需要的朋友可以参考下
详解使用Docker快速部署ELK环境（最新5.5.1版本）
09-29
主要介绍了详解使用Docker快速部署ELK环境（最新5.5.1版本），文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值，需要的朋友们下面随着小编来一起学习学习吧
docker-elk-main.rar
05-20
docker-compose 方式部署ELK方案
ELK6.3和Filebeat6.3在Docker-Compose环境下安装和使用
11-29
Centos系统ELK6.3和Filebeat6.3在Docker-Compose环境下安装和使用
docker安装elk6.7.1-搜集java日志
07-09
Docker 是一个流行的容器化平台，能够帮助我们快速部署应用程序。在本文中，我们将使用 Docker 19.03.2。 首先，安装 Docker： ``` [root@localhost ~]# docker info Server Version: 19.03.2 ``` 其次，设置 ...
详解如何使用Docker快速部署ELK环境(最新5.5.1版本)
09-30
主要介绍了详解如何使用Docker快速部署ELK环境(最新5.5.1版本),小编觉得挺不错的，现在分享给大家，也给大家做个参考。一起跟随小编过来看看吧
elk的docker-compose配置
12-17
使用docker-compose 搭建elk日志系统.使用docker-compose 搭建elk日志系统
Docker-compose部署ELK的示例代码
01-20
环境 主机IP 192.168.0.9 Docker version 19.03.2...一、ELK-dockerfile文件编写及配置文件 ● elasticsearch 1、elasticsearch-dockerfile FROM centos:latest ADD elasticsearch-6.6.1.tar.gz /usr/local/ COPY el
ELK搭建及基础使用（docker版）
weixin_49566876的博客
03-23 992
ELK平台是一套完整的日志集中处理解决方案，将 ElasticSearch、Logstash 和 Kiabana 三个开源工具配合使用，完成更强大的用户对日志的查询、排序、统计需求● ElasticSearch（日志存储和搜索）：是基于Lucene（一个全文检索引擎的架构）开发的分布式存储检索引擎，用来存储各类日志。Elasticsearch 是用 Java 开发的，可通过 RESTful Web 接口，让用户可以通过浏览器与 Elasticsearch通信。
docker安装ELK详细步骤
上善若水
10-21 2709
elk 安装
ELK部署 (docker-compose方式)
hanguofei的博客
11-20 5444
ELK的部署方式有很多种，这里我介绍一种简单的部署方式：docker-compose 一、docker-compose.yml version: "3.2" services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.4.2 container_name: elast...
docker 部署elk
09-01
你可以使用Docker来部署ELK（Elasticsearch, Logstash, Kibana）堆栈。以下是一些步骤： 1. 安装Docker和Docker Compose：请确保你的机器上已经安装了Docker和Docker Compose。 2. 创建一个新的目录并在该目录下...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值