环境
Windows10、Docker Version 20.10.7、ELK(Elasticsearch7.13.2+Kibana7.13.2+Logstash7.13.2)
步骤
1. 准备ELK启动环境
1.1 在Windows下建立C:\elk\
目录,建立目录及加入文件如下
目录 | 文件 | 说明 |
---|---|---|
C:\elk | docker-compose.yml | 启动ELK的compose文件,能批量启动docker |
C:\elk | logstash.yaml | Logstash配置文件 |
C:\elk\pipeline | mylog.conf | 日志输入、匹配、输出文件 |
C:\elk\logs | test.log | 日志文件,用于匹配的数据 |
其中,各文件内容如下:
- docker-compose.yml
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.13.2
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02
- cluster.initial_master_nodes=es01,es02
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.13.2
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01
- cluster.initial_master_nodes=es01,es02
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
networks:
- elastic
kib01:
image: docker.elastic.co/kibana/kibana:7.13.2
container_name: kib01
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://es01:9200
ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200"]'
networks:
- elastic
log01:
image: docker.elastic.co/logstash/logstash:7.13.2
container_name: log01
volumes:
- C:/elk/logstash.yaml:/usr/share/logstash/config/logstash.yaml
- C:/elk/pipeline:/usr/share/logstash/pipeline
- C:/elk/logs/test.log:/data/logs/test.log
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
networks:
elastic:
driver: bridge
- logstash.yaml
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://es01:9200" ]
- mylog.conf
input {
file {
path => "/data/logs/test.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => { "message" => "ts=%{TIMESTAMP_ISO8601:ts} level=%{LOGLEVEL:level} msg=%{GREEDYDATA:msg}" }
}
}
output {
elasticsearch {
hosts => ["es01:9200"]
}
}
- test.log
ts=2020-11-13T09:16:18.679+08:00 level=info msg="Shutting down all..."
ts=2021-08-05T15:32:18.679+08:00 level=info msg="test message"
2. Docker启动ELK验证
2.1 执行命令C:\elk>docker-compose up -d
,从Docker hub上拉去并执行镜像。
注意:首次执行需要等待较长时间,因为拉取镜像需要较长时间。之后执行时,因为本地已有ELK的相关镜像,启动会很快。
2.2 执行命令C:\elk>docker ps
,查看Docker启动情况。
2.3 等待2~3分钟,检查4个容器是否继续运行。
- 如果发现两个elasticsearch(es1、es2)挂掉,则执行
docker logs log01
查看启动日志。如发现如下错误
ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/es-docker-cluster.log
这说明Docker的启动内存不够,可以通过命令修改和Docker Desktop修改。
a. 手动修改,需要在C:\elk
下执行如下两条命令
wsl -d docker-desktop
sysctl -w vm.max_map_count=262144
注意:可能需要在C:\User\UserName\
下新建一个.wslconfig
文件,并添加如下内容。
[wsl2]
memory=4GB
processors=2
b. Docker Desktop修改。
3. Elasticsearch验证
3.1 输入网址http://localhost:5601/
,查看elastic启动情况。
3.2 从面板进入如下位置,新建索引,命名为logstash-***
。
3.3 选择Time field。
3.4 选择面板,点击Dicover,选择logstash,查看刚刚本地映射到docker中的test.log文件中的内容。
参考
ELK原理与介绍
Docker安装部署ELK教程 (Elasticsearch+Kibana+Logstash+Filebeat)
ELK安装使用教程
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]