文章目录
机器准备
10.20.1.114 | node01 |
---|---|
10.20.1.115 | node02 |
10.20.1.116 | node03 |
集群内各台机器安装Elasticsearch
1、下载Elasticsearch的安装包
官方地址:https://www.elastic.co/cn/downloads/past-releases#elasticsearch
2、Elasticsearch安装(每台机器都执行)
#上传到安装包存放在/data/soft目录
#安装
rpm -ivh elasticsearch-6.4.2.rpm
#查看elasticsearch配置文件目录
rpm -qc elasticsearch
#编辑elasticsearch.yml配置文件
vi /etc/elasticsearch/elasticsearch.yml
grep "^[a-Z]" /etc/elasticsearch/elasticsearch.yml
node.name: node03 #节点名称,同一个集群内所有节点的节点名称不能重复
path.data: /data/elasticsearch #将es的数据存在该目录,注意创建该目录
path.logs: /var/log/elasticsearch #日志目录,会创建以集群名称的一个日志目录 eg:es-app.log
bootstrap.memory_lock: true #内存锁定
network.host: 10.20.xx.xx #绑定监听地址
http.port: 9200 #默认端口号
#创建es数据存储目录
mkdir /data/elasticsearch -p
#将es数据存储目录指定给elasticsearch用户
chown -R elasticsearch:elasticsearch /data/elasticsearch/
###锁定内存失败(memory locking requested for elasticsearch process but memory is not locked)的解决办法:
参考官方解决方案:
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/setup-configuration-memory.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/setting-system-settings.html#sysconfig
#编辑配置文件
systemctl edit elasticsearch
#将以下配置写入到配置文件&&保存
[Service]
LimitMEMLOCK=infinity
#启动elasticsearch服务&&查看es端口是否开启
systemctl start elasticsearch.service
systemctl status elasticsearch.service
netstat -natp | grep 9200
3、ES集群部署
#在elasticsearch.yml配置文件中打开注释,设置相关参数
cluster.name: es-app #集群名称,同一个集群内所有节点集群名称相同
discovery.zen.ping.unicast.hosts: ["10.20.1.114", "10.20.1.115", "10.20.1.116"] #集群发现节点配置
discovery.zen.minimum_master_nodes: 2 #选举相关参数,公式: 节点数/2 +1
向集群中创建一些索引,数据
创建索引
[root@master elasticsearch]# curl -XPUT '10.20.1.114:9200/vipinfo?pretty'
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "vipinfo"
}
插入文档数据
curl -XPUT '10.20.1.114:9200/vipinfo/user/1?pretty' -H 'Content-Type: application/json' -d'
{
"first_name" : "John",
"last_name": "Smith",
"age" : 25,
"about" : "I love to go rock climbing", "interests": [ "sports", "music" ]
}
'
curl -XPUT '10.20.1.114:9200/vipinfo/user/2?pretty' -H 'Content-Type: application/json' -d' {
"first_name": "Jane",
"last_name" : "Smith",
"age" : 32,
"about" : "I like to collect rock albums", "interests": [ "music" ]
}'
#查看集群状态
curl -XGET 'http://10.20.1.114:9200/_cat/nodes?human&pretty'
4、安装Elasticsearch-head插件(可视化插件)
进入谷歌应用商店:https://chrome.google.com/webstore/category/extensions
搜索关键字:elasticsearch-head
将其添加到应用程序
5、安装Kibana
#安装kibana-6.6.0-x86_64.rpm包
rpm -ivh kibana-6.6.0-x86_64.rpm
#修改Kibana配置文件
[root@master ~]# grep "^[a-Z]" /etc/kibana/kibana.yml
server.port: 5601
server.host: "10.20.1.114"
server.name: "node01"
elasticsearch.hosts: ["http://10.20.1.114:9200"]
kibana.index: ".kibana"
#启动Kibana&&查看服务状态&&通过IP访问界面
systemctl start kibana.service
systemctl status kibana.service
6、安装filebeat
#安装filebeat包
rpm -ivh filebeat-6.4.2-x86_64.rpm
#查看filebeat配置文件
rpm -qc filebeat
#启动filebeat服务&&查看服务状态
#若filebeat无法启动并无报错日志,执行以下指令可看到报错
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml
ELK收集Nginx的json日志
思路:1、将nginx中的日志以json格式记录
2、filebeat采的时候说明是json格式
3、传入es的日志为json,那么显示在kibana的格式也是json,便于日志管理
1、配置nginx的日志以json格式记录
#修改/etc/nginx/nginx.conf配置文件,加入以下内容,yml文件注意缩进
log_format json '{ "time_local": "$time_local", '
'"remote_addr": "$remote_addr", '
'"referer": "$http_referer", '
'"request": "$request", '
'"status": $status, '
'"bytes": $body_bytes_sent, '
'"agent": "$http_user_agent", '
'"x_forwarded": "$http_x_forwarded_for", '
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"upstream_time": "$upstream_response_time",'
'"request_time": "$request_time"'
' }';
access_log /var/log/nginx/access.log json;
#重启nginx服务
systemctl restart nginx.service
#再次进行压测&&查看nginx日志是否记录显示为json格式的键值对&&查看可知已是json格式
ab -n 100 -c