一.Elasticsearch安装
(1).操作系统调优(必须配置,否则ES起不来)
【1】内存优化
在/etc/sysctl.conf添加如下内容
fs.file-max=655360
vm.max_map_count=655360
刷新生效
sysctl -p
解释:
(1)vm.max_map_count=655360
系统最大打开文件描述符数
(2)vm.max_map_count=655360
限制一个进程拥有虚拟内存区域的大小
【2】修改
vim /etc/security/limits.conf
-
* soft nofile 65536 * hard nofile 65536 * soft nproc 65536 * hard nproc 65536 * soft memlock unlimited * hard memlock unlimited
解释:
(nofile)最大开打开文件描述符
(nproc)最大用户进程数
(memlock)最大锁定内存地址空间
(2)新建文件夹并上传elasticsearch文件
mkdir /opt/elasticsearch
tar -zxvf elasticsearch-7.4.2-linux-x86_64.tar.gz
(3)修改es配置文件
vim ./config/elasticsearch.yml
cluster.name: my-application # 集群名称
node.name: node-1 # 节点名称,仅仅是描述名称,用于在日志中区分
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /opt/elasticsearch/data # 数据的默认存放路径
#
# Path to log files:
#
path.logs: /opt/elasticsearch/logs # 日志的默认存放路径
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["172.31.0.24"] #集群个节点IP地址
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"] #master节点资格
#
# For more information, consult the discovery and cluster formation module documentation.
(4)添加elasticsearch启动用户
adduser es //创建一个账号为 es的用户
chown -R es:es /opt/elasticsearch/ //给es用户赋予操作此文件夹的权限
su es // 使用 es账户操作,如果是 su root 意为使用 root用户操作
(5)安装ik分词器
在/opt/elasticsearch/elasticsearch/plugins下新建ik文件夹
mkdir /opt/elasticsearch/elasticsearch/plugins/ik
将elasticsearch-analysis-ik-7.4.2.zip上传到ik文件夹下并解压
yum install -y unzip //下载unzip命令
unzip elasticsearch-analysis-ik-7.4.2.zip //解压文件夹
bin目录下查看已经加载的插件
./elasticsearch-plugin list
(6)启动es
在elasticsearch的bin目录下
//需要用es用户启动
./elasticsearch //前台启动
./elasticsearch -d //后台启动
http://127.0.0.1:9200/_cat/nodes?v&s=index //查看节点信息
http://127.0.0.1:9200/_cat/indices?v //查看索引信息
二.kibana安装
(1)上传安装包 解压并改名
tar -zxvf kibana-7.4.2-linux-x86_64.tar.gz
mv ./kibana-7.4.2 /kibana
chown -R es:es /opt/kibana
(2)配置config/kibana.yml
server.port: 5601 //端口
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0" //允许来自远程用户的连接
server.name: "my_kibana" //服务名称
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://172.31.0.24:9200"] //elasticsearch地址
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
i18n.locale: "zh-CN" //中文
(3)启动kibana
su es //使用es用户启动
bin/kibana //前台启动
nohup bin/kibana & //后台启动
三.安装logstash
(1)上传安装包 解压并改名
tar -zxvf /opt/elasticsearch/logstash-7.4.2.tar.gz
cd /opt/elasticsearch/logstash-7.4.2
(2)新建config/收集nginx接口日志
vim nginxtest.conf
input{
file{
path => "/var/log/nginx/access.log" # 想要收集的日志信息文件-注意全路径
type=>"logstash_log" # 指定一个类型
start_position =>"beginning" # 说明从文件最开始读
}
}
#@timestamp比当前时间慢8小时
filter {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)"
}
ruby {
code => "event.set('@timestamp',event.get('timestamp'))"
}
mutate {
remove_field => ["timestamp"]
}
}
##@timestamp比当前时间快8小时
#修改kibana当前时区 GMT+0 (默认获取浏览器时间)
output{
elasticsearch{ # 配置elasticsearch接收数据信息
hosts=>["http://127.0.0.1:9200"] # 配置elasticsearch端口信息
index=>"log-%{+YYYY.MM.dd}" # 配置Kibana新建index,比如我这里是”log-“开头,在Kibana可以写”log-*”
#使用自定义索引模板
template => "/opt/elasticsearch/logstash-7.4.2/nginxtest.json"
template_name => "log-*"
template_overwrite => true
}
}
(3)新建json文件/nginxtest.json
Elasticsearch-7x版本的基本模板
https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/main/lib/logstash/outputs/elasticsearch/templates/ecs-disabled/elasticsearch-7x.json
Es为单节点生成日志索引副本数默认是1,新建的索引会有未分配碎片
vim nginxtest.json
{
"index_patterns" : "log-*",
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s",
"index.number_of_replicas" : 0, //副本数改为0
"number_of_shards": 1
},
"mappings" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
(4)启动logstash
./bin/logstash -f nginxtest.conf -t //检查配置文件是否正确
./bin/logstash -f nginxtest.conf //启动logstash