环境:os:centos 7
jdk:java 8
1、下载ES https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.2.zip
2、下载kibana(图形管理界面 )https://artifacts.elastic.co/downloads/kibana/kibana-6.2.2-linux-x86_64.tar.gz
3、logstash https://artifacts.elastic.co/downloads/logstash/logstash-6.2.2.tar.gz
一、安装ES
解压ES,再目录里面有elasticsearch.yml jvm.options log4j2.properties三个配置文件,从上面可以看到三个配置大概的功能,做cluster需要更改elasticsearch.yml 里面两个配置。
1、修改配置
#集群名称,整个集群内的名称要一致
cluster.name=esDev
# 节点名称
# Set the bind address to a specific IP (IPv4 or IPv6):
network.host: 10.*.*.14
#使用默认端口9200
network.port: 9200
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["10.39.14.13", "10.39.14.29"]
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
# 为了避免脑裂,配置主节点个数(总节点数/2+1)
discovery.zen.minimum_master_nodes: 2
# Path to directory where to store the data (separate multiple locations by comma):
# 存储数据的地址配置
path.data: /data/es/elasticsearch-6.2.2/data
2、启动 sh bin/elasticsearch
3、错误解决
[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
解决:文件数太小,root用户下调整/etc/profile ulimit -n 65536(此方法为临时解决,持久调整参考其他)
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决:echo "vm.max_map_count=262144" >>/etc/sysctl.conf
如果已经存在vm.max_map_count配置,查看是否值大小不对。 max_map_count文件包含限制一个进程可以拥有的VMA(虚拟内存区域)的数量。虚拟内存区域是一个连续的虚拟地址空间区域。在进程的生命周期中,每当程序尝试在内存中映射文件,链接到共享内存段,或者分配堆空间的时候,这些区域将被创建。调优这个值将限制进程可拥有VMA的数量。限制一个进程拥有VMA的总数可能导致应用程序出错,因为当进程达到了VMA上线但又只能释放少量的内存给其他的内核进程使用时,操作系统会抛出内存不足的错误。如果你的操作系统在NORMAL区域仅占用少量的内存,那么调低这个值可以帮助释放内存给内核用
二:安装Kibana
解压下载到的Kibana,config/kibinana.yml是其配置文件,修改如下
Server.host 配置机器ip/hostname
Server.name 此kibana服务的名称
elasticsearch.url es master节点url
启动kibana
三:安装logstash
解压安装文件,修改配置文件config/logstash.yml
http.host: "10.39.14.29"
http.port:9600
node.name: logstash29
安装logstash-input-jdbc 执行 bin/logstash-plugin install logstash-input-jdbc
等一会儿出现:Validating logstash-input-jdbc
Installing logstash-input-jdbcInstallation successful
表示安装成功
安装完插件以后,添加配置,同步数据到es中,
新建mysql.conf
input {
stdin {
}
jdbc {
# mysql jdbc connection string to our backup databse
jdbc_connection_string => "jdbc:mysql://10.39.7.240:3306/zhwl_tms1"
# the user we wish to excute our statement as
jdbc_user => "test"
jdbc_password => "test"
# the path to our downloaded jdbc driver
jdbc_driver_library => "/data/driver/mysql-connector-java-5.1.44.jar"
# the name of the driver class for mysql
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement_filepath => "mysql.sql"
schedule => "* * * * *"
type => "jdbc"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
elasticsearch {
hosts => "127.0.0.1:9200"
#port => "9200"
#has been removed
#protocol => "http"
index => "mysql01"
document_id => "%{id}"
#cluster => "logstash-elasticsearch"
}
stdout {
codec => json_lines
}
}
新建mysql.sql里面写入同步数据的sql
然后启动sh logstash -f mysql.conf
1、注意:mysql的两个配置文件要和logstash 放到同一目录,启动完毕一会儿就能观察到同步日志,更多同步策略后续发出。
2、注意:output中配置master ES地址即可,不然会导致ES 内部通信错乱。如果错乱,停掉logstash,过一会儿ES 会自行恢复