【记录】Ubuntu 21 搭建 Elasticsearch Logstash Kibana

进入官网

https://www.elastic.co/cn/downloads/

elasticsearch 安装

下载

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.1-linux-x86_64.tar.gz

解压

tar -zvxf elasticsearch-7.13.1-linux-x86_64.tar.gz

创建目录

新建数据目录和日志目录

lxp@lxp:~/es$ ls
data  elasticsearch-7.13.1  logs
lxp@lxp:~/es$

修改配置

vim config/elasticsearch.yml

lxp@lxp:~/es$ cd elasticsearch-7.13.1/
lxp@lxp:~/es/elasticsearch-7.13.1$ vim config/elasticsearch.yml
lxp@lxp:~/es/elasticsearch-7.13.1$

elasticsearch.yml修改如下

# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /home/lxp/es/data
#
# Path to log files:
#
path.logs: /home/lxp/es/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: false

# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 192.168.1.101
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]

系统准备

修改文件限制

lxp@lxp:~$ sudo vim /etc/security/limits.conf
[sudo] password for lxp:
lxp@lxp:~$

* soft nproc 5000
* hard nproc 5000
root soft nproc 5000
root hard nproc 5000

重启后才能生效

修改最大并发连接

需要修改最大并发连接，不然启动会报错
设置sudo vim /etc/sysctl.conf

添加如下配置

vm.max_map_count=262144

lxp@lxp:~/es/elasticsearch-7.13.1$ vim config/elasticsearch.yml
lxp@lxp:~/es/elasticsearch-7.13.1$ sudo vim /etc/security/limits.conf
lxp@lxp:~/es/elasticsearch-7.13.1$ sudo vim /etc/sysctl.conf
lxp@lxp:~/es/elasticsearch-7.13.1$ sudo sysctl -p
vm.max_map_count = 262144
lxp@lxp:~/es/elasticsearch-7.13.1$

并执行 sudo sysctl -p

lxp@lxp:~$ sudo sysctl -p
vm.max_map_count = 262144

安装jdk

sudo apt install openjdk-11-jdk

lxp@lxp:~/es/elasticsearch-7.13.1$ sudo apt install openjdk-11-jdk
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done

启动ES

前台启动

./bin/elasticsearch

后台启动

方式1：nohup ./bin/elasticsearch > es_out.log 2>&1 &

方式2： ./bin/elasticsearch -d

验证

浏览器访问 http://192.168.1.101:9200/

{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "fEgyk_AzT56PwFnFsXDuwg",
  "version" : {
    "number" : "7.13.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "9a7758028e4ea59bcab41c12004603c5a7dd84a9",
    "build_date" : "2021-05-28T17:40:59.346932922Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

kibana 安装

下载

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.13.1-linux-x86_64.tar.gz

解压

tar -zvxf  kibana-7.13.1-linux-x86_64.tar.gz

lxp@lxp:~/kb$ ls
kibana-7.13.1-linux-x86_64  kibana-7.13.1-linux-x86_64.tar.gz
lxp@lxp:~/kb$

安装依赖包

lxp@lxp:~/kb$ sudo apt update
lxp@lxp:~$ sudo apt install libnss3-dev

配置文件修改

进入kibana-7.8.1根目录

cd /usr/elk/kibana-7.8.1
#修改配置
vim config/kibana.yml

增加以下内容

i18n.locale: "zh-CN"
server.port: 5601
server.host: "{ip}"     #kibana本机的地址
elasticsearch.hosts: "http://{IP}:port"  #ES主节点地址+端口
kibana.index: ".kibana"

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "192.168.1.101"
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://192.168.1.101:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
i18n.locale: "zh-CN"

启动

前台启动

./bin/kibana

后台启动

nohup ./bin/kibana > kibana_out.log 2>&1 &

验证

浏览器访问： http://192.168.1.101:5601/

安装Logstash

下载logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.13.1-linux-x86_64.tar.gz

解压

tar -zvxf  logstash-7.13.1-linux-x86_64.tar.gz

创建data 和logs 目录

lxp@lxp:~/lt$
lxp@lxp:~/lt$ ls
data  logs  logstash-7.13.1  logstash-7.13.1-linux-x86_64.tar.gz
lxp@lxp:~/lt$

编辑配置文件

vim config/logstash.yml

# ------------ Data path ------------------
#
# Which directory should be used by logstash and its plugins
# for any persistent needs. Defaults to LOGSTASH_HOME/data
#
path.data: /home/lxp/lt/data

# ------------ Debugging Settings --------------
#
# Options for log.level:
#   * fatal
#   * error
#   * warn
#   * info (default)
#   * debug
#   * trace
#
# log.level: info
path.logs: /home/lxp/lt/logs/

新建input文件

vim config/input-output.conf

input {
    file {
        path => ["/home/lxp/spring/logs/*.log"]

        codec => multiline {
            pattern => "^(\[%{TIMESTAMP_ISO8601}\])"
            negate => true
            what => "previous"
        }
        sincedb_path => "NUL"
        type => "spring"
        start_position => "beginning"
    }
}

filter {
	if [type] == "spring" {
		grok {
			match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{WORD:service}\]\[%{LOGLEVEL:level}\]\[%{NOTSPACE:thread}\]\[%{WORD:logId}:%{NOTSPACE:traceId}\]" }
		}
	}
}


output {
    if [type] == "spring" {
        elasticsearch {
            hosts => ["192.168.1.101:9200"]
            index => "spring-%{+YYYY.MM.dd}"
        }
    }
	stdout { 
		codec => rubydebug 
	}
}

启动

#进入Logstash根目录
cd /home/lxp/lt/logstash-7.13.1
#启动
./bin/logstash -f config/input-output.conf