Install ELK Stack 6.7 and Elastiflow 3.4.1 on CentOS 7

Elastiflow 3.4.1 和最新的 ELK Stack 7 不兼容，所以要安装ELK Stack 6.7

“ELK” is the acronym for?Elasticsearch, Logstash, and Kibana. A short description of these tools is covered in the next block.
* Elasticsearch: This is an open source, distributed, RESTful, JSON-based search engine. It is scalable,?easy to use, and flexible
* Logstash?: This is a serverside data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch.
* Kibana?lets users visualize data with charts and graphs in Elasticsearch.
For RHEL 8, refer to:
How to Install ELK Stack on RHEL 8 / CentOS 8
Please follow our steps below to install and configure ELK stack tools on CentOS 7 / Fedora 29 / Fedora 28.
Step 1: Install Java
CentOS 7自带 openjdk8，可以直接使用
 
Step 2: Add ELK repository
Once you have Java installed, add ELK stack repository which provides ELK stack packages. Please run the commands below as root user:
cat > /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
The above command will configure?Elasticsearch?6.x repository. If you want Elasticsearch 5.x, add below instead
cat > /etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
After adding the repo, import GPG key:
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Clear and update your YUM package index.
sudo yum clean all
sudo yum makecache
Step 3: Install and Configure Elasticsearch
Elasticsearch repository is ready for use. You can install Elasticsearch using the command below:
sudo yum -y install elasticsearch
Confirm package installation.
$ rpm -qi elasticsearch 
 Name        : elasticsearch
 Epoch       : 0
 Version     : 6.5.4
 Release     : 1
 Architecture: noarch
 Install Date: Wed 02 Jan 2019 02:12:22 PM EAT
 Group       : Application/Internet
 Size        : 167695796
 License     : Elastic License
 Signature   : RSA/SHA512, Tue 18 Dec 2018 01:18:33 AM EAT, Key ID d27d666cd88e42b4
 Source RPM  : elasticsearch-6.5.4-1-src.rpm
 Build Date  : Tue 18 Dec 2018 12:21:58 AM EAT
 Build Host  : packer-virtualbox-iso-1542822913
 Relocations : /usr 
 Packager    : Elasticsearch
 Vendor      : Elasticsearch
 URL         : https://www.elastic.co/
 Summary     : Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
 Description :
 Elasticsearch subproject :distribution:packages
You can set JVM options like memory limits by editing the file:?/etc/elasticsearch/jvm.options
Example below sets initial/maximum size of total heap space
-Xms1g
-Xmx1g
If your system has less memory, you can configure it to use small megabytes of ram.
-Xms256m
-Xmx512m
Start and enable elasticsearch service on boot:
$ sudo systemctl enable --now elasticsearch.service 
 Synchronizing state of elasticsearch.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
 Executing: /usr/lib/systemd/systemd-sysv-install enable elasticsearch
 Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /usr/lib/systemd/system/elasticsearch.service.
Test to verify that it is working:
$ curl http://127.0.0.1:9200 
 {
   "name" : "bBzN5Kg",
   "cluster_name" : "elasticsearch",
   "cluster_uuid" : "LKyqXXSvRvCpX9QAwKlP2Q",
   "version" : {
     "number" : "6.5.4",
     "build_flavor" : "default",
     "build_type" : "rpm",
     "build_hash" : "d2ef93d",
     "build_date" : "2018-12-17T21:17:40.758843Z",
     "build_snapshot" : false,
     "lucene_version" : "7.5.0",
     "minimum_wire_compatibility_version" : "5.6.0",
     "minimum_index_compatibility_version" : "5.0.0"
   },
   "tagline" : "You Know, for Search"
 }
Create a test index:
$ curl -X PUT "http://127.0.0.1:9200/mytest_index"
{"acknowledged":true,"shards_acknowledged":true,"index":"mytest_index"}
Step 3: Install and Configure Kibana
Download and install Kibana from added Elasticsearch repository.
sudo yum -y install kibana
After a successful installation, configure Kibana:
$ sudo vim /etc/kibana/kibana.yml
 server.host: "0.0.0.0"
 server.name: "kibana.example.com"
 elasticsearch.url: "http://localhost:9200"
Change other settings as desired then start kibana service:
sudo systemctl enable --now kibana
Access?http://ip-address:5601?to open Kibana Dashboard:

If you have an active firewall service, allow TCP port?5601
sudo firewall-cmd --add-port=5601/tcp --permanent
sudo firewall-cmd --reload
Step 4: Install and Configure Logstash
The last installation is for Logstash. It will act as a centralized logs server for your client systems which runs an agent like?filebeat.
sudo yum -y install logstash
Logstash custom configurations can be placed under?the?/etc/logstash/conf.d/directory.
Check?Logstash Configuration manual?for more details.
Step 5:
一定要设置4g，默认1g不够用，内存会溢出
Set jvm mem settings
sudo sed -i 's|-Xms1g|-Xms4g|g' /etc/logstash/jvm.options
sudo sed -i 's|-Xmx1g|-Xmx4g|g' /etc/logstash/jvm.options
Add req. plugins
sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
mkdir -p /opt/flowtemp
Navigate to the temp folder you just created

cd /opt/flowtemp
Download Elastiflow install files

wget https://github.com/robcowart/elastiflow/archive/master.zip
Install unzip, so you can extract the archive file you just downloaded

unzip master.zip
Copy ElastiFlow configuration files to the Logstash directory

# cp -a elastiflow-master/logstash/elastiflow/. /etc/logstash/elastiflow/
Setup environment variable helper files
Copy the elastiflow.conf to systemd

# cp -a elastiflow-master/logstash.service.d/. /etc/systemd/system/logstash.service.d/
Add the ElastiFlow pipeline to pipelines.yml
Open the Logstash pipline configuration file for editing.

1
vi /etc/logstash/pipelines.yml
Add the two line below to the bottom of the pipelines.yml file

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"
 或者
cp /etc/logstash/pipelines.yml /etc/logstash/pipelines.yml-DIST
tee -a /etc/logstash/pipelines.yml << EOF
- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"
EOF
if you want to check your yamlfile.
sudo apt install -y yamllint
yamllint /etc/logstash/pipelines.yml
/usr/share/logstash/bin/system-install
systemctl daemon-reload
systemctl enable logstash
systemctl start logstash

tail /var/log/logstash/logstash-plain.log -f