http://knowm.org/how-to-set-up-the-elk-stack-elasticsearch-logstash-and-kibana/
Elastic Search, Logstash and Kibana – the ELK Stack – is emerging as the best technology stack to collect, manage and visualize big data. If you came here looking for help installing the ELK stack, you don’t need an introduction, so let’s get right down to the dirty work. The following guide shows how to install Java 8, Elasticsearch 2.3, Logstash 2.3 and Kibana 4 on Ubuntu with init.d (system v
) or alternatively with systemd
. You can do one or the other depending on your system and/or preferences. In two previous posts Integrate Bro IDS with ELK Stack and How to Install Bro Network Security Monitor on Ubuntu
, we showed how to install Bro and parse the generated Bro logs with Logstash. With the entire stack installed, running, and parsing logs generated by Bro, Kibana allows for a wonderful data analysis and discovery process. Of course, almost any data source can be used, and not just Bro.
Oracle Java 8
Elasticsearch
Note: Check for the latest Elastic Search release version here: downloads/elasticsearch
System V
Systemd
Configure
Note: If you want to access your Elasticsearch instance from clients on a different IP address via Javascript, add the following inside elasticsearch.yml
:
Also note that if you want to access Elasticsearch
of any of the plugins like kopf
from a host besides local host, you’ll need to add the following to elasticsearch.yml
:
FYI, the Elasticsearch stores your actual data in /var/lib/elasticsearch/elasticsearch/nodes/...
.
Test
In browser: http://localhost:9200/
Hello World Data
Debugging
Debug startup errors by running elasticsearch in the console
Elasticsearch Kopf Plugin (an aside)
The kopf plugin provides an admin GUI for Elasticsearch. It helps in debugging and managing clusters and shards. It’s really easy to install (check here for latest verion):
View in browser at: http://localhost:9200/_plugin/kopf/#!/cluster. You should see something like this:
Logstash
Note: Check for the latest Logstash release version here: downloads/logstash
System V
Systemd
Configure
By default Logstash filters will only work on a single thread, and thus also one CPU core. To increase the number of cores available to LogStash, edit the file /etc/default/logstash and set the -w parameter to the number of cores:LS_OPTS="-w 8"
.
You can increase the Java heap size here as well. Make sure to uncomment the line you are updating. Don’t forget to restart logstash afterwards.
Test
Directly change java options in script if you are starting logstash
from the command line and not as a linux service.
Hello World (warning Logstash at the command line is slow to start, so be patient)
Hello World with Elastic Search
Plugins
For non-standard parsing features, we access plugins. The following terminal commands show how to install thelogstash-filter-translate plugin. For a more in-depth explanation of installing logstash
plugins see How to Install Logstash Plugins for Version 1.5.
Kibana
Note: Check for the latest Kibana release version here: downloads/kibana
System V
Systemd
Configure
Test
In browser: http://localhost:5601
Final Words
If all went well, the next step is to tap into a datasource with Logstash and view it with Kibana. In two previous postsIntegrate Bro IDS with ELK Stack and How to Install Bro Network Security Monitor on Ubuntu
, we showed how to install Bro and parse the generated Bro logs with Logstash. The following is a screen shot from a Kibana dashboard we made for one of our websites bitcoinium.com showing some nice bar and pie charts. Once everything is set up and running it immediately becomes clear how useful the ELK stack is.
Related Resources
Buy the Plug and Play Network Monitor directly from knowm.org: http://knowm.org/product/plug-and-play-network-monitor/
Integrating Bro with the ELK Stack: http://knowm.org/integrate-bro-ids-with-elk-stack/
How to Created a Bonded Network Interface: http://knowm.org/how-to-create-a-bonded-network-interface/