ELK stack部署 (elasticsearch 2.4.4、Logstash 2.4.1、Kibana 4.6.4)

ELK stack

ELK stack 是以 Elasticsearch、Logstash、Kibana 三个开源软件为主的数据处理工具链。

在实时数据检索和分析场合，三者通常是配合使用，而且又先后归于Elastic.io公司名下，故有Elastic Stack简称。

ELK stack部署 (elasticsearch 2.4.4、Logstash 2.4.1、Kibana 4.6.4)

分为Linux Server Host部署方式和Docker部署方式。

初始化Linux Server Host

修改时区：
> sudo cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
> sudo dpkg-reconfigure -f noninteractive tzdata

Clone本仓库
> cd ~/
> git clone https://git.oschina.net/atompi/elk-docker.git
> cd ~/elk-docker

修改apt源为阿里云：
> sudo cp sources.list /etc/apt/sources.list
> sudo apt-get update

elasticsearch部署 (单节点)

Linux Server Host部署方式 (Ubuntu 16.04)

安装openjdk-8

> sudo apt install -y openjdk-8-jdk

修改内核参数

> echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
> sudo sysctl -w vm.max_map_count=262144

下载elasticsearch deb包

> cd /usr/local/src
> sudo wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.4.4/elasticsearch-2.4.4.deb

安装elasticsearch

> cd /usr/local/src
> sudo dpkg -i elasticsearch-2.4.4.deb

配置elasticsearch

echo "network.host: $ELS_HOST" >> /etc/elasticsearch/elasticsearch.yml      # $ELS_HOST为elasticsearch部署的节点host ip

启动elasticsearch

sudo service elasticsearch start

查看elasticsearch日志

tail -f /var/log/elasticsearch/elasticsearch.log

Logstash部署

Linux Server Host部署方式 (Ubuntu 16.04)

安装redis-3.2.8

> cd /usr/local/src
> sudo wget http://download.redis.io/releases/redis-3.2.8.tar.gz /usr/local/src/
> cd /usr/local/src && sudo tar -zxf redis-3.2.8.tar.gz
> cd /usr/local/src/redis-3.2.8
> make && sudo make install
> cd utils && (echo;echo;echo;echo;echo;echo)|sudo ./install_server.sh      # 默认方式安装redis server 也可以去掉 '|' 前面的命令组 (echo;echo;echo;echo;echo;echo) ，部署redis server时手动设置参数
> sudo sed -i 's/127.0.0.1/0.0.0.0/g' /etc/redis/6379.conf       # 通过redis手机shipper日志，所以要配置远程可访问

安装Logstash

> cd /usr/local/src
> sudo wget https://download.elastic.co/logstash/logstash/packages/debian/logstash-2.4.1_all.deb
> sudo dpkg -i logstash-2.4.1_all.deb

配置Logstash indexer (收集nginx access log)

> sudo mkdir -p /opt/logstash/patterns
> sudo cp elk/logstash-2.4.1/nginx /opt/logstash/patterns/
> sudo cp elk/logstash-2.4.1/nginx_access_indexer.conf /etc/logstash/conf.d/

配置Logstash shipper (收集nginx access log)

>sudo cp elk/logstash-2.4.1/nginx_access_shipper.conf /etc/logstash/conf.d/

启动Logstash

> sudo service logstash start

nginx log format (源码位于elk/logstash-2.4.1/nginx_logformat.conf)

log_format access_log    '$remote_addr - $remote_user [$time_local] "$request" '
                         '$status $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" $http_x_forwarded_for '
                         '$request_time $upstream_response_time ';

access_log  logs/access.log access_log;

Kibana部署

Linux Server Host部署方式 (Ubuntu 16.04)

安装nginx、apache2-utils

> sudo apt install -y nginx apache2-utils

配置nginx、htpasswd验证

> sudo cp kibana.conf /etc/nginx/conf.d/
> sudo htpasswd -cb /opt/kibana/htpasswd.users kibana $KIBANA_PWD        # $KIBANA_PWD kibana用户密码

安装Kibana

> cd /usr/local/src
> sudo wget https://download.elastic.co/kibana/kibana/kibana-4.6.4-linux-x86_64.tar.gz
> sudo tar -zxf kibana-4.6.4-linux-x86_64.tar.gz
> sudo mv kibana-4.6.4-linux-x86_64 /opt/kibana

配置Kibana

> sudo echo "server.host: \"localhost\"" >> /opt/kibana/conf/kibana.conf
> sudo echo "elasticsearch.url: \"http:\/\/$ELS_HOST:9200\"

启动Kibana

> sudo mkdir /var/log/kibana
> su
> nohup /opt/kibana/bin/kibana > /var/log/kibana/kibana.log &

启动nginx

> sudo nginx -t
> sudo nginx -s reload

查看Kibala日志

> /var/log/kibana/kibana.log

浏览器访问

修改host，添加条目：$KIBANA_HOST elk.oschina.cc
浏览器访问地址：http://elk.oschina.cc
用户名：kibana
密码：$KIBANA_PWD设置的密码

转载于:https://my.oschina.net/atompi/blog/885881