ELK stack
ELK stack 是以 Elasticsearch、Logstash、Kibana 三个开源软件为主的数据处理工具链。
在实时数据检索和分析场合,三者通常是配合使用,而且又先后归于Elastic.io公司名下,故有Elastic Stack简称。
ELK stack部署 (elasticsearch 2.4.4、Logstash 2.4.1、Kibana 4.6.4)
分为Linux Server Host部署方式和Docker部署方式。
初始化Linux Server Host
修改时区:
> sudo cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
> sudo dpkg-reconfigure -f noninteractive tzdata
Clone本仓库
> cd ~/
> git clone https://git.oschina.net/atompi/elk-docker.git
> cd ~/elk-docker
修改apt源为阿里云:
> sudo cp sources.list /etc/apt/sources.list
> sudo apt-get update
elasticsearch部署 (单节点)
Linux Server Host部署方式 (Ubuntu 16.04)
安装openjdk-8
> sudo apt install -y openjdk-8-jdk
修改内核参数
> echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
> sudo sysctl -w vm.max_map_count=262144
下载elasticsearch deb包
> cd /usr/local/src
> sudo wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.4.4/elasticsearch-2.4.4.deb
安装elasticsearch
> cd /usr/local/src
> sudo dpkg -i elasticsearch-2.4.4.deb
配置elasticsearch
echo "network.host: $ELS_HOST" >> /etc/elasticsearch/elasticsearch.yml # $ELS_HOST为elasticsearch部署的节点host ip
启动elasticsearch
sudo service elasticsearch start
查看elasticsearch日志
tail -f /var/log/elasticsearch/elasticsearch.log
Logstash部署
Linux Server Host部署方式 (Ubuntu 16.04)
安装redis-3.2.8
> cd /usr/local/src
> sudo wget http://download.redis.io/releases/redis-3.2.8.tar.gz /usr/local/src/
> cd /usr/local/src && sudo tar -zxf redis-3.2.8.tar.gz
> cd /usr/local/src/redis-3.2.8
> make && sudo make install
> cd utils && (echo;echo;echo;echo;echo;echo)|sudo ./install_server.sh # 默认方式安装redis server 也可以去掉 '|' 前面的命令组 (echo;echo;echo;echo;echo;echo) ,部署redis server时手动设置参数
> sudo sed -i 's/127.0.0.1/0.0.0.0/g' /etc/redis/6379.conf # 通过redis手机shipper日志,所以要配置远程可访问
安装Logstash
> cd /usr/local/src
> sudo wget https://download.elastic.co/logstash/logstash/packages/debian/logstash-2.4.1_all.deb
> sudo dpkg -i logstash-2.4.1_all.deb
配置Logstash indexer (收集nginx access log)
> sudo mkdir -p /opt/logstash/patterns
> sudo cp elk/logstash-2.4.1/nginx /opt/logstash/patterns/
> sudo cp elk/logstash-2.4.1/nginx_access_indexer.conf /etc/logstash/conf.d/
配置Logstash shipper (收集nginx access log)
>sudo cp elk/logstash-2.4.1/nginx_access_shipper.conf /etc/logstash/conf.d/
启动Logstash
> sudo service logstash start
nginx log format (源码位于elk/logstash-2.4.1/nginx_logformat.conf)
log_format access_log '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $http_x_forwarded_for '
'$request_time $upstream_response_time ';
access_log logs/access.log access_log;
Kibana部署
Linux Server Host部署方式 (Ubuntu 16.04)
安装nginx、apache2-utils
> sudo apt install -y nginx apache2-utils
配置nginx、htpasswd验证
> sudo cp kibana.conf /etc/nginx/conf.d/
> sudo htpasswd -cb /opt/kibana/htpasswd.users kibana $KIBANA_PWD # $KIBANA_PWD kibana用户密码
安装Kibana
> cd /usr/local/src
> sudo wget https://download.elastic.co/kibana/kibana/kibana-4.6.4-linux-x86_64.tar.gz
> sudo tar -zxf kibana-4.6.4-linux-x86_64.tar.gz
> sudo mv kibana-4.6.4-linux-x86_64 /opt/kibana
配置Kibana
> sudo echo "server.host: \"localhost\"" >> /opt/kibana/conf/kibana.conf
> sudo echo "elasticsearch.url: \"http:\/\/$ELS_HOST:9200\"
启动Kibana
> sudo mkdir /var/log/kibana
> su
> nohup /opt/kibana/bin/kibana > /var/log/kibana/kibana.log &
启动nginx
> sudo nginx -t
> sudo nginx -s reload
查看Kibala日志
> /var/log/kibana/kibana.log
浏览器访问
修改host,添加条目:$KIBANA_HOST elk.oschina.cc
浏览器访问地址:http://elk.oschina.cc
用户名:kibana
密码:$KIBANA_PWD设置的密码