使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践--转载

最新推荐文章于 2023-12-13 09:32:48 发布

aebdm757009

最新推荐文章于 2023-12-13 09:32:48 发布

阅读量153

点赞数

文章标签：大数据 java 运维

原文链接：http://www.cnblogs.com/davidwang456/p/5238973.html

版权

原文地址：https://wsgzao.github.io/post/elk/

另外可以参考：https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

前言

Elasticsearch + Logstash + Kibana（ELK）是一套开源的日志管理方案，分析网站的访问情况时我们一般会借助Google/百度/CNZZ等方式嵌入JS做数据统计，但是当网站访问异常或者被攻击时我们需要在后台分析如Nginx的具体日志，而Nginx日志分割/GoAccess/Awstats都是相对简单的单节点解决方案，针对分布式集群或者数据量级较大时会显得心有余而力不足，而ELK的出现可以使我们从容面对新的挑战。

Logstash：负责日志的收集，处理和储存
Elasticsearch：负责日志检索和分析
Kibana：负责日志的可视化

ELK(Elasticsearch + Logstash + Kibana)

更新记录

2015年08月31日 - 初稿

阅读原文 - http://wsgzao.github.io/post/elk/

扩展阅读

CentOS 7.x安装ELK(Elasticsearch+Logstash+Kibana) - http://www.chenshake.com/centos-install-7-x-elk-elasticsearchlogstashkibana/
Centos 6.5 安装nginx日志分析系统 elasticsearch + logstash + redis + kibana - http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582
logstash-forwarder and grok examples - https://www.ulyaoth.net/threads/logstash-forwarder-and-grok-examples.32413/
三斗室 - http://chenlinux.com/
elastic - https://www.elastic.co/guide
LTMP索引 - http://wsgzao.github.io/index/#LTMP

组件预览

JDK - http://www.oracle.com/technetwork/java/javase/downloads/index.html
Elasticsearch - https://www.elastic.co/downloads/elasticsearch
Logstash - https://www.elastic.co/downloads/logstash
Kibana - https://www.elastic.co/downloads/kibana
redis - http://redis.io/download

设置FQDN

创建SSL证书的时候需要配置FQDN


#修改hostname
cat /etc/hostname
elk

#修改hosts
cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1 10-10-87-19
10.10.87.19 elk.ooxx.com elk

#刷新环境
hostname -F /etc/hostname

#复查结果
hostname -f
elk.ooxx.com

hostname
elk

服务端

Java

cat /etc/redhat-release
CentOS release 6.5 (Final)

yum install java-1.7.0-openjdk
java -version

java version "1.7.0_85"
OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01)
OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)

Elasticsearch


#下载安装
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm
yum localinstall elasticsearch-1.7.1.noarch.rpm 

#启动相关服务
service elasticsearch start
service elasticsearch status

#查看Elasticsearch的配置文件
rpm -qc elasticsearch

/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/init.d/elasticsearch
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf

#查看端口使用情况
netstat -nltp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name 
tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN 1765/java 
tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN 1765/java 
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1509/sshd 
tcp 0 0 :::22 :::* LISTEN 1509/sshd 

#测试访问
curl -X GET http://localhost:9200/

Kibana


#下载tar包
wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
#解压
tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/
cd /usr/local/
mv kibana-4.1.1-linux-x64 kibana

#创建kibana服务
vi /etc/rc.d/init.d/kibana

#!/bin/bash
### BEGIN INIT INFO
# Provides: kibana
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Runs kibana daemon
# Description: Runs the kibana daemon as a non-root user
### END INIT INFO

# Process name
NAME=kibana
DESC="Kibana4"
PROG="/etc/init.d/kibana"

# Configure location of Kibana bin
KIBANA_BIN=/usr/local/kibana/bin

# PID Info
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/$NAME

# Configure User to run daemon process
DAEMON_USER=root
# Configure logging location
KIBANA_LOG=/var/log/kibana.log

# Begin Script
RETVAL=0

if [ `id -u` -ne 0 ]; then
 echo "You need root privileges to run this script"
 exit 1
fi

# Function library
. /etc/init.d/functions
 
start() {
 echo -n "Starting $DESC : "

pid=`pidofproc -p $PID_FILE kibana`
 if [ -n "$pid" ] ; then
 echo "Already running."
 exit 0
 else
 # Start Daemon
if [ ! -d "$PID_FOLDER" ] ; then
 mkdir $PID_FOLDER
 fi
daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &
 sleep 2
 pidofproc node > $PID_FILE
 RETVAL=$?
 [[ $? -eq 0 ]] && success || failure
echo
 [ $RETVAL = 0 ] && touch $LOCK_FILE
 return $RETVAL
 fi
}

reload()
{
 echo "Reload command is not implemented for this service."
 return $RETVAL
}

stop() {
 echo -n "Stopping $DESC : "
 killproc -p $PID_FILE $DAEMON
 RETVAL=$?
echo
 [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
}
 
case "$1" in
 start)
 start
;;
 stop)
 stop
 ;;
 status)
 status -p $PID_FILE $DAEMON
 RETVAL=$?
 ;;
 restart)
 stop
 start
 ;;
 reload)
reload
;;
 *)
# Invalid Arguments, print the following message.
 echo "Usage: $0 {start|stop|status|restart}" >&2
exit 2
 ;;
esac

#修改启动权限
chmod +x /etc/rc.d/init.d/kibana

#启动kibana服务
service kibana start
service kibana status

#查看端口