Elk实时日志分析平台5.0版本源码安装配置

最新推荐文章于 2024-08-23 09:47:52 发布
最新推荐文章于 2024-08-23 09:47:52 发布
阅读量10w+ 收藏 2
点赞数
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq942477618/article/details/53518372
版权

最近有朋友老说用RPM安装ELK教程不适用，不通用；很多生产环境不联网，操作系统版本也不同；叫我弄个源码安装的教程。所以就整理一个ELK5.0版本源码安装过程，水平有限，凑合着看！！最后附上安装包


一、配置JAVA环境变量

# mkdir /usr/local/java/ –p
# cd /usr/local/java/
# tar zxvf /data/elk5.0/jdk-8u111-linux-x64.tar.gz
 
# cat >>/etc/profile<<EOF
 
export JAVA_HOME=/usr/local/java/jdk1.8.0_111
export PATH=$PATH:$JAVA_HOME/bin
exportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
EOF
 
# source /etc/profile
# java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixedmode)


 

二、安装elasticsearch

# mkdir  /data/PRG/-p
# cd /data/PRG/
# tar zxvf /data/elk5.0/elasticsearch-5.0.0.tar.gz
# mv elasticsearch-5.0.0 elasticsearch
# useradd elasticsearch -s /sbin/nologin
# chown elasticsearch. elasticsearch /data/PRG/elasticsearch/


添加启动脚本

vi /etc/init.d/elasticsearch

 

#!/bin/sh
#
# elasticsearch <summary>
#
# chkconfig:   2345 80 20
# description: Starts and stops a single elasticsearch instance on this system
#


### BEGIN INIT INFO
# Provides: Elasticsearch
# Required-Start: $network $named
# Required-Stop: $network $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: This service manages the elasticsearch daemon
# Description: Elasticsearch is a very scalable, schema-free and high-performance search solution supporting multi-tenancy and near realtime search.
### END INIT INFO


#
# init.d / servicectl compatibility (openSUSE)
#
if [ -f /etc/rc.status ]; then
    . /etc/rc.status
    rc_reset
fi


#
# Source function library.
#
if [ -f /etc/rc.d/init.d/functions ]; then
    . /etc/rc.d/init.d/functions
fi






# Sets the default values for elasticsearch variables used in this script
export JAVA_HOME=/usr/local/java/jdk1.8.0_111
ES_USER="elasticsearch"
ES_GROUP="elasticsearch"
name=elasticsearch
ES_HOME="/data/PRG/elasticsearch"
MAX_OPEN_FILES=65536
MAX_MAP_COUNT=262144
LOG_DIR="$ES_HOME/log/"
DATA_DIR="$ES_HOME/lib/"
CONF_DIR="$ES_HOME/config"
mkdir -p $LOG_DIR
chown -R elasticsearch.elasticsearch $ES_HOME


PID_DIR="$ES_HOME/log"


# Source the default env file
ES_ENV_FILE="/etc/sysconfig/elasticsearch"
if [ -f "$ES_ENV_FILE" ]; then
    . "$ES_ENV_FILE"
fi


# CONF_FILE setting was removed
if [ ! -z "$CONF_FILE" ]; then
    echo "CONF_FILE setting is no longer supported. elasticsearch.yml must be placed in the config directory and cannot be renamed."
    exit 1
fi


exec="$ES_HOME/bin/elasticsearch"
prog="elasticsearch"
pidfile="$PID_DIR/${prog}.pid"


export ES_HEAP_SIZE
export ES_HEAP_NEWSIZE
export ES_DIRECT_SIZE
export ES_JAVA_OPTS
export ES_GC_LOG_FILE
export ES_STARTUP_SLEEP_TIME
export JAVA_HOME
export ES_INCLUDE
ulimit -n $MAX_OPEN_FILES
lockfile=$ES_HOME/log/$prog


# backwards compatibility for old config sysconfig files, pre 0.90.1
if [ -n $USER ] && [ -z $ES_USER ] ; then
   ES_USER=$USER
fi


checkJava() {
    if [ -x "$JAVA_HOME/bin/java" ]; then
        JAVA="$JAVA_HOME/bin/java"
    else
        JAVA=`which java`
    fi


    if [ ! -x "$JAVA" ]; then
        echo "Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME"
        exit 1
    fi
}


start() {
    checkJava
    [ -x $exec ] || exit 5
    if [ -n "$MAX_LOCKED_MEMORY" -a -z "$ES_HEAP_SIZE" ]; then
        echo "MAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set"
        return 7
    fi
    if [ -n "$MAX_OPEN_FILES" ]; then
        ulimit -n $MAX_OPEN_FILES
    fi
    if [ -n "$MAX_LOCKED_MEMORY" ]; then
        ulimit -l $MAX_LOCKED_MEMORY
    fi
    if [ -n "$MAX_MAP_COUNT" -a -f /proc/sys/vm/max_map_count ]; then
        sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT
    fi
    export ES_GC_LOG_FILE


    # Ensure that the PID_DIR exists (it is cleaned at OS startup time)
    if [ -n "$PID_DIR" ] && [ ! -e "$PID_DIR" ]; then
        mkdir -p "$PID_DIR" && chown "$ES_USER":"$ES_GROUP" "$PID_DIR"
    fi
    if [ -n "$pidfile" ] && [ ! -e "$pidfile" ]; then
        touch "$pidfile" && chown "$ES_USER":"$ES_GROUP" "$pidfile"
    fi


    cd $ES_HOME
    echo -n $"Starting $prog: "
    # if not running, start it up here, usually something like "daemon $exec"
    daemon --user $ES_USER  --pidfile $pidfile $exec -p $pidfile -d


    #daemon --user $ES_USER --pidfile $pidfile $exec -p $pidfile -d -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.conf=$CONF_DIR
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}


stop() {
    echo -n $"Stopping $prog: "
    # stop it here, often "killproc $prog"
    killproc -p $pidfile -d 86400 $prog
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}


restart() {
    stop
    start
}


reload() {
    restart
}


force_reload() {
    restart
}


rh_status() {
    # run checks to determine if the service is running or use generic status
    status -p $pidfile $prog
}


rh_status_q() {
    rh_status >/dev/null 2>&1
}




case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
exit $?


# chmod +x /etc/init.d/elasticsearch
 
# /etc/init.d/elasticsearch start
 
# /etc/init.d/elasticsearch status
elasticsearch (pid 20895) is running...
# netstat -ntlp |grep 9[2-3]00
tcp        0     0 :::9300                    :::*                        LISTEN      20895/java         
tcp        0      0 :::9200                     :::*                        LISTEN      20895/java




三、配置elasticsearch 

内存低于2G，需要修改jvm配置

# vim /data/PRG/elasticsearch/config/jvm.options
-Xms512m
-Xmx512m

 

# cat /data/PRG/elasticsearch/config/elasticsearch.yml|grep -v '#'
network.host: 0.0.0.0           ###开启监听地址，
action.auto_create_index:.security,.monitoring*,.watches,.triggered_watches,.watcher-history*
####以下模块视情况是否开启
xpack.security.enabled: true          ####开启用户认证
xpack.monitoring.enabled: true
xpack.graph.enabled: true
xpack.watcher.enabled: true
xpack.security.authc.realms:     ####用户认证模式，ldap、file、pki、ActiveDirectory等
    file1:
      type: file
      order: 0


四、安装logstash

# cd /data/PRG/
# tar zxvf /data/elk5.0/logstash-5.0.0.tar.gz
# mv logstash-5.0.0 logstash
# useradd logstash -s /sbin/nologin
# chown logstash. logstash /data/PRG/logstash


         添加启动脚本

vim /etc/init.d/logstash

#!/bin/sh
# Init script for logstash
# Maintained by Elasticsearch
# Generated by pleaserun.
# Implemented based on LSB Core 3.1:
#   * Sections: 20.2, 20.3
#
### BEGIN INIT INFO
# Provides:          logstash
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description:
# Description:        Starts Logstash as a daemon.
### END INIT INFO

PATH=/sbin:/usr/sbin:/bin:/usr/bin:/data/PRG/logstash/bin
export PATH

if [ `id -u` -ne 0 ]; then
   echo "You need root privileges to run this script"
   exit 1
fi

name=logstash

LS_USER=logstash
LS_GROUP=logstash
LS_HOME=/data/PRG/logstash
LS_HEAP_SIZE="1g"
LS_LOG_DIR=/data/PRG/logstash/logs
LS_LOG_FILE="${LS_LOG_DIR}/$name.log"
pidfile="${LS_LOG_DIR}/$name.pid"
LS_CONF_DIR=/data/PRG/logstash/conf.d
LS_OPEN_FILES=16384
LS_NICE=19
KILL_ON_STOP_TIMEOUT=${KILL_ON_STOP_TIMEOUT-0} #default value is zero to this variable but could be updated by user request
LS_OPTS=""


[ -r /etc/default/$name ] && . /etc/default/$name
[ -r /etc/sysconfig/$name ] && . /etc/sysconfig/$name

program=$LS_HOME/bin/logstash
args=" -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

quiet() {
  "$@" > /dev/null 2>&1
  return $?
}

start() {

  LS_JAVA_OPTS="${LS_JAVA_OPTS} -Djava.io.tmpdir=${LS_HOME}"
  HOME=${LS_HOME}
  export PATH HOME LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING LS_GC_LOG_FILE

  # chown doesn't grab the suplimental groups when setting the user:group - so we have to do it for it.
  # Boy, I hope we're root here.
  SGROUPS=$(id -Gn "$LS_USER" | tr " " "," | sed 's/,$//'; echo '')

  if [ ! -z $SGROUPS ]
  then
        EXTRA_GROUPS="--groups $SGROUPS"
  fi

  # set ulimit as (root, presumably) first, before we drop privileges
  ulimit -n ${LS_OPEN_FILES}

  # Run the program!
  nice -n ${LS_NICE} chroot --userspec $LS_USER:$LS_GROUP $EXTRA_GROUPS / sh -c "
    cd $LS_HOME
    ulimit -n ${LS_OPEN_FILES}
    $program $args > ${LS_LOG_DIR}/$name.stdout" 2> "${LS_LOG_DIR}/$name.err" &

  # Generate the pidfile from here. If we instead made the forked process
  # generate it there will be a race condition between the pidfile writing
  # and a process possibly asking for status.
  echo $! > $pidfile

  echo "$name started."
  return 0
}

stop() {
  # Try a few times to kill TERM the program
  if status ; then
    pid=`cat "$pidfile"`
    echo "Killing $name (pid $pid) with SIGTERM"
    ps -ef |grep $pid |grep -v 'grep' |awk '{print $2}' | xargs kill -9
    # Wait for it to exit.
    for i in 1 2 3 4 5 6 7 8 9 ; do
      echo "Waiting $name (pid $pid) to die..."
      status || break
      sleep 1
    done
    if status ; then
      if [ $KILL_ON_STOP_TIMEOUT -eq 1 ] ; then
        echo "Timeout reached. Killing $name (pid $pid) with SIGKILL. This may result in data loss."
        kill -KILL $pid
        echo "$name killed with SIGKILL."
      else
        echo "$name stop failed; still running."
        return 1 # stop timed out and not forced
      fi
    else
      echo "$name stopped."
    fi
  fi
}

status() {
  if [ -f "$pidfile" ] ; then
    pid=`cat "$pidfile"`
    if kill -0 $pid > /dev/null 2> /dev/null ; then
      # process by this pid is running.
      # It may not be our pid, but that's what you get with just pidfiles.
      # TODO(sissel): Check if this process seems to be the same as the one we
      # expect. It'd be nice to use flock here, but flock uses fork, not exec,
      # so it makes it quite awkward to use in this case.
      return 0
    else
      return 2 # program is dead but pid file exists
    fi
  else
    return 3 # program is not running
  fi
}

configtest() {
  # Check if a config file exists
  if [ ! "$(ls -A ${LS_CONF_DIR}/* 2> /dev/null)" ]; then
    echo "There aren't any configuration files in ${LS_CONF_DIR}"
    return 1
  fi

  HOME=${LS_HOME}
  export PATH HOME

  test_args="-t -f ${LS_CONF_DIR} ${LS_OPTS} "
  $program ${test_args}
  [ $? -eq 0 ] && return 0
  # Program not configured
  return 6
}

case "$1" in
  start)
    status
    code=$?
    if [ $code -eq 0 ]; then
      echo "$name is already running"
    else
      start
      code=$?
    fi
    exit $code
    ;;
  stop) stop ;;
  force-stop) force_stop ;;
  status)
    status
    code=$?
    if [ $code -eq 0 ] ; then
      echo "$name is running"
    else
      echo "$name is not running"
    fi
    exit $code
    ;;
  reload) reload ;;
  restart)
    stop && start
    ;;
  check)
    configtest
    exit $?
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|status|restart|check}" >&2
    exit 3
  ;;
esac

exit $?


# chmod +x /etc/init.d/logstash
# /etc/init.d/logstash start
# /etc/init.d/logstash status  
logstash is running
 
# netstat -ntlp|grep 9600
tcp        0      0 :::9600                     :::*                        LISTEN      10141/java


 

五、配置logstash

# cat /data/PRG/logstash/config/logstash.yml  |grep -v '#'

http.host: "0.0.0.0"       ###开启监听地址

 

ngin日志收集

# cat /data/PRG/logstash/conf.d/filter.conf
input {
  beats {
    port => 10200
  }
}
 
filter {
    grok {     
            match=> {
               message => "%{IPORHOST:remote_addr} , \[%{HTTPDATE:timestamp}\], %{IPORHOST:http_host} , \"%{WORD:http_verb}(?:%{PATH:baseurl}\?%{NOTSPACE:params}|%{DATA:raw_http_request})\" , %{NUMBER:http_status_code}, %{NUMBER:bytes_read} , %{QS:referrer} , %{QS:agent} ,\"%{IPORHOST:client_ip}, %{IPORHOST:proxy_server}\" , - , - , - ,%{IPORHOST:server_ip} , %{BASE10NUM:request_duration}"
            }
 
            match=> {
               message => "%{IPORHOST:remote_addr} , \[%{HTTPDATE:timestamp}\], %{IPORHOST:http_host} , \"%{WORD:http_verb}(?:%{PATH:baseurl}\?%{NOTSPACE:params}|%{DATA:raw_http_request})\" , %{NUMBER:http_status_code}, %{NUMBER:bytes_read} , %{QUOTEDSTRING:referrer} , %{QS:agent} ,\"%{IPORHOST:client_ip}, %{IPORHOST:proxy_server}\" ,%{IPORHOST}:%{INT} , %{INT} , %{BASE10NUM} , %{IPORHOST} ,%{BASE10NUM:request_duration}"
            }
    }
 
}
output {
 elasticsearch {
   hosts =>["192.168.62.200:9200"]
   index =>"operation-%{+YYYY.MM.dd}"
   document_type=> "nginx2"
   user => 'admin'              #### elasticsearch的用户名，用X-PACK插件创建
   password =>'kbsonlong'       #### elasticsearch的用户名
  }
 stdout { codec =>rubydebug }
}


 

 

 

六、安装kibana

# cd /data/PRG/
# tar zxvf /data/elk5.0/kibana-5.0.0-linux-x86_64.tar.gz
# mv kibana-5.0.0-linux-x86_64 kibana
# useradd kibana –s /sbin/nologin
# chown kibana. kibana /data/PRG/kibana

添加启动脚本

# vim /etc/init.d/kibana

 

#!/bin/sh
# Init script for kibana
# Maintained by 
# Generated by pleaserun.
# Implemented based on LSB Core 3.1:
#   * Sections: 20.2, 20.3
#
### BEGIN INIT INFO
# Provides:          kibana
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: 
# Description:       Kibana
### END INIT INFO

PATH=/sbin:/usr/sbin:/bin:/usr/bin
export PATH

KIBANA_HOME=/data/PRG/kibana
name=kibana
program=$KIBANA_HOME/bin/kibana
args=''
pidfile="$KIBANA_HOME/logs/$name.pid"
LOG_HOME="$KIBANA_HOME/logs"

[ -r /etc/default/$name ] && . /etc/default/$name
[ -r /etc/sysconfig/$name ] && . /etc/sysconfig/$name

[ -z "$nice" ] && nice=0

trace() {
  logger -t "/etc/init.d/kibana" "$@"
}

emit() {
  trace "$@"
  echo "$@"
}

start() {

  # Ensure the log directory is setup correctly.
  [ ! -d "$LOG_HOME" ] && mkdir "$LOG_HOME"
  chmod 755 "$LOG_HOME"


  # Setup any environmental stuff beforehand
  

  # Run the program!
  
  #chroot --userspec "$user":"$group" "$chroot" sh -c "
    
  $program $args >> $LOG_HOME/kibana.stdout 2>> $LOG_HOME/kibana.stderr &

  # Generate the pidfile from here. If we instead made the forked process
  # generate it there will be a race condition between the pidfile writing
  # and a process possibly asking for status.
  echo $! > $pidfile

  emit "$name started"
  return 0
}

stop() {
  # Try a few times to kill TERM the program
  if status ; then
    pid=$(cat "$pidfile")
    echo "Killing $name (pid $pid) with SIGTERM"
    ps -ef |grep $pid |grep -v 'grep' |awk '{print $2}' | xargs kill -9
    # Wait for it to exit.
    for i in 1 2 3 4 5 ; do
      trace "Waiting $name (pid $pid) to die..."
      status || break
      sleep 1
    done
    if status ; then
      if [ "$KILL_ON_STOP_TIMEOUT" -eq 1 ] ; then
        trace "Timeout reached. Killing $name (pid $pid) with SIGKILL.  This may result in data loss."
        kill -KILL $pid
        emit "$name killed with SIGKILL."
      else
        emit "$name stop failed; still running."
      fi
    else
      emit "$name stopped."
    fi
  fi
}

status() {
  if [ -f "$pidfile" ] ; then
    pid=$(cat "$pidfile")
    if ps -p $pid > /dev/null 2> /dev/null ; then
      # process by this pid is running.
      # It may not be our pid, but that's what you get with just pidfiles.
      # TODO(sissel): Check if this process seems to be the same as the one we
      # expect. It'd be nice to use flock here, but flock uses fork, not exec,
      # so it makes it quite awkward to use in this case.
      return 0
    else
      return 2 # program is dead but pid file exists
    fi
  else
    return 3 # program is not running
  fi
}




case "$1" in
  force-start|start|stop|status|restart)
    trace "Attempting '$1' on kibana"
    ;;
esac

case "$1" in
  force-start)
    PRESTART=no
    exec "$0" start
    ;;
  start)
    status
    code=$?
    if [ $code -eq 0 ]; then
      emit "$name is already running"
      exit $code
    else
      start
      exit $?
    fi
    ;;
  stop) stop ;;
  status)
    status
    code=$?
    if [ $code -eq 0 ] ; then
      emit "$name is running"
    else
      emit "$name is not running"
    fi
    exit $code
    ;;
  restart)
    
    stop && start
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|force-start|stop|force-start|force-stop|status|restart}" >&2
    exit 3
  ;;
esac

exit $?


# chmod +x /etc/init.d/kibana
# /etc/init.d/kibana start
# /etc/init.d/kibana status
# netstat -ntlp |grep 5601
tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      13052/node


七、配置kibana

# cat /data/PRG/kibana/config/kibana.yml |grep -v '#'                                     
server.host: "0.0.0.0"
 
####以下模块视情况是否开启
xpack.security.enabled: true
xpack.monitoring.enabled: true
xpack.graph.enabled: true
xpack.reporting.enabled: true


 

 

八、x-pack插件安装

 

# /data/PRG/kibana/bin/kibana-plugin install file:///root/x-pack-5.0.0.zip

 

# /data/PRG/elasticsearch/bin/elasticsearch-plugin install file:///root/x-pack-5.0.0.zip

 

离线安装x-pack要修改用户脚本，默认创建用户配置文件在/etc/elasticsearch/x-pack目录

# vim /data/PRG/elasticsearch/bin/x-pack/users

否则在创建用户的时候提示/etc/elasticsearch/x-pack/users…tmp不存在

# mkdir /etc/elasticsearch/x-pack/

# chown elasticsearch. elasticsearch /etc/elasticsearch/x-pack/-R

 

九、           x-pack管理用户

1、          添加用户

 

# cd /data/PRG/elasticsearch 
# bin/x-pack/users useradd admin -p kbsonlong -rsuperuser


 

2、          查看用户

# /data/PRG/elasticsearch/bin/x-pack/users list
admin          :superuser
test           : -                ###创建用户时没有添加-r参数，所以没有用户角色


3、          测试用户登录

# curl http://localhost:9200/_xpack/ --useradmin:kbsonlong 
{"build":{"hash":"7763f8e","date":"2016-10-26T04:51:59.202Z"},"license":{"uid":"06a82587-66ac-4d4a-90c4-857d9ca7f3bc","type":"trial","mode":"trial","status":"active","expiry_date_in_millis":1483753731066},"features":{"graph":{"description":"GraphData Exploration for the ElasticStack","available":true,"enabled":true},"monitoring":{"description":"Monitoringfor the ElasticStack","available":true,"enabled":true},"security":{"description":"Securityfor the ElasticStack","available":true,"enabled":true},"watcher":{"description":"Alerting,Notification and Automation for the ElasticStack","available":true,"enabled":true}},"tagline":"Youknow, for X"}


 

4、          删除用户

# /data/PRG/elasticsearch/bin/x-pack/users userdel test
 
# /data/PRG/elasticsearch/bin/x-pack/users list
admin          :superuser


十、安装filebeat

# cd /data/PRG
# tar zxvf / data/elk5.0/filebeat-5.0.0-linux-x86_64.tar.gz
# mv filebeat-5.0.0-linux-x86_64  filebeat


配置启动脚本

# vim /etc/init.d/filebeat

 

#!/bin/bash
#
# filebeat          filebeat shipper
#
# chkconfig: 2345 98 02
#

### BEGIN INIT INFO
# Provides:          filebeat
# Required-Start:    $local_fs $network $syslog
# Required-Stop:     $local_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Sends log files to Logstash or directly to Elasticsearch.
# Description:       filebeat is a shipper part of the Elastic Beats 
#					 family. Please see: https://www.elastic.co/products/beats
### END INIT INFO



PATH=/usr/bin:/sbin:/bin:/usr/sbin
export PATH

[ -f /etc/sysconfig/filebeat ] && . /etc/sysconfig/filebeat
pidfile=${PIDFILE-/data/PRG/filebeat/filebeat.pid}
agent=${PB_AGENT-/data/PRG/filebeat/filebeat}
args="-c /data/PRG/filebeat/filebeat.yml"
test_args="-e -configtest"
wrapper="filebeat-god"
wrapperopts="-r / -n -p $pidfile"
RETVAL=0

# Source function library.
. /etc/rc.d/init.d/functions

# Determine if we can use the -p option to daemon, killproc, and status.
# RHEL < 5 can't.
if status | grep -q -- '-p' 2>/dev/null; then
    daemonopts="--pidfile $pidfile"
    pidopts="-p $pidfile"
fi

test() {
	$agent $args $test_args
}

start() {
    echo -n $"Starting filebeat: "
	test
	if [ $? -ne 0 ]; then
		echo
		exit 1
	fi
    daemon $daemonopts $wrapper $wrapperopts -- $agent $args
    RETVAL=$?
    echo
    return $RETVAL
}

stop() {
    echo -n $"Stopping filebeat: "
    killproc $pidopts $wrapper
    RETVAL=$?
    echo
    [ $RETVAL = 0 ] && rm -f ${pidfile}
}

restart() {
	test
	if [ $? -ne 0 ]; then
		return 1
	fi
    stop
    start
}

rh_status() {
    status $pidopts $wrapper
    RETVAL=$?
    return $RETVAL
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        start
    ;;
    stop)
        stop
    ;;
    restart)
        restart
    ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
    ;;
    status)
        rh_status
    ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart}"
        exit 1
esac

exit $RETVAL

配置filebeat

# cat filebeat/filebeat.yml |grep -v '#'
filebeat.prospectors:
- input_type: log
 
 paths:
    -/tmp/nginx.log
output.logstash:
 enabled: true
 hosts: ["localhost:10200"]


 

启动filebeat

# /etc/init.d/filebeat5 start
Starting filebeat: 2016/12/0807:18:37.177631 beat.go:264: INFO Home path: [/data/PRG/filebeat] Config path:[/data/PRG/filebeat] Data path: [/data/PRG/filebeat/data] Logs path:[/data/PRG/filebeat/logs]
2016/12/08 07:18:37.177681 beat.go:174:INFO Setup Beat: filebeat; Version: 5.0.0
2016/12/08 07:18:37.177760 logstash.go:90:INFO Max Retries set to: 3
2016/12/08 07:18:37.177828 outputs.go:106:INFO Activated logstash as output plugin.
2016/12/08 07:18:37.177912 publish.go:291:INFO Publisher name: operation
2016/12/08 07:18:37.178158 async.go:63:INFO Flush Interval set to: 1s
2016/12/08 07:18:37.178170 async.go:64:INFO Max Bulk Size set to: 2048
Config OK
                                                          [  OK  ]
 
# /etc/init.d/filebeat5 status
filebeat-god (pid  7365) is running...
 
# ps -ef |grep filebeat
root     7405     1  0 15:18 pts/1    00:00:00 filebeat-god -r / -n -p/data/PRG/filebeat/filebeat.pid -- /data/PRG/filebeat/filebeat -c/data/PRG/filebeat/filebeat.yml
root     7406  7405  0 15:18 pts/1    00:00:00 /data/PRG/filebeat/filebeat -c/data/PRG/filebeat/filebeat.yml



附上安装源码包，包括x-pack、beat等

百度云盘http://pan.baidu.com/s/1skT4zCx




蜷缩的蜗牛
关注
ELK日志分析系统概述及部署
Kzxjjjs的博客
08-02 527
ElasticSearch是基于Lucene（一个全文检索引擎的架构）开发的分布式存储检索引擎，用来存储各类日志。Elasticsearch 是用 Java 开发的，可通过 RESTful Web 接口，让用户可以通过浏览器与 Elasticsearch 通信。Elasticsearch是一个实时的、分布式的可扩展的搜索引擎，允许进行全文、结构化搜索，它通常用于索引和搜索大容量的日志数据，也可用于搜索许多不同类型的文档。②Kiabana。
ELK 日志分析系统的部署
Moshizhu的博客
07-04 361
1.我们需要收集哪些日志？ 收集系统日志：为监控做准备，要收集tomcat系统日志，tomcat所在节点的日志 收集服务日志：比如数据库mysql，收集慢查询日志、错误日志、普通日志，要收集tomcat服务日志。 收集业务日志（业务日志必须收集）：业务口子日志在log4j，log4j是由java环境开发的，跑在tomcat上 2.日志收集后，如何展示（可视化）？ kibana grafana:可以以地图的方式展示，功能更全面一些，界面会好看一些 3.日志收集展示出来后，怎么使用？ 用于给大数据分析，作为立体
ELK实时日志分析平台 --（1）
weixin_47252600的博客
01-14 341
ELK实时日志分析平台介绍
ELK进阶-安全认证设置流程介绍
最新发布
Young_深情不及里子的博客
08-23 1299
在数字化转型的浪潮中，企业对于数据的依赖日益增强，而日志数据作为业务运行的“脉络图”，其重要性不言而喻。ELFK（Elasticsearch, Logstash, Filebeat, Kibana）作为一套强大的日志监控系统，为企业提供了从数据采集、处理到可视化的完整解决方案。然而，随着数据价值的提升，安全问题也日益凸显，如何在享受ELFK带来的高效与便捷的同时，确保数据的安全与合规，成为了企业必须面对的挑战。安全认证，作为守护数据安全的第一道防线，其在ELFK系统中的必要性不容忽视。
开源实时日志分析ELK平台部署
Mungo的专栏
08-01 4048
ELK版本：5.4.3 下载：https://www.elastic.co/cn/downloads 环境： CentOS7 JDK 1.8 Linux用户：mungo ElasticSearch http://10.81.1.6:9200/ Logstash Logstash 项目诞生于 2009 年 8 月 2 日。其作者是世界著名的运维工程师乔丹西塞
搭建ELK日志分析系统
热门推荐
菲宇运维
07-17 9万+
ELK Stack 是Elasticsearch、Logstash、Kiban三个开源软件的组合。在实时数据检索和分析场合，三者通常是配合共用，而且又都先后归于 Elastic.co 公司名下，故有此简称。 ELK Stack成为机器数据分析，或者说实时日志处理领域，开源界的第一选择。和传统的日志处理方案相比，ELK Stack 具有如下几个优点： • 处理方式灵活。Elasticsearch 是...
ELK日志分析平台(三) (Logstash简介+安装+常用插件示例)
weixin_45649763的博客
05-26 548
文章目录Logstash简介Logstash安装与配置file 输出插件标准输入到标准输出标准输入到文件elasticsearch输出插件标准输入到es主机file输入插件把文件内容输出到es主机logstash如何区分设备、文件名、文件的不同版本syslog输入插件多行过滤插件多行过滤插件过滤远程主机的日志grok过滤插件apache服务日志过滤实战 Logstash简介 Logstash是一个开源的服务器端数据处理管道，它的主要作用是做数据采集。 logstash拥有200多个插件，能够同时从多个来源采
ELK日志分析集群部署
wwwu1998的博客
07-13 772
ELK日志分析集群部署
ELK日志分析系统
ynyysn的博客
04-27 503
一、ELK前言 1）我们需要收集哪些日志？ 收集系统日志：为监控做准备，要收集tomcat系统日志，tomcat所在节点的日志 收集服务日志：比如数据库mysql，收集慢查询日志、错误日志、普通日志，要收集tomcat服务日志。 收集业务日志（业务日志必须收集）：业务口子日志在log4j，log4j是由java环境开发的，跑在tomcat上 2）日志收集后，如何展示（可视化）？ kibana grafana:可以以地图的方式展示，功能更全面一些，界面会好看一些 3）日志收集展示出来后，怎么使用？ 用于给大数
elk cloner 源码
01-30
1982年的1月31日，15岁的高中生里奇·斯克伦塔写了世界上第一个被广泛传播的个人电脑病毒“elk cloner”。这个病毒完全没有恶意，靠磁盘传播，电脑被感染后每被启动50次就会显示一首名为“elk cloner: the program with a personality”的诗。 以下为这首诗的全文： Elk Cloner: The program with a personality It will get on all your disks It will infiltrate your chips Yes, it's Cloner! It will stick to you like glue It will modify RAM too Send in the Cloner!
elk源码安装
wzz_ccr的博客
01-23 2053
[root@localhostelk]# ls elasticsearch-2.2.1.tar.gzjdk-8u101-linux-i586.gz kibana-4.4.2-linux-x64.tar.gz   logstash-2.2.2.tar.gz   解压jdk1.8 [root@localhostelk]# tar xvf jdk-8u101-linux-i586.gz
ELK
03-28
在最初通过检查elastic.co上的文档来设置ELK堆栈之后，在浏览文档时先向前移动，而不是手动执行kibana和logstash。 我决定尝试查找具有我可以修改的基本模板的现有docker映像或存储库。 因此，我在上找到了基本的ELK堆栈设置
搭建ELK日志系统（笔记、整理）
qq_31908235的博客
06-02 301
系统环境 系统环境为：CentOS7 java版本为：1.8 elasticsearch版本为：elasticsearch-6.5.4 kibana版本为：kibana-6.5.4 logstash版本为：logstash-6.5.4 内部资源原本打算组件ELK+kafuka架构因时间原因暂时放弃了， 组建ELK+kafuka的资源可能不够。。。 组建简单的ELK日志系统还是可以的 资源下载地址： 链接：https://pan.baidu.com/s/1H1FMLgt_4es5oKuWKX5Tmw 提取码：
源码部署ELK日志分析管理系统
每天都要开心呀(#^.^#)
09-18 975
ELK是elasticsearch、logstash、kibana三个开源软件的缩写，用于日志收集查看分析
ELK源码6.6.1部署学习实践
Gun&rose
06-19 403
日志收集，是运维工作中很重要的一环。不管是针对线上的系统服务，还是基础架构上的网络设备，有一套完整合适的日志收集系统，是必不可少的，这将成为之后遇到故障问题，提供一个有效的依据和排查手段。 目前已接触的日志收集系统： Windows下的Kiwi syslog sever Linux下的rsyslog，Graylog，ELK，前者是老牌日志收集工具，后两者简单看了一下介绍，日志收集只是其中一个功能...
ELK 源码详细安装步骤
WEN_BK的博客
05-20 1000
一、安装准备环境 1，jdk 1.8及以上的版本；2，elasticsearch-7.7.0 ；3，logstash-7.7.0；4，kibana-7.7.0；5，CentOS Linux release 7.5.1804 (Core) 我用三台主机搭建，收集我之前docker搭建好的lamp的日志信息 192.168.116.128：logstash 192.168.116.129：elasticsearch 192.168.116.130：kibana 防火墙与selinux关掉，也可以自己添加规则 二
Logstash安装-源码
weixin_34088583的博客
11-12 668
Logstash下载有点麻烦，由于国内到elk官网速度很慢，下载过程中容易中断。 所以，可以通过有断点续传的工具（迅雷）进行下载好，再进行安装使用。 系统环境： [root@test ~]# cat /etc/issue CentOS release 6.7 (Final) Kernel \r on an \m [root@test ~]# ...
编译安装Logstash
buyue
11-05 1002
官网 cd / wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-linux-x86_64.tar.gz tar -zxvf elasticsearch-7.4.2-linux-x86_64.tar.gz
评论 11
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值