单机docker部署elk测试环境

my docker-machine ip：

 ✘ ⚙ xiezhenjia@xiezhenjiadeMac-mini  ~/go/src/h12.me/kafka/tools/kafpro   master ●  docker-machine ip default

192.168.99.100

1、拉取镜像:

docker pull sebp/elk

2、运行:

docker run -p 5601:5601 -p 9200:9200 -p 9300:9300 -p 5044:5044 -p 5000:5000 -d -it --name elk sebp/elk

3、进入容器：

docker exec -it elk /bin.bash

4、配置logstash：

logstash位置：/opt/logstash

cd /opt/logstash/bin

直接配置：

./logstash -e 'input { kafka { type => "test" zk_connect => "192.168.99.100:32774/kafka" group_id => "test" topic_id => "my_test_4" consumer_threads => 1}  } output { elasticsearch { hosts => ["localhost"] } }'

通过配置文件配置：

logstash.conf:

input {

  kafka {

    type => "test"

    zk_connect => "192.168.99.100:32774/kafka"

    group_id => "test"

    topic_id => "my_test_4"

    consumer_threads => 1  #as more as the num of partitions

  }

}

output {

  elasticsearch {

    hosts => ["localhost"]

  }

}

加载配置文件

./logstash -f logstash.conf &

⚠️注：多个input源可以直接在kafka下加即可：

input {

  kafka {

    type => "test"

    zk_connect => "192.168.99.100:32774/kafka"

    group_id => "test"

    topic_id => "my_test_4"

    consumer_threads => 1  #as more as the num of partitions

  }

  kafka {

    type => "test1"

    zk_connect => "192.168.99.100:32774/kafka"

    group_id => "test1"

    topic_id => "my_test_5"

    consumer_threads => 1  #as more as the num of partitions

  }

}

output {

  elasticsearch {

    hosts => ["localhost"]

  }

}

5、elasticsearch 插件安装

elasticsearch位置： /usr/share/elasticsearch

安装head插件：

root@e639c39cba78:/usr/share/elasticsearch/bin# ./plugin install mobz/elasticsearch-head

open  http://192.168.99.100:9200/_plugin/head/

logstash接入错误：

log4j, [2016-01-27T08:05:31.560]  WARN: kafka.consumer.ConsumerFetcherThread: [ConsumerFetcherThread-test_e639c39cba78-1453881870694-52a303a9-0-4], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 0; ClientId: test; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [my_test_4,0] -> PartitionFetchInfo(3131649,1048576). Possible cause: java.nio.channels.ClosedChannelException

发现是kafka集群中的一台机器挂了。。

kibana：

http://192.168.99.100:5601/