my docker-machine ip:
✘ ⚙ xiezhenjia@xiezhenjiadeMac-mini ~/go/src/h12.me/kafka/tools/kafpro master ● docker-machine ip default
192.168.99.100
1、拉取镜像:
docker pull sebp/elk
2、运行:
docker run -p 5601:5601 -p 9200:9200 -p 9300:9300 -p 5044:5044 -p 5000:5000 -d -it --name elk sebp/elk
3、进入容器:
docker exec -it elk /bin.bash
4、配置logstash:
logstash位置:/opt/logstash
cd /opt/logstash/bin
直接配置:
./logstash -e 'input { kafka { type => "test" zk_connect => "192.168.99.100:32774/kafka" group_id => "test" topic_id => "my_test_4" consumer_threads => 1} } output { elasticsearch { hosts => ["localhost"] } }'
通过配置文件配置:
logstash.conf:
input {
kafka {
type => "test"
zk_connect => "192.168.99.100:32774/kafka"
group_id => "test"
topic_id => "my_test_4"
consumer_threads => 1 #as more as the num of partitions
}
}
output {
elasticsearch {
hosts => ["localhost"]
}
}
加载配置文件
./logstash -f logstash.conf &
⚠️注:多个input源可以直接在kafka下加即可:
input {
kafka {
type => "test"
zk_connect => "192.168.99.100:32774/kafka"
group_id => "test"
topic_id => "my_test_4"
consumer_threads => 1 #as more as the num of partitions
}
kafka {
type => "test1"
zk_connect => "192.168.99.100:32774/kafka"
group_id => "test1"
topic_id => "my_test_5"
consumer_threads => 1 #as more as the num of partitions
}
}
output {
elasticsearch {
hosts => ["localhost"]
}
}
5、elasticsearch 插件安装
elasticsearch位置: /usr/share/elasticsearch
安装head插件:
root@e639c39cba78:/usr/share/elasticsearch/bin# ./plugin install mobz/elasticsearch-head
logstash接入错误:
log4j, [2016-01-27T08:05:31.560] WARN: kafka.consumer.ConsumerFetcherThread: [ConsumerFetcherThread-test_e639c39cba78-1453881870694-52a303a9-0-4], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 0; ClientId: test; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [my_test_4,0] -> PartitionFetchInfo(3131649,1048576). Possible cause: java.nio.channels.ClosedChannelException
发现是kafka集群中的一台机器挂了。。
http://192.168.99.100:5601/