CollectD + Kafka + InfluxDB + Grafana 数据监控部署笔记

Reference：https://jansipke.nl/monitoring-hosts-with-collectd-influxdb-and-grafana/

CollectD + Kafka + InfluxDB + Grafana 版本信息

CentOS Linux release 7.5.1804 (development toolkit installed)
Linux master 3.10.0-862.el7.x86_64
CollectD version 5.8.0
Kafka version 2.11-1.1.1
InfluxDB version 1.6.0
Grafana version 5.2.2.1
GitHub project librdkafka (master branch)
GitHub project kafka-influxdb (master branch)

1. Install CollectD and API to Kafka

1. Download CollectD 5.8 source code
2. Download GitHub: librdkafka “master branch”, configure, then make all install
   (resolved dependencies: gcc, gcc-c++, kernel-devel, etc…)
3. CollectD: modify collectd-5.8.0/src/write_kafka.c
   (due to a deprecated “error” function)
4. CollectD: configure, make, then make install
   (Check available plugins during configure, esp. including write_kafka)
5. Find CollectD by default installed at /opt/collectd
   i. Modify configuration file: /opt/collectd/etc/collectd.conf
   ii. Start service “manually” : /opt/collectd/sbin/collectd

2. Install Kafka and check producer and consumer

1. Download Kafka 2.11-1.1.1 installation package

2. Run zookeeper and kafka server

bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &

3. Check if “write_kafka” has created topic “collectd” and produced data

bin/kafka-topics.sh --list --zookeeper localhost:2181
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic collectd --from-beginning

3. Install InfluxDB and API to Kafka

1. Install InfluxDB and uncomment “http” features, and systemctl start influx
   In case security connection refuse issue, upgrade wget and install openssl-static.
2. Download Github: kafka-influxdb “master branch”, and pip install kafka_influxdb
   (resolved dependencies: epel-release, python-pip, etc…)
   Python 3 is preferred, while python 2 actually installed in demo by default.
3. Modify kafka-influxdb config file and startup
   For JSON format, the encoder need to be changed from “collectd_graphite_encoder” to “collectd_json_encoder”.

kafka_influxdb -c config_example.yaml

4. Check database in InfluxDB, run command influx

show databases   #see if collectd already created in InfluxDB
# If yes
# use collectd
show measurements  #see if collectd features already created as table head 

4. Install Grafana and configure InfluxDB

1. Install Grafana and systemctl start grafana.service
2. Open Grafana at http://localhost:3000, and follow the wizard to finish configuration.

5. Others

1. To stop Grafana and InfluxDB, use systemctl.
2. To stop Kafka and Zookeeper, use stop scripts in kafka/bin. Or kill all kafka-related processes.
3. To stop CollectD, kill process “collectd”.