Reference:https://jansipke.nl/monitoring-hosts-with-collectd-influxdb-and-grafana/
CollectD + Kafka + InfluxDB + Grafana 版本信息
CentOS Linux release 7.5.1804 (development toolkit installed)
Linux master 3.10.0-862.el7.x86_64
CollectD version 5.8.0
Kafka version 2.11-1.1.1
InfluxDB version 1.6.0
Grafana version 5.2.2.1
GitHub project librdkafka (master branch)
GitHub project kafka-influxdb (master branch)
1. Install CollectD and API to Kafka
- Download CollectD 5.8 source code
- Download GitHub: librdkafka “master branch”,
configure
, thenmake all install
(resolved dependencies: gcc, gcc-c++, kernel-devel, etc…) - CollectD: modify collectd-5.8.0/src/write_kafka.c
(due to a deprecated “error” function) - CollectD:
configure
,make
, thenmake install
(Check available plugins during configure, esp. including write_kafka) - Find CollectD by default installed at /opt/collectd
i. Modify configuration file: /opt/collectd/etc/collectd.conf
ii. Start service “manually” :/opt/collectd/sbin/collectd
2. Install Kafka and check producer and consumer
-
Download Kafka 2.11-1.1.1 installation package
-
Run zookeeper and kafka server
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
- Check if “write_kafka” has created topic “collectd” and produced data
bin/kafka-topics.sh --list --zookeeper localhost:2181
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic collectd --from-beginning
3. Install InfluxDB and API to Kafka
- Install InfluxDB and uncomment “http” features, and
systemctl start influx
In case security connection refuse issue, upgrade wget and install openssl-static. - Download Github: kafka-influxdb “master branch”, and
pip install kafka_influxdb
(resolved dependencies: epel-release, python-pip, etc…)
Python 3 is preferred, while python 2 actually installed in demo by default. - Modify kafka-influxdb config file and startup
For JSON format, the encoder need to be changed from “collectd_graphite_encoder” to “collectd_json_encoder”.
kafka_influxdb -c config_example.yaml
- Check database in InfluxDB, run command influx
show databases #see if collectd already created in InfluxDB
# If yes
# use collectd
show measurements #see if collectd features already created as table head
4. Install Grafana and configure InfluxDB
- Install Grafana and
systemctl start grafana.service
- Open Grafana at http://localhost:3000, and follow the wizard to finish configuration.
5. Others
- To stop Grafana and InfluxDB, use systemctl.
- To stop Kafka and Zookeeper, use stop scripts in kafka/bin. Or kill all kafka-related processes.
- To stop CollectD, kill process “collectd”.