flume-kafka整合--实时日志采集

flume采用架构

exec-source + memory-channel + avro-sink

avro-source + memory-channel + kafka-sink

kafka采用架构

启动zookeeper

zkServer.sh start

启动kafka(启动一个“篮子”)

kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties &

模拟

启动一个kafka消费者监听日志文件

kafka-console-consumer.sh –zookeeper 192.168.145.128:2181 –from-beginning –topic firstTopic

向日志文件中添加数据

echo hi,flume-kafka framework >> flume-kafka.txt

---

---

---

exec-memory-avro.conf

flume-ng agent \
  --name exec-memory-avro \
  --conf $FLUME_HOME/conf \
  --conf-file