本文记录笔者第一次调通:python写数据到txt文件—flume采集txt文件数据—kafka(channel)接收并在终端查看消息的过程。
1.python从csv写入txt,代码实例如下:
# -*- coding: utf-8 -*-
import random
import time
readFileName="/usr/local/src/data/orders.csv"
writeFileName="./flume_exec_test.txt"
with open(writeFileName,'a+')as wf:
with open(readFileName,'rb') as f:
for line in f.readlines():
for word in line.split(" "):
ss = line.strip()
if len(ss)<1:
continue
wf.write(ss+'\n')
2.设置在flume相应的conf里设置相关信息,如下(制定kafka,设置topic):
3.新启动一个master,修改名字为consumer,逐步执行如下操作:
1).新建topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1