我的大数据之路--Flume1.6+kafka1.0实战

flume–kafka（亲测成功）

准备数据,随便从网上下载一篇英语作文

vi demo.txt
-----------------------------------------------------
什么让你开心 What Makes You Happy
In general, people believe that fame and fortune can bring great happiness. The successful persons can do what they want and enjoy the luxury life. But according to the research, more than 60% poor people feel comfortable about their life and they are moderately happy, while the rich people feel not satisfied with their current situation.
一般来说，人们认为名誉和财富可以带来莫大的快乐。成功人士可以做他们想做的事，享受奢侈的生活。但据调查，超过60%的穷人对自己的生活感到舒适，也很快乐，而富人们则对自己的现状感到不满意。
The good education and high salary are believed to be the main factors that contribute to the great happiness. This group of people belong to the middle class. They make good money and are able to pay for the comfortable life. But the fact is that most of them are under great pressure from work. The strong competition makes them to fight for the efficiency. Working for extra hours happens all the time.
良好的教育和高工资被认为是最幸福的来源。这个群体的人属于中产阶级，他们赚钱，有能力支付舒适的生活。但事实是，大多数人承受着巨大压力，激烈的竞争使得他们争分夺秒，加班已经是家常便饭。
For the poor people, though they don't have a lot of money, they are satisfied as they make ends meet. When the families get together and enjoy the dinner time, they feel greatest happiness. Money can't buy all the things, and the real happiness depends on the value you treat. Some people take family and health for the first place, while others take fame and money as the most important thing.
对于穷人来说，尽管他们钱不多，但是他们满足于维持生计。家人聚在一起，享受晚餐的时间，他们觉得就是最大的幸福。钱不能买到所有的东西，而真正的幸福取决于你的价值观。有些人把家庭和健康放在第一位，而有些人则把名誉和钱看作最重要的东西。

0. Flume1.6 在conf下创建 my-test-topic.properties

vi my-test-topic.properties
---------------------------------------------------
#具体定义source
a1.sources.r1.type = spooldir
#先创建此目录，注意用户组 hadoop hadoop，保证里面空的
a1.sources.r1.spoolDir = /user/test3
#sink到kafka里面
a1.sinks.k1.channel = c1
a1.sinks.k1.type =org.apache.flume.sink.kafka.KafkaSink
#设置Kafka的Topic
a1.sinks.k1.topic = test3
#设置Kafka的broker地址和端口号
a1.sinks.k1.brokerList = Master:9092
#配置批量提交的数量
a1.sinks.k1.flumeBatchSize = 20
a1.sinks.k1.producer.acks = 1
a1.sinks.k1.producer.linger.ms = 1
a1.sinks.ki.producer.compression.type= snappy
 
#对于channel的配置描述 使用文件做数据的临时缓存 这种的安全性要高
a1.channels.c1.type = file
 
 #事先创建好，注意用户组为 hadoop hadoop
a1.channels.c1.checkpointDir = /user/flume-1.6/checkpoint
a1.channels.c1.dataDirs = /user/flume-1.6/data
 
#通过channel c1将source r1和sink k1关联起来
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

1. 创建主题 test3，配置好环境变量。vi ~/.bashrc PATH=/user/flume-1.6/bin:$PATH

kafka-topics.sh --create --zookeeper Master:2181 --replication-factor 1 --partitions 1 --topic test3

2. 查看

bin/kafka-topics.sh --list --zookeeper Master:2181

3. 启动Flume,在flume目录下

bin/flume-ng agent --conf conf --conf-file conf/my-test-topic.properties --name a1 -Dflume.root.logger=INFO,console

4. 启动消费者查看，在Slave1主机上

./bin/kafka-console-consumer.sh -zookeeper Slave1:2181 --from-beginning --topic test3

5. 在Master主机上放之前下载好的英语作文

cp demo.text /user/test3

6.查看Flume是否采集成功

7. 再查看Slave1，接收成功

BUG

1、Could not find or load main class org.apache.flume.node.Application
可能是之前有装过flume其他版本的，而且配置了环境变量，把它删掉,记得要sources
export看一下是不是有个FLUME_HOME的环境变量指向原来的文件夹，

2、ERROR node.Application: A fatal error occurred while running. Exception follows.
启动Flume命令错误，自己仔细点

3、在消费者窗口看不到输出结果或者没采集到目录。自己检查是不是哪里写错了，值得注意的是文件的用户组 一定为 hadoop hadoop