4.11kafka和flume

1.安装netcat

1、下载安装
wget https://sourceforge.NET/projects/netcat/files/netcat/0.7.1/netcat-0.7.1.tar.gz/download
tar -zxvf netcat-0.7.1.tar.gz -C /usr/local
cd /usr/local
mv netcat-0.7.1 netcat
cd /usr/local/netcat
./configure
make && make install
2、配置
vim /etc/profile

添加以下内容：
# set  netcat path
export NETCAT_HOME=/usr/local/netcat
export PATH=$PATH:$NETCAT_HOME/bin
保存，退出，并使配置生效：
source /etc/profile
3、测试
nc -help成功

2.flume创建生产者

cd /opt/apps/flume/job
vi nc-kafka.conf
#添加
a1.sources=r1
a1.channels=c1

a1.sources.r1.type=netcat
a1.sources.r1.bind=localhost
a1.sources.r1.port=8888

a1.channels.c1.type=org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers=192.168.74.139:9020,192.168.74.128:9092
a1.channels.c1.kafka.topic=other

a1.sources.r1.channels=c1
#保存退出
cd ..
bin/flume-ng agent -c conf -n a1 -f job/nc-kafka.conf -Dflume.root.logger=INFO,console

另外的控制台输入

netcat localhost 8888

即可输入传输数据

另一个系统创建消费者

kafka-console-consumer.sh --bootstrap-server 192.168.74.139:9092 --topic other

 3.消费者

配置文件

a1.channels=c1
a1.sinks=k1

a1.channels.c1.type=org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers=192.168.74.139:9020,192.168.74.128:9092
a1.channels.c1.kafka.topic=hao
a1.channels.c1.kafka.consumer.group.id=flume
a1.channels.c1.kafka.consumer.auto.offset.reset=earliest
a1.channels.c1.parseAsFlumeEvent=false

a1.sinks.k1.type=logger

a1.sinks.k1.channel=c1

bin/flume-ng agent -c conf -n a1 -f job/kafka-nc.conf -Dflume.root.logger=INFO,console