大数据毕设笔记

大数据毕设

1.数据清洗

flume作为监测器查看日志数据，kafaka进行接收

1.启动datanode1，datanode2，datanode3上的kafka，zookeeper
zkServer.sh start
kafka-server-start.sh -daemon /home/mabowen/opt/kafka/kafka_2.13-2.7.0/config/server.properties

2.配置namenode1上/home/mabowen/opt/flume/apache-flume-1.9.0-bin/conf/ufo.conf
配置文件如下
#Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

#Describe/configue the source
a1.sources.r1.type = exec

#自己文件的地址
a1.sources.r1.command = tail -F /home/mabowen/opt/data/complete.csv
a1.sources.r1.port = 44444

#自己的IP
a1.sources.r1.host = 192.168.85.20
a1.sources.r1.channels = c1

#Describe the sink
#设置kafkaSink 注意大小写
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
#设置kafka的主题topic
a1.sinks.k1.topic = ufo
#设置kafka 的 broker地址以及端口号
a1.sinks.k1.kafka.bootstrap.servers = namenode1:9092,namenode2:9092,datanode1:9092,datanode2:9092,datanode3:9092
#设置kafka序列化方式
a1.sinks.k1.serializer.class = kafka.serializer.StringEncoder

#use a channel which buffers events in memory
#a1.channels.c1.type = memory
#a1.channels.c1.capacity = 100
#a1.channels.c1.transactionCapacity &#