hadoop php mysql_PHP+Hadoop+Hive+Thrift+Mysql实现数据统计分析

转存

Flume监听文件目录，将数据传输到能访问Hdfs集群的服务器上,这里传输到了224机器的7000端口

#agent3表示代理名称 login

agent3.sources=source1

agent3.sinks=sink1

agent3.channels=channel1

配置source1

配置source1

agent3.sources.source1.type=spooldir

agent3.sources.source1.spoolDir=/data/releases/stat/Data/10001/agent3.sources.source1.channels=channel1

agent3.sources.source1.fileHeader = false

配置sink1

agent3.sinks.sink1.type=avro

agent3.sinks.sink1.hostname=192.168.23.224agent3.sinks.sink1.port=7000agent3.sinks.sink1.channel=channel1

配置channel1

agent3.channels.channel1.type=fileagent3.channels.channel1.checkpointDir=/data/flume_data/checkpoint_login

agent3.channels.channel1.dataDirs=/data/flume_data/channelData_login

启动flume

加到supervisor守护进程

/home/flume/bin/flume-ng agent -n agent3 -c /home/flume/conf/ -f /home/flume/conf/statistics/login_flume.conf -Dflume.root.logger=info,console

224机器监听7000端口，将数据写到hdfs集群

agent1表示代理名称

agent4.sources=source1

agent4.sinks=sink1

agent4.channels=channel1

配置source1

agent4.sources.source1.type=avro

agent4.sources.source1.bind=192.168.23.224agent4.sources.source1.port=7000agent4.sources.source1.channels=channel1

配置sink1

agent4.sinks.sink1.type=hdfs

agent4.sinks.sink1.hdfs.path=hdfs://hdfs/umr-ubvzlf/uhiveubnhq5/warehouse/web_stat.db/login/date\=%Y-%m-%d

agent4.sinks.sink1.hdfs.fileType=DataStream

agent4.sinks.sink1.hdfs.filePrefix=buffer_census_

agent4.sinks.sink1.hdfs.writeFormat=TEXT

agent4.sinks.sink1.hdfs.rollInterval=30agent4.sinks.sink1.hdfs.inUsePrefix = .agent4.sinks.sink1.hdfs.rollSize=536870912agent4.sinks.sink1.hdfs.useLocalTimeStamp = trueagent4.sinks.sink1.hdfs.rollCount=0agent4.sinks.sink1.channel=channel1

配置channel1

agent4.channels.channel1.type=fileagent4.channels.channel1.checkpointDir=/data/flume_data/login_checkpoint

agent4.channels.channel1.dataDirs=/data/flume_data/login_channelData

启动

加到supervisor守护进程

/usr/local/flume/bin/flume-ng agent -n agent4 -c /usr/local/flume/conf/ -f /usr/local/flume/conf/statistics/login_flume.conf -Dflume.root.logger=info,console