目录
2.SimpleAsyncHbaseEventSerializer
一.Flume汇入数据到Hive中
方法一:汇入到Hive指定的HDFS路径中:
1)在hive中创建数据库和外部表
create table flume;
create external table flume_into_hive(name string,age int) partitioned by (dt string) row format delimited fields terminated by ',' location '/user/hive/warehouse/flume.db/flume_into_hive';
2)在/root中创建hive.log文件
mkdir flume-hive
cd flume-hive/
vi hive.log
3)在flume的conf路径中编写配置文件flume-into-hive-1.conf
agent.sources=r1
agent.channels=c1
agent.sinks=s1agent.sources.r1.type=exec
agent.sources.r1.command=tail -F /root/flume-hive/hive.logagent.channels.c1.type=memory
agent.channels.capacity=1000
agent.channels.c1.transactionCapacity=100agent.sinks.s1.type=hdfs
agent.sinks.s1.hdfs.path = hdfs://node01:9000/user/hive/warehouse/flume.db/flume_into_hive/dt=%Y%m%d
agent.sinks.s1.hdfs.filePrefix = upload-
agent.sinks.s1.hdfs.fileSuffix=.txt
#是否按照时间滚动文件夹
agent.sinks.s1.hdfs.round = true
#多少时间单位创建一个新的文件夹
agent.sinks.s1.hdfs.roundValue = 1
#重新定义时间单位
agent.sinks.s1.hdfs.roundUnit = hour
#是否使用本地时间戳
agent.sinks.s1.hdfs.useLocalTimeStamp = true
#积攒多少个 Event 才 flush 到 HDFS 一次
agent.sinks.s1.hdfs.batchSize = 100
#设置文件类型,可支持压缩
agent.sinks.s1.hdfs.fileType = DataStream
agent.sinks.s1.hdfs.writeFormat=Text
#多久生成一个新的文件
agent.sinks.s1.hdfs.rollInterval = 60
#设置每个文件的滚动大小大概是 128M
agent.sinks.s1.hdfs.rollSize = 134217700
#文件的滚动与 Event 数量无关agent.sinks.s1.hdfs.rollCount = 0
agent.sources.r1.channels=c1
agent.sinks.s1.channel=c
4)运行flume
bin/flume-ng agent -c conf -f conf/flume-into-hive-1.conf -n agent
5)查询hdfs中的数据
hdfs dfs -cat /user/hive/warehouse/flume.db/flume_into_hive/dt=20221110/upload-.1668082651723.txt
6)在hive表中加载数据
load data inpath '/user/hive/warehouse/flume.db/flume_into_hive/dt=20221110' into table flume_into_hive partition(dt=20221110);
select * from flume_into_hive;
方法二:利用HiveSink汇入数据
1)从hive/lib和和hive/hcatalog/share/hcatalog/中找寻下列JAR包,放入到fl