flume从mysql采集数据同时存入hdfs和kafka，加载数据到hive中

最新推荐文章于 2022-04-26 10:40:17 发布

置顶

大数据sxt

最新推荐文章于 2022-04-26 10:40:17 发布

阅读量2.9k

点赞数 3

分类专栏： flume kafka

本文链接：https://blog.csdn.net/u014082561/article/details/87950343

版权

该博客详细介绍了如何使用Flume从MySQL数据库中采集数据，同时将数据存入HDFS和Kafka，并最终加载到Hive中进行分析。步骤包括创建MySQL表、配置Flume、准备JAR包、启动Kafka消费者、运行Flume Agent、验证数据传输以及准实时增量抽取。此外，还提到了Flume增量同步的限制，例如仅能识别新增数据，且对源库的影响取决于轮询间隔。

摘要由CSDN通过智能技术生成

1.在mysql创建表导入数据

use test;
 
create table  wlslog  
(id         int not null,
 time_stamp varchar(40),
 category   varchar(40),
 type       varchar(40),
 servername varchar(40),
 code       varchar(40),
 msg        varchar(40),
 primary key ( id )
);
 
insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(1,'apr-8-2014-7:06:16-pm-pdt','notice','weblogicserver','adminserver','bea-000365','server state changed to standby');
insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(2,'apr-8-2014-7:06:17-pm-pdt','notice','weblogicserver','adminserver','bea-000365','server state changed to starting');
insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(3,'apr-8-2014-7:06:18-pm-pdt','notice','weblogicserver','adminserver','bea-000365','server state changed to admin');
insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(4,'apr-8-2014-7:06:19-pm-pdt','notice','weblogicserver','adminserver','bea-000365','server state changed to resuming');
insert into wlslog(id,time_stamp,category,type,servername,code,msg) values(5,'apr-8-2014-7:06:20-pm-pdt','notice','weblogicserver','adminserver','bea-000361','started weblogic adminserver');
insert into wlslog(id,time_stamp,category,type