假期旅行数仓项目--OLAP

澄绪猿

已于 2024-09-24 14:09:48 修改

阅读量376

点赞数 5

文章标签： java flume kafka 数据库大数据数据仓库

于 2024-09-24 14:08:34 首次发布

本文链接：https://blog.csdn.net/python8181/article/details/142487846

版权

需要这个完整离线数仓项目的源码和流程PPT可以私信我，可以帮助解决项目中遇到的问题，做完项目可以让你对数仓有更加清晰的认识

项目流程：

配置文件

kafka server.properties

hive : hvie-site.xml

启动mysql 的binlog日志

修改maxwell配置文件监听mysql的数据同步到kafka

配置flume-ng文件采集kafka—incdb 主题消费到的数据并上传至hdfs

flume-config

# ------------------- define data source ---------------------- 
# source alias 
agent.sources = source_from_kafka 
# channels alias 
agent.channels = mem_channel 
# sink alias 
agent.sinks = hdfs_sink 
# define kafka source 
agent.sources.source_from_kafka.type = org.apache.flume.source.kafka.KafkaSource agent.sources.source_from_kafka.channels = mem_channel agent.sources.source_from_kafka.batchSize = 5000 
# set kafka broker address
agent.sources.source_from_kafka.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092 
# set kafka topic
agent.sources.source_from_kafka.kafka.topics = incdb 
# set kafka groupid
agent.sources.source_from_kafka.kafka.consumer.group.id = incdb_id 
# defind hdfs sink
agent.sinks.hdfs_sink.type = hdfs 
# specify the channel the sink should use 
agent.sinks.hdfs_sink.channel = mem_channel 
# set store hdfs path 
agent.sinks.hdfs_sink.hdfs.path = /flume/kafka/%Y%m%d 
# set file size to trigger roll
agent.sinks.hdfs_sink.hdfs.rollSize = 0 
agent.sinks.hdfs_sink.hdfs.rollCount = 0 
agent.sinks.hdfs_sink.hdfs.rollInterval = 3600 
agent.sinks.hdfs_sink.hdfs.threadsPoolSize = 30 agent.sinks.hdfs_sink.hdfs.fileType=DataStream agent.sinks.hdfs_sink.hdfs.writeFormat=Text 
# define channel from kafka source to hdfs sink 
agent.channels.mem_channel.type = memory 
# channel store size 
agent.channels.mem_channel.capacity = 100000 
# transaction size 
agent.channels.mem_channel.transactionCapacity = 10000

开启maxwell监听：