四(2)、flume遇到的问题

使用的是6.0.1cdh上面的1.8版本的flume:
从mysql抽取数据到hdfs
flume-ng agent --conf conf --conf-file conf/flume-conf04.properties --name a1 -Dflume.root.logger=INFO,console

a1.channels.ch1.type = memory
a1.sources.sql-source.channels = ch1
a1.channels = ch1
a1.sinks = HDFS
a1.sources = sql-source

a1.sources.sql-source.type = org.keedio.flume.source.SQLSource

a1.sources.sql-source.connection.url = jdbc:mysql://ip地址:3306/库名
a1.sources.sql-source.user = 用户名
a1.sources.sql-source.password = 密码
a1.sources.sql-source.table = 表名
a1.sources.sql-source.columns.to.select = *

a1.sources.sql-source.incremental.column.name = id
a1.sources.sql-source.incremental.value = 0

a1.sources.sql-source.run.query.delay=5000

a1.sources.sql-source.status.file.path = /var/lib/flume-ng/flume
a1.sources.sql-source.status.file.name = sql-source.status

a1.sinks.HDFS.channel = ch1
a1.sinks.HDFS.type = hdfs
a1.sinks.HDFS.hdfs.path = hdfs://node01/user/hive/warehouse/test.db/dim_period_d
a1.sinks.HDFS.hdfs.fileType = DataStream
a1.sinks.HDFS.hdfs.writeFormat = Text
a1.sinks.HDFS.hdfs.rollSize = 268435456
a1.sinks.HDFS.hdfs.rollInterval = 0
a1.sinks.HDFS.hdfs.rollCount = 0

报错:

2019-03-19 10:10:46,206 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2019-03-19 10:10:46,206 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2019-03-19 10:10:46,206 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2019-03-19 10:10:46,206 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: HDFS Agent: a1
2019-03-19 10:10:46,206 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2019-03-19 10:10:46,206 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2019-03-19 10:10:46,207 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2019-03-19 10:10:46,207 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2019-03-19 10:10:46,207 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2019-03-19 10:10:46,224 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2019-03-19 10:10:46,224 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)] Creating channels
2019-03-19 10:10:46,230 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel ch1 type memory
2019-03-19 10:10:46,234 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)] Created channel ch1
2019-03-19 10:10:46,235 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source sql-source, type org.keedio.flume.source.SQLSource
2019-03-19 10:10:46,242 (conf-file-poller-0) [INFO - org.keedio.flume.source.SQLSource.configure(SQLSource.java:63)] Reading and processing configuration values for source sql-source
2019-03-19 10:10:46,249 (conf-file-poller-0) [ERROR - org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:361)] Source sql-source has been removed due to an error during configuration
org.apache.flume.conf.ConfigurationException: hibernate.connection.url property not set
	at org.keedio.flume.source.SQLSourceHelper.checkMandatoryProperties(SQLSourceHelper.java:294)
	at org.keedio.flume.source.SQLSourceHelper.<init>(SQLSourceHelper.java:100)
	at org.keedio.flume.source.SQLSource.configure(SQLSource.java:66)
	at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:326)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:101)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-03-19 10:10:46,252 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: HDFS, type: hdfs
2019-03-19 10:10:46,265 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)] Channel ch1 connected to [HDFS]
2019-03-19 10:10:46,267 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:161)] Starting new configuration:{ sourceRunners:{} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4f5a783e counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} }
2019-03-19 10:10:46,268 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:168)] Starting Channel ch1
2019-03-19 10:10:46,310 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean.
2019-03-19 10:10:46,310 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounter
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值