flume source kafka channels memory sinks hdfs

source：

org.apache.flume.source.kafka.KafkaSource

channels：

memory

sinks：

hdfs

kafka_sources.sources  = source1
kafka_sources.channels = channel1
kafka_sources.sinks = sink1

kafka_sources.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
kafka_sources.sources.source1.kafka.bootstrap.servers = 172.168.1.17:9092,172.168.1.16:9092,172.168.1.14:9092
kafka_sources.sources.source1.kafka.topics = first_floor_test
kafka_sources.sources.source1.kafka.consumer.group.id = flume
kafka_sources.sources.source1.kafka.consumer.auto.commit.interval.ms = 60000
kafka_sources.sources.source1.channels = channel1

kafka_sources.channels.channel1.type = memory
kafka_sources.channels.channel1.capacity = 1000000
kafka_sources.channels.channel1.transactionCapacity = 1000000

kafka_sources.sinks.sink1.type = hdfs
kafka_sources.sinks.sink1.hdfs.path = hdfs://172.168.1.17:9000/first_floor/%Y%m%d
kafka_sources.sinks.sink1.hdfs.filePrefix = log
kafka_sources.sinks.sink1.compression.type = snappy
kafka_sources.sinks.sink1.channel = channel1

异常:

2019-11-09 15:23:52,414 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:150)] Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
        at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:246)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:453)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:106)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 12 more

 解决方案:

${HADOOP_HOME}share/hadoop/common/hadoop-common-2.7.2.jar

找到这个jar文件，copy到flume安装目录下的lib目录下就ok了