spark读取oracle写入kafka,spark读取kafka写入hdfs

记录一下，spark读取kafka写入hdfs，直接上代码

build.sbt

name := "KafkaToHdfs"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(

"org.apache.spark" %% "spark-core" % "1.5.2" % "provided",

"org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided",

"org.apache.spark" %% "spark-streaming-kafka" % "1.5.2",

"org.json4s" %% "json4s-native" % "3.2.10",

"org.json4s" %% "json4s-jackson" % "3.2.10"

)

mergeStrategy in assembly := {

case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard

case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard

case "log4j.properties" => MergeStrategy.discard

case m if m.toLowerCase.startsWith("meta-inf/services/") => MergeStrategy.filterDistinctLines

case "reference.conf" => MergeStrategy.concat

case _ => MergeStrategy.first

}

KafkaToHdfs.scala

import java.util.Properties

import org.apache.spark.streaming._

import org.apache.spark.streaming.StreamingContext._

import org.apache.spark.streaming.kafka._

import org.apache.spark.SparkConf

object KafkaToHdfs {

def main(args:Array[String]) {

if (args.length < 6) {

System.err.println("Usage: KafkaToHdfs ")

System.exit(1)

}

val Array(zkQuorum, group, topics, numThreads, output, seconds) = args

val sparkConf = new SparkConf().setAppName("KafkaToHdfs")

val ssc = new StreamingContext(sparkConf, Seconds(seconds.toInt))

ssc.checkpoint("checkpoint")

val topicpMap = topics.split(",").map((_,numThreads.toInt)).toMap

val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicpMap).map(_._2)

lines.saveAsTextFiles(output)

ssc.start()

ssc.awaitTermination()

}

}

编译打包需要，sbt的assembly插件哦

编译完成，执行下面的命令，提交到分布式执行

/usr/local/spark-1.5.2-bin-hadoop2.4/bin/spark-submit --class KafkaToHdfs --master spark://master:7077 --executor-memory 1g

/home/hadoop/spark_scala_work/test_spark_kafka_to_hdfs/target/scala-2.10/KafkaToHdfs-assembly-1.0.jar master:2182,slave1:2182,slave2:2182 test-

consumer-group test 3 hdfs://master:9000/tmp/taglog 10

附带一个kafka的创建topic的命令，这里创建一个test的topic

/usr/local/confluent-1.0.1/bin/kafka-topics --create --zookeeper master:2182,slave1:2182,slave2:2182 --replication-factor 2 --partitions 1 --topic

test

最后，附上一个完整的测试源码。安装sbt和sbt就可以进行编译打包了。

test_spark_kafka_to_hdfs.tar