记录一下,spark读取kafka写入hdfs,直接上代码
build.sbt
name := "KafkaToHdfs"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.2" % "provided",
"org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided",
"org.apache.spark" %% "spark-streaming-kafka" % "1.5.2",
"org.json4s" %% "json4s-native" % "3.2.10",
"org.json4s" %% "json4s-jackson" % "3.2.10"
)
mergeStrategy in assembly := {
case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard
case "log4j.properties" => MergeStrategy.discard
case m if m.toLowerCase.startsWith("meta-inf/services/") => MergeStrategy.filterDistinctLines
case "reference.conf" => MergeStrategy.concat
case _ => MergeStrategy.first
}
KafkaToHdfs.scala
import java.util.Properties
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._
import org.apache.spark.SparkConf
object KafkaToHdfs {
def main(args:Array[String]) {
if (args.length < 6) {
System.err.println("Usage: KafkaToHdfs ")
System.exit(1)
}
val Array(zkQuorum, group, topics, numThreads, output, seconds) = args
val sparkConf = new SparkConf().setAppName("KafkaToHdfs")
val ssc = new StreamingContext(sparkConf, Seconds(seconds.toInt))
ssc.checkpoint("checkpoint")
val topicpMap = topics.split(",").map((_,numThreads.toInt)).toMap
val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicpMap).map(_._2)
lines.saveAsTextFiles(output)
ssc.start()
ssc.awaitTermination()
}
}
编译打包需要,sbt的assembly插件哦
编译完成,执行下面的命令,提交到分布式执行
/usr/local/spark-1.5.2-bin-hadoop2.4/bin/spark-submit --class KafkaToHdfs --master spark://master:7077 --executor-memory 1g
/home/hadoop/spark_scala_work/test_spark_kafka_to_hdfs/target/scala-2.10/KafkaToHdfs-assembly-1.0.jar master:2182,slave1:2182,slave2:2182 test-
consumer-group test 3 hdfs://master:9000/tmp/taglog 10
附带一个kafka的创建topic的命令,这里创建一个test的topic
/usr/local/confluent-1.0.1/bin/kafka-topics --create --zookeeper master:2182,slave1:2182,slave2:2182 --replication-factor 2 --partitions 1 --topic
test
最后,附上一个完整的测试源码。安装sbt和sbt就可以进行编译打包了。
test_spark_kafka_to_hdfs.tar