Spark-Streaming连接kafka0.8 能连接却不能消费问题

最新推荐文章于 2023-01-15 10:53:22 发布
最新推荐文章于 2023-01-15 10:53:22 发布
阅读量1.8k 收藏
点赞数 1
文章标签： sparkstreaming kafka cdh scala
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_42510528/article/details/84348916
版权

 话不多说,直接上代码

package cn.sparkstreaming.kafka

import kafka.serializer.{StringDecoder, Decoder}
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkContext, SparkConf}

import scala.reflect.ClassTag

/**
  * Created by Administrator on 2018/11/22.
  */
object SparkStreamDirectDemo {
  def main(args: Array[String]) {

    val conf = new SparkConf()
    conf.setAppName("spark_streaming")
    //conf.setMaster("local[*]")

    val sc = new SparkContext(conf)
    sc.setCheckpointDir("file:///segment2/Alarm_data/checkpoints")
    //sc.setCheckpointDir("checkpoints")
    sc.setLogLevel("ERROR")
    //多少秒消费一次
    val ssc = new StreamingContext(sc, Seconds(60))

    val topics = Map("WXALARM" -> 2)

    val kafkaParams = Map[String, String](
      "bootstrap.servers" -> "10.216.5.152:9093,10.216.5.153:9093,10.216.5.154:9093",
      "group.id" -> "WYWX_123",
      "auto.offset.reset" -> "smallest"
    )

    // 直连方式拉取数据，这种方式不会修改数据的偏移量，需要手动的更新
    val lines = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, Set("WXALARM")).map(_._2)
    //val lines = KafkaUtils.createStream(ssc, "10.216.5.152:2183,10.216.5.153:2183,10.216.5.154:2183", "WYWX", topics).map(_._2)

    //lines.print()
    lines.saveAsTextFiles("file:///segment2/Alarm_data/Direct.txt")
    ssc.start()
    ssc.awaitTermination()

  }
}

 

 

pom.xml

 

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>esb_kafka</groupId>
    <artifactId>esb_kafka</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.version>2.11.8</scala.version>
        <spark.version>2.3.0</spark.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.4</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.specs</groupId>
            <artifactId>specs</artifactId>
            <version>1.2.5</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.6</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <version>0.13.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.4.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.4.1</version>
        </dependency>
        <dependency>
            <groupId>io.spray</groupId>
            <artifactId>spray-json_2.10</artifactId>
            <version>1.3.2</version>
        </dependency>

        <dependency>
            <groupId>postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>9.1-901.jdbc4</version>
        </dependency>
        <dependency>
            <groupId>c3p0</groupId>
            <artifactId>c3p0</artifactId>
            <version>0.9.1.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-ftp</artifactId>
            <version>2.13.2</version>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

 因为在自己的集群上是可以消费数据的,但是放在生产环境上不能消费,可能是kafka版本太低,也可能是cdh版本太高,导致不能兼容

生产环境异常:

18/11/21 17:44:07 INFO spark.SparkContext: Running Spark version 2.3.0.cloudera4
18/11/21 17:44:07 INFO spark.SparkContext: Submitted application: spark_streaming
18/11/21 17:44:07 INFO spark.SecurityManager: Changing view acls to: root
18/11/21 17:44:07 INFO spark.SecurityManager: Changing modify acls to: root
18/11/21 17:44:07 INFO spark.SecurityManager: Changing view acls groups to: 
18/11/21 17:44:07 INFO spark.SecurityManager: Changing modify acls groups to: 
18/11/21 17:44:07 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
18/11/21 17:44:08 INFO util.Utils: Successfully started service 'sparkDriver' on port 7355.
18/11/21 17:44:08 INFO spark.SparkEnv: Registering MapOutputTracker
18/11/21 17:44:08 INFO spark.SparkEnv: Registering BlockManagerMaster
18/11/21 17:44:08 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/11/21 17:44:08 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/11/21 17:44:08 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-78ed315e-aa29-40b1-a44c-b1e05b0aea67
18/11/21 17:44:08 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
18/11/21 17:44:08 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/11/21 17:44:08 INFO util.log: Logging initialized @2256ms
18/11/21 17:44:08 INFO server.Server: jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
18/11/21 17:44:08 INFO server.Server: Started @2371ms
18/11/21 17:44:08 INFO server.AbstractConnector: Started ServerConnector@14bb2297{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/11/21 17:44:08 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@77b7ffa4{/jobs,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@213e3629{/jobs/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4e9658b5{/jobs/job,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@70eecdc2{/jobs/job/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@c41709a{/stages,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7db0565c{/stages/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54ec8cc9{/stages/stage,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2a551a63{/stages/stage/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1a6f5124{/stages/pool,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1edb61b1{/stages/pool/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@ec2bf82{/storage,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@cc62a3b{/storage/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6cc0bcf6{/storage/rdd,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@29539e36{/storage/rdd/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@32f61a31{/environment,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@f5c79a6{/environment/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@669253b7{/executors,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5305c37d{/executors/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@51a06cbe{/executors/threadDump,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3dddbe65{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@49a64d82{/static,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1fc793c2{/,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2575f671{/api,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@27f9e982{/jobs/job/kill,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4593ff34{/stages/stage/kill,null,AVAILABLE,@Spark}
18/11/21 17:44:08 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://hbwy37:4040
18/11/21 17:44:08 INFO spark.SparkContext: Added JAR file:/segment2/Alarm_data/esb_kafka-1.0-SNAPSHOT3.jar at spark://hbwy37:7355/jars/esb_kafka-1.0-SNAPSHOT3.jar with timestamp 1542793448794
18/11/21 17:44:08 INFO executor.Executor: Starting executor ID driver on host localhost
18/11/21 17:44:08 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 15748.
18/11/21 17:44:08 INFO netty.NettyBlockTransferService: Server created on hbwy37:15748
18/11/21 17:44:08 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/11/21 17:44:08 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hbwy37, 15748, None)
18/11/21 17:44:08 INFO storage.BlockManagerMasterEndpoint: Registering block manager hbwy37:15748 with 366.3 MB RAM, BlockManagerId(driver, hbwy37, 15748, None)
18/11/21 17:44:08 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hbwy37, 15748, None)
18/11/21 17:44:08 INFO storage.BlockManager: external shuffle service port = 7337
18/11/21 17:44:08 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, hbwy37, 15748, None)
18/11/21 17:44:09 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6090f3ca{/metrics/json,null,AVAILABLE,@Spark}
18/11/21 17:44:10 INFO scheduler.EventLoggingListener: Logging events to hdfs://nameservice1/user/spark/spark2ApplicationHistory/local-1542793448839
18/11/21 17:44:10 INFO spark.SparkContext: Registered listener com.cloudera.spark.lineage.NavigatorAppListener
18/11/21 17:44:16 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
        at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:196)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:212)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:389)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:381)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
18/11/21 17:44:16 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
18/11/21 17:44:16 ERROR scheduler.JobScheduler: Error running job streaming job 1542793455000 ms.0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
        at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:196)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:212)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:389)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:381)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
        at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1364)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.take(RDD.scala:1337)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:735)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:734)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
        at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
        at scala.util.Try$.apply(Try.scala:192)
        at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
        at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
        at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
        at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
        at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
        at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:196)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:212)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:389)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:381)
        ... 3 more
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
        at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:196)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:212)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:389)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:381)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
        at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1364)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.take(RDD.scala:1337)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:735)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:734)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
        at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
        at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
        at scala.util.Try$.apply(Try.scala:192)
        at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
        at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
        at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
        at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
        at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
        at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
        at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
        at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:196)
        at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:212)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:389)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$28.apply(RDD.scala:1364)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:381)
        ... 3 more

 

找了许久也没有找到对应的解决方法,考虑到cdh的spark包和原生态的spark包存在一定的差异,就去官方看了一下

http://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html#approach-2-direct-approach-no-receivers

果不其然

 

需要把 spark-streaming-kafka-0-8-assembly_2.11.jar  放到spark的jars目录下

下载地址:https://search.maven.org/search?q=a:spark-streaming-kafka-0-8-assembly_2.11%20AND%20v:2.4.0

因为我们cdh装的最新spark版本2.3.0(cdh暂时没有spark2.4.0版本)

官方又推荐的是spark 2.4.0版本,我发现 无论是2.4.0还是2.3.0的spark 都是可以运行的

 

 groupId = org.apache.spark
 artifactId = spark-streaming-kafka-0-8_2.11
 version = 2.4.0

希望我遇到的这个坑,能让你解决当前问题

狱蝶阿一
关注
Spark Streaming连接Kafka入门教程
主要分享大数据相关的知识，如Spark、Hudi
05-18 5253
转载请务必注明原创地址为：https://dongkelun.com/2018/05/17/sparkKafka/ 前言 首先要安装好kafka,这里不做kafka安装的介绍,本文是Spark Streaming入门教程，只是简单的介绍如何利用spark 连接kafka，并消费数据，由于博主也是才学，所以其中代码以实现为主，可能并不是最好的实现方式。 1、对应依赖 根据kafka版本选...
Spark 连接 KafKa
TT15751097576的博客
10-16 481
Spark 连接 KafKa 数据的流程与细节方向 前几天把数据导入kafka中，现在要把数据从kafka中使用Scala将数据导入数据仓库； 新建maven工程（基础步骤）， 在开百度中打开maven工程–>找到kafka的jar 导入scala框架 <dependency> <groupId>org.apache.kafka</groupId&g...
本地开发spark streaming无法消费云主机kafka消息
weixin_39182877的博客
04-28 583
1、Kafka集群在一个192.168.0.x网段的，而我们的生产者在集群外，无法将消息发送过去 错误：11:21:13,936 ERROR KafkaProducer - Batch containing 11 record(s) expired due to timeout while requesting metadata from brokers for onlinelogs-1
kafka broker Leader -1引起spark Streaming不能消费的故障解决方法
微信公众号：大数据从业者
09-30 438
一、问题描述：Kafka生产集群中有一台机器cdh-003由于物理故障原因挂掉了，并且系统起不来了，使得线上的spark Streaming实时任务不能正常消费，重启实时任务都不行。查看kafka topic状态，发现broker Leader出现-1的情况，如下图 二、问题分析Kafka Broker Leader 为-1，表示有partition在选举Leader的时候失败了，因此引起了消费...
Spark 对接 Kafka
qq_45887821的博客
04-24 611
Spark对接Kafka Kafka作为工作中最常用到的消息中间件, 以其高速处理消息出现在实时处理领域. 要使用Kafka做实时处理,那么就必须说道Spark和Kafka之间的对接 对接方式 Kafka作为Spark Executor的生产者, Spark Executor从Kafka中拉取数据并进行内容的计算. 准备 创建一个Maven, 导入需要的依赖 <dependencies> <!--spark core--> <dependency>
spark-streaming-kafka-0-8_2.11-2.4.0.jar
01-22
spark-streaming-kafka-0-8_2.11-2.4.0.jar
spark-streaming-kafka-0-8-assembly_2.11-2.4.5
03-31
`spark-streaming-kafka-0-8-assembly_2.11-2.4.5`是Apache Spark的一个库，用于在Spark Streaming中与Apache Kafka 0.8版本进行交互。这个库的版本号`2.4.5`表示它是针对Spark 2.4.x系列的，并且依赖于Scala的构建...
spark-streaming 和kafka集成采坑总结，jar包问题
yx_keith的博客
05-09 5919
使用spark-streaming 和kafka集成进行wordcount，spark和kafka版本分别为：spark-1.6.0和kafka_2.10-0.8.2.2，开发时只需要spark-assembly-1.6.0-hadoop2.6.0.jar包和spark-streaming-kafka_2.10-1.6.0.jar包就够了，但是当将最终代码打成jar包提交spark集群运行时，会报...
sparkstreaming2.3_kafka0.8_jar包集合
04-26
资源不能免费。。。。。。。。。。。。。。。。。。。。。。 sparkstreaming集成kafka总结请看https://blog.csdn.net/qq_35946969/article/details/79995514
Spark直连Kafka
开水烫鱼的博客
02-24 843
bject KafkaDirectWordCount { def main(args: Array[String]): Unit = { //1.创建sparkStreaming程序入口 val conf: SparkConf = new SparkConf().setAppName("KafkaDirectWordCount") .setMaster("local...
使用spark 对接kafka
weixin_42584586的博客
01-12 309
使用 Spark 连接 Kafka，需要先安装 Kafka 驱动程序。然后，可以使用 Spark 中的 KafkaUtils 库来读取和写入 Kafka 中的数据。例如，可以使用以下代码读取 Kafka 中的数据： val kafkaParams =Map[String, Object]( "bootstrap.servers" -> "localhost:9092", "key.d...
Spark Streaming对接Kafka
weixin_43894652的博客
01-15 1719
上述两种方式中，Direct Approach方式更加可靠，不需要Spark Streaming自己去保证维护数据的可靠性，而是由善于处理这类工作的Kafka来做。在前面的案例中，我们监听了来自网络端口的数据，实现了WordCount，但是在实际开发中并不是这样。需求：利用Spark Streaming不断处理来自Kafka生产者生产的数据，并统计出现的单词数量。3，开启Spark Streaming消费数据，将产生的日志结果存储到日志中。2，编辑Spark Streaming代码，统计单词出现的数量。
spark与kafka连接测试
韩利鹏
12-15 1889
测试环境：scala版本为2.11.8,jdk版本为java1.7.79 搭建的工程为maven工程，所需要的依赖有：<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.11</artifactId> <version>2
sparkStream基于kafka源连接
mn_kw的博客
08-20 553
基于Direct方式 这种方式周期性地查询Kafka，来获得每个topic+partition的最新的offset,从而定义每个batch的offset的范围。   这种方式有以下优点： 1. 简化并行读取：如果要读取多个partition,不需要创建多个DStream，然后对他们进行union操作，spark会创建和kafka partition一样多的RDD partition，并且会...
SparkStreaming与Kafka整合遇到的问题及解决方案
weixin_33728268的博客
08-03 1162
最近工作中是做日志分析的平台，采用了sparkstreaming+kafka，采用kafka主要是看中了它对大数据量处理的高性能，处理日志类应用再好不过了，采用了sparkstreaming的流处理框架 主要是考虑到它本身是基于spark核心的，以后的批处理可以一站式服务，并且可以提供准实时服务到elasticsearch中，可以实现准实时定位系统日志。...
SparkStreaming消费不到Kafka的数据
Why Do You Run
12-26 1460
我这边遇到的问题是由于手动维护了offset，指定了分组 我新增了一个Topic，新创建了一个Consumer。但是保存Redis里面的分组还是原来的test 所以消费不到啊！！！！！ ...
sparkstreaming对接kafka数据，在idea程序上出现警告信息并卡住不动的解决办法
anddyhua's Cookbook
05-02 842
spark jira 用中文的同学看了我笑了，牛啊 Unexpected delay before fetch response transmission Details Type:Bug Status:Open Priority: Major Resolution: Unresolved Affects Version/s: 0.9.0.0, 0.9.0.1, 0.10.0.0, (3) 0.10.0.1, 0.10.1.0, 0.10.1.1 Fix Version/s:None Component
Spark Streaming中空RDD处理及流处理程序 的结束
weixin_33704591的博客
06-07 141
2019独角兽企业重金招聘Python工程师标准>>> ...
spark-streaming-kafka-0-8_2.11-2.1.0.jar下载
最新发布
07-27
要下载 spark-streaming-kafka-0-8_2.11-2.1.0.jar，首先需要打开一个可用的Web浏览器，并前往一个可靠的源（如Apache官方网站或Maven中央存储库）。然后，按照以下步骤进行下载： 1. 在浏览器的搜索栏中键入“spark-streaming-kafka-0-8_2.11-2.1.0.jar”。 2. 在搜索结果中选择可靠的源。 3. 进入所选择的源的网站并找到Spark Streaming Kafka 0.8库的下载页面。 4. 确保选择与您的编译环境和Spark版本相匹配的jar文件。例如，对于Scala版本2.11和Spark版本2.1.0，您需要选择spark-streaming-kafka-0-8_2.11-2.1.0.jar。 5. 单击下载链接来开始下载该文件。请注意，您可能需要花费一些时间等待下载完成，具体时间取决于您的网络速度和连接稳定性。 下载完成后，您可以将spark-streaming-kafka-0-8_2.11-2.1.0.jar添加到您的项目中，以便在Spark Streaming应用程序中使用Kafka 0.8版本的集成功能。您可以通过将jar文件添加到您的项目的类路径中或使用构建管理工具（如Maven或SBT）来导入该库。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值