问题汇总
问题1
Caused by: java.lang.AssertionError: assertion failed: Failed to get records for spark-executor-abcd1 test1 8 1 after polling for 512
at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:74)
at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:223)
at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:189)
解决方案
spark.streaming.kafka.consumer.poll.ms
问题2
解决方案
把上面代码移动进forechRDD里面
问题3
当我把Spark的版本从2.0提高到2.3的时候，运行程序报如下错误
/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/bin/java "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=49987:/Applications/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/tools.jar:/Users/Aaron/Downloads/LogAnalysis/target/classes:/Users/Aaron/.m2/repository/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-client/2.6.0-cdh5.10.0/hadoop-client-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.6.0-cdh5.10.0/hadoop-mapreduce-client-app-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.6.0-cdh5.10.0/hadoop-mapreduce-client-common-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.6.0-cdh5.10.0/hadoop-yarn-client-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.6.0-cdh5.10.0/hadoop-yarn-server-common-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.6.0-cdh5.10.0/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.6.0-cdh5.10.0/hadoop-yarn-api-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.6.0-cdh5.10.0/hadoop-mapreduce-client-core-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.6.0-cdh5.10.0/hadoop-yarn-common-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/Aaron/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/Aaron/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/Aaron/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.6.0-cdh5.10.0/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-aws/2.6.0-cdh5.10.0/hadoop-aws-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/com/amazonaws/aws-java-sdk-s3/1.10.6/aws-java-sdk-s3-1.10.6.jar:/Users/Aaron/.m2/repository/com/amazonaws/aws-java-sdk-kms/1.10.6/aws-java-sdk-kms-1.10.6.jar:/Users/Aaron/.m2/repository/com/amazonaws/aws-java-sdk-core/1.10.6/aws-java-sdk-core-1.10.6.jar:/Users/Aaron/.m2/repository/com/amazonaws/aws-java-sdk-sts/1.10.6/aws-java-sdk-sts-1.10.6.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-annotations/2.6.0-cdh5.10.0/hadoop-annotations-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0-cdh5.10.0/hadoop-common-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/Aaron/.m2/repository/org/apache/commons/commons-math3/3.1.1/commons-math3-3.1.1.jar:/Users/Aaron/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/Aaron/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/Aaron/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/Aaron/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/Aaron/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/Users/Aaron/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/Aaron/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/Users/Aaron/.m2/repository/org/mortbay/jetty/jetty/6.1.26.cloudera.4/jetty-6.1.26.cloudera.4.jar:/Users/Aaron/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26.cloudera.4/jetty-util-6.1.26.cloudera.4.jar:/Users/Aaron/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/Users/Aaron/.m2/repository/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/Users/Aaron/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/Aaron/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/Aaron/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar:/Users/Aaron/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar:/Users/Aaron/.m2/repository/com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar:/Users/Aaron/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/Aaron/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar:/Users/Aaron/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar:/Users/Aaron/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/Aaron/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/Aaron/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/Aaron/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/Aaron/.m2/repository/net/java/dev/jets3t/jets3t/0.9.0/jets3t-0.9.0.jar:/Users/Aaron/.m2/repository/org/apache/httpcomponents/httpcore/4.1.2/httpcore-4.1.2.jar:/Users/Aaron/.m2/repository/com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar:/Users/Aaron/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/Aaron/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/Aaron/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/Aaron/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/Aaron/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/Aaron/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/Users/Aaron/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar:/Users/Aaron/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:/Users/Aaron/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/Users/Aaron/.m2/repository/org/apache/avro/avro/1.7.6-cdh5.10.0/avro-1.7.6-cdh5.10.0.jar:/Users/Aaron/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/Users/Aaron/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/Aaron/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-auth/2.6.0-cdh5.10.0/hadoop-auth-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/Aaron/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/Users/Aaron/.m2/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/Users/Aaron/.m2/repository/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/Users/Aaron/.m2/repository/org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.jar:/Users/Aaron/.m2/repository/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar:/Users/Aaron/.m2/repository/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar:/Users/Aaron/.m2/repository/org/apache/curator/curator-recipes/2.7.1/curator-recipes-2.7.1.jar:/Users/Aaron/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.jar:/Users/Aaron/.m2/repository/org/apache/htrace/htrace-core4/4.0.1-incubating/htrace-core4-4.0.1-incubating.jar:/Users/Aaron/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5-cdh5.10.0/zookeeper-3.4.5-cdh5.10.0.jar:/Users/Aaron/.m2/repository/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/Users/Aaron/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar:/Users/Aaron/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.6.0-cdh5.10.0/hadoop-hdfs-2.6.0-cdh5.10.0.jar:/Users/Aaron/.m2/repository/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar:/Users/Aaron/.m2/repository/io/netty/netty/3.10.5.Final/netty-3.10.5.Final.jar:/Users/Aaron/.m2/repository/io/netty/netty-all/4.0.23.Final/netty-all-4.0.23.Final.jar:/Users/Aaron/.m2/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar:/Users/Aaron/.m2/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar:/Users/Aaron/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-core_2.11/2.3.0/spark-core_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/apache/avro/avro-mapred/1.7.7/avro-mapred-1.7.7-hadoop2.jar:/Users/Aaron/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7.jar:/Users/Aaron/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7-tests.jar:/Users/Aaron/.m2/repository/com/twitter/chill_2.11/0.8.4/chill_2.11-0.8.4.jar:/Users/Aaron/.m2/repository/com/esotericsoftware/kryo-shaded/3.0.3/kryo-shaded-3.0.3.jar:/Users/Aaron/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/Users/Aaron/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/Users/Aaron/.m2/repository/com/twitter/chill-java/0.8.4/chill-java-0.8.4.jar:/Users/Aaron/.m2/repository/org/apache/xbean/xbean-asm5-shaded/4.4/xbean-asm5-shaded-4.4.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-launcher_2.11/2.3.0/spark-launcher_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-kvstore_2.11/2.3.0/spark-kvstore_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar:/Users/Aaron/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.7/jackson-annotations-2.6.7.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-network-common_2.11/2.3.0/spark-network-common_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-network-shuffle_2.11/2.3.0/spark-network-shuffle_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-unsafe_2.11/2.3.0/spark-unsafe_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/Aaron/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/Users/Aaron/.m2/repository/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/Users/Aaron/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/Users/Aaron/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/Aaron/.m2/repository/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar:/Users/Aaron/.m2/repository/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0.jar:/Users/Aaron/.m2/repository/com/github/luben/zstd-jni/1.3.2-2/zstd-jni-1.3.2-2.jar:/Users/Aaron/.m2/repository/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/Users/Aaron/.m2/repository/org/json4s/json4s-jackson_2.11/3.2.11/json4s-jackson_2.11-3.2.11.jar:/Users/Aaron/.m2/repository/org/json4s/json4s-core_2.11/3.2.11/json4s-core_2.11-3.2.11.jar:/Users/Aaron/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar:/Users/Aaron/.m2/repository/org/scala-lang/scalap/2.11.0/scalap-2.11.0.jar:/Users/Aaron/.m2/repository/org/scala-lang/scala-compiler/2.11.0/scala-compiler-2.11.0.jar:/Users/Aaron/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.1/scala-xml_2.11-1.0.1.jar:/Users/Aaron/.m2/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/Users/Aaron/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/Users/Aaron/.m2/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/Users/Aaron/.m2/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/Users/Aaron/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/Users/Aaron/.m2/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/Users/Aaron/.m2/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/Users/Aaron/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/Aaron/.m2/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/Users/Aaron/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/Users/Aaron/.m2/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/Users/Aaron/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/Users/Aaron/.m2/repository/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/Users/Aaron/.m2/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/Users/Aaron/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/Aaron/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/Users/Aaron/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/Users/Aaron/.m2/repository/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/Users/Aaron/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.5/metrics-core-3.1.5.jar:/Users/Aaron/.m2/repository/io/dropwizard/metrics/metrics-jvm/3.1.5/metrics-jvm-3.1.5.jar:/Users/Aaron/.m2/repository/io/dropwizard/metrics/metrics-json/3.1.5/metrics-json-3.1.5.jar:/Users/Aaron/.m2/repository/io/dropwizard/metrics/metrics-graphite/3.1.5/metrics-graphite-3.1.5.jar:/Users/Aaron/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar:/Users/Aaron/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.7.1/jackson-module-scala_2.11-2.6.7.1.jar:/Users/Aaron/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/Users/Aaron/.m2/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.7.9/jackson-module-paranamer-2.7.9.jar:/Users/Aaron/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/Aaron/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/Aaron/.m2/repository/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/Users/Aaron/.m2/repository/net/sf/py4j/py4j/0.10.6/py4j-0.10.6.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-tags_2.11/2.3.0/spark-tags_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/Users/Aaron/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-sql_2.11/2.3.0/spark-sql_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/com/univocity/univocity-parsers/2.5.9/univocity-parsers-2.5.9.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-sketch_2.11/2.3.0/spark-sketch_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-catalyst_2.11/2.3.0/spark-catalyst_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/codehaus/janino/janino/3.0.8/janino-3.0.8.jar:/Users/Aaron/.m2/repository/org/codehaus/janino/commons-compiler/3.0.8/commons-compiler-3.0.8.jar:/Users/Aaron/.m2/repository/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.jar:/Users/Aaron/.m2/repository/org/apache/orc/orc-core/1.4.1/orc-core-1.4.1-nohive.jar:/Users/Aaron/.m2/repository/io/airlift/aircompressor/0.8/aircompressor-0.8.jar:/Users/Aaron/.m2/repository/org/apache/orc/orc-mapreduce/1.4.1/orc-mapreduce-1.4.1-nohive.jar:/Users/Aaron/.m2/repository/org/apache/parquet/parquet-column/1.8.2/parquet-column-1.8.2.jar:/Users/Aaron/.m2/repository/org/apache/parquet/parquet-common/1.8.2/parquet-common-1.8.2.jar:/Users/Aaron/.m2/repository/org/apache/parquet/parquet-encoding/1.8.2/parquet-encoding-1.8.2.jar:/Users/Aaron/.m2/repository/org/apache/parquet/parquet-hadoop/1.8.2/parquet-hadoop-1.8.2.jar:/Users/Aaron/.m2/repository/org/apache/parquet/parquet-format/2.3.1/parquet-format-2.3.1.jar:/Users/Aaron/.m2/repository/org/apache/parquet/parquet-jackson/1.8.2/parquet-jackson-1.8.2.jar:/Users/Aaron/.m2/repository/org/apache/arrow/arrow-vector/0.8.0/arrow-vector-0.8.0.jar:/Users/Aaron/.m2/repository/org/apache/arrow/arrow-format/0.8.0/arrow-format-0.8.0.jar:/Users/Aaron/.m2/repository/org/apache/arrow/arrow-memory/0.8.0/arrow-memory-0.8.0.jar:/Users/Aaron/.m2/repository/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.jar:/Users/Aaron/.m2/repository/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-streaming_2.11/2.3.0/spark-streaming_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-hive_2.11/2.3.0/spark-hive_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/com/twitter/parquet-hadoop-bundle/1.6.0/parquet-hadoop-bundle-1.6.0.jar:/Users/Aaron/.m2/repository/org/spark-project/hive/hive-exec/1.2.1.spark2/hive-exec-1.2.1.spark2.jar:/Users/Aaron/.m2/repository/javolution/javolution/5.5.1/javolution-5.5.1.jar:/Users/Aaron/.m2/repository/log4j/apache-log4j-extras/1.2.17/apache-log4j-extras-1.2.17.jar:/Users/Aaron/.m2/repository/org/antlr/antlr-runtime/3.4/antlr-runtime-3.4.jar:/Users/Aaron/.m2/repository/org/antlr/stringtemplate/3.2.1/stringtemplate-3.2.1.jar:/Users/Aaron/.m2/repository/antlr/antlr/2.7.7/antlr-2.7.7.jar:/Users/Aaron/.m2/repository/org/antlr/ST4/4.0.4/ST4-4.0.4.jar:/Users/Aaron/.m2/repository/com/googlecode/javaewah/JavaEWAH/0.3.2/JavaEWAH-0.3.2.jar:/Users/Aaron/.m2/repository/org/iq80/snappy/snappy/0.2/snappy-0.2.jar:/Users/Aaron/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/Aaron/.m2/repository/net/sf/opencsv/opencsv/2.3/opencsv-2.3.jar:/Users/Aaron/.m2/repository/org/spark-project/hive/hive-metastore/1.2.1.spark2/hive-metastore-1.2.1.spark2.jar:/Users/Aaron/.m2/repository/com/jolbox/bonecp/0.8.0.RELEASE/bonecp-0.8.0.RELEASE.jar:/Users/Aaron/.m2/repository/org/datanucleus/datanucleus-api-jdo/3.2.6/datanucleus-api-jdo-3.2.6.jar:/Users/Aaron/.m2/repository/org/datanucleus/datanucleus-rdbms/3.2.9/datanucleus-rdbms-3.2.9.jar:/Users/Aaron/.m2/repository/commons-pool/commons-pool/1.5.4/commons-pool-1.5.4.jar:/Users/Aaron/.m2/repository/commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar:/Users/Aaron/.m2/repository/javax/jdo/jdo-api/3.0.1/jdo-api-3.0.1.jar:/Users/Aaron/.m2/repository/javax/transaction/jta/1.1/jta-1.1.jar:/Users/Aaron/.m2/repository/org/apache/calcite/calcite-avatica/1.2.0-incubating/calcite-avatica-1.2.0-incubating.jar:/Users/Aaron/.m2/repository/org/apache/calcite/calcite-core/1.2.0-incubating/calcite-core-1.2.0-incubating.jar:/Users/Aaron/.m2/repository/org/apache/calcite/calcite-linq4j/1.2.0-incubating/calcite-linq4j-1.2.0-incubating.jar:/Users/Aaron/.m2/repository/net/hydromatic/eigenbase-properties/1.1.5/eigenbase-properties-1.1.5.jar:/Users/Aaron/.m2/repository/org/apache/httpcomponents/httpclient/4.5.4/httpclient-4.5.4.jar:/Users/Aaron/.m2/repository/joda-time/joda-time/2.9.3/joda-time-2.9.3.jar:/Users/Aaron/.m2/repository/org/jodd/jodd-core/3.5.2/jodd-core-3.5.2.jar:/Users/Aaron/.m2/repository/org/datanucleus/datanucleus-core/3.2.10/datanucleus-core-3.2.10.jar:/Users/Aaron/.m2/repository/org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar:/Users/Aaron/.m2/repository/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar:/Users/Aaron/.m2/repository/org/apache/derby/derby/10.12.1.1/derby-10.12.1.1.jar:/Users/Aaron/.m2/repository/org/apache/spark/spark-streaming-kafka-0-10_2.11/2.3.0/spark-streaming-kafka-0-10_2.11-2.3.0.jar:/Users/Aaron/.m2/repository/org/apache/kafka/kafka_2.11/0.10.0-kafka-2.1.1/kafka_2.11-0.10.0-kafka-2.1.1.jar:/Users/Aaron/.m2/repository/net/sf/jopt-simple/jopt-simple/4.9/jopt-simple-4.9.jar:/Users/Aaron/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar:/Users/Aaron/.m2/repository/com/101tec/zkclient/0.8/zkclient-0.8.jar:/Users/Aaron/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar:/Users/Aaron/.m2/repository/org/apache/kafka/kafka-clients/0.10.0-kafka-2.1.1/kafka-clients-0.10.0-kafka-2.1.1.jar:/Users/Aaron/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar:/Users/Aaron/.m2/repository/mysql/mysql-connector-java/6.0.6/mysql-connector-java-6.0.6.jar:/Users/Aaron/.m2/repository/redis/clients/jedis/2.9.0/jedis-2.9.0.jar:/Users/Aaron/.m2/repository/org/apache/commons/commons-pool2/2.4.2/commons-pool2-2.4.2.jar:/Users/Aaron/.m2/repository/org/influxdb/influxdb-java/2.5/influxdb-java-2.5.jar:/Users/Aaron/.m2/repository/com/squareup/retrofit2/retrofit/2.1.0/retrofit-2.1.0.jar:/Users/Aaron/.m2/repository/com/squareup/retrofit2/converter-moshi/2.1.0/converter-moshi-2.1.0.jar:/Users/Aaron/.m2/repository/com/squareup/moshi/moshi/1.2.0/moshi-1.2.0.jar:/Users/Aaron/.m2/repository/com/squareup/okhttp3/okhttp/3.5.0/okhttp-3.5.0.jar:/Users/Aaron/.m2/repository/com/squareup/okio/okio/1.11.0/okio-1.11.0.jar:/Users/Aaron/.m2/repository/com/squareup/okhttp3/logging-interceptor/3.5.0/logging-interceptor-3.5.0.jar:/Users/Aaron/.m2/repository/com/google/guava/guava/20.0/guava-20.0.jar:/Users/Aaron/.m2/repository/com/alibaba/fastjson/1.2.47/fastjson-1.2.47.jar:/Users/Aaron/.m2/repository/org/scalikejdbc/scalikejdbc_2.11/3.3.5/scalikejdbc_2.11-3.3.5.jar:/Users/Aaron/.m2/repository/org/scalikejdbc/scalikejdbc-core_2.11/3.3.5/scalikejdbc-core_2.11-3.3.5.jar:/Users/Aaron/.m2/repository/org/apache/commons/commons-dbcp2/2.5.0/commons-dbcp2-2.5.0.jar:/Users/Aaron/.m2/repository/org/scala-lang/modules/scala-collection-compat_2.11/0.1.1/scala-collection-compat_2.11-0.1.1.jar:/Users/Aaron/.m2/repository/org/scalikejdbc/scalikejdbc-interpolation_2.11/3.3.5/scalikejdbc-interpolation_2.11-3.3.5.jar:/Users/Aaron/.m2/repository/org/scalikejdbc/scalikejdbc-interpolation-macro_2.11/3.3.5/scalikejdbc-interpolation-macro_2.11-3.3.5.jar com.onlinelog.www.OnLineLogAnalysisV2
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/07/31 12:22:35 INFO SparkContext: Running Spark version 2.3.0
19/07/31 12:22:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/07/31 12:22:36 INFO SparkContext: Submitted application: OnLineLogAnalysis
19/07/31 12:22:36 INFO SecurityManager: Changing view acls to: Aaron
19/07/31 12:22:36 INFO SecurityManager: Changing modify acls to: Aaron
19/07/31 12:22:36 INFO SecurityManager: Changing view acls groups to:
19/07/31 12:22:36 INFO SecurityManager: Changing modify acls groups to:
19/07/31 12:22:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Aaron); groups with view permissions: Set(); users with modify permissions: Set(Aaron); groups with modify permissions: Set()
Exception in thread "main" java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric;
at org.apache.spark.network.util.NettyMemoryMetrics.registerMetrics(NettyMemoryMetrics.java:80)
at org.apache.spark.network.util.NettyMemoryMetrics.<init>(NettyMemoryMetrics.java:76)
at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:109)
at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:249)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:256)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:423)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at com.onlinelog.www.OnLineLogAnalysisV2$.main(OnLineLogAnalysisV2.scala:27)
at com.onlinelog.www.OnLineLogAnalysisV2.main(OnLineLogAnalysisV2.scala)
解决方案
dependencyManagement>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.1.18.Final</version>
</dependency>
</dependencies>
</dependencyManagemen
问题4
java.lang.ClassCastException: org.apache.spark.rdd.MapPartitionsRDD cannot be cast to org.apache.spark.streaming.kafka010.HasOffsetRanges
at com.onlinelog.www.OnLineLogAnalysisV3$$anonfun$main$3.apply(OnLineLogAnalysisV3.scala:78)
at com.onlinelog.www.OnLineLogAnalysisV3$$anonfun$main$3.apply(OnLineLogAnalysisV3.scala:73)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
java.lang.ClassCastException: org.apache.spark.rdd.MapPartitionsRDD cannot be cast to org.apache.spark.streaming.kafka010.HasOffsetRanges
19/08/01 17:52:32 INFO StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook
解决方案
https://blog.csdn.net/xianpanjia4616/article/details/85871063
原来的代码
val recordsDStream = KafkaUtils.createDirectStream[String, String](
ssc,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
)
recordsDStream
.map(record => {
val logJson = record.value()
if (
logJson.contains("INFO") || logJson.contains("WARN") || logJson.contains("ERROR") || logJson.contains("DEBUG") || logJson.contains("FATAL")
) {
// 解析Json
CDHLogParser.parseJson(logJson, totals, errors)
}
else {
null
}
})
.filter(_ != null)
.window(windowLength, slideInterval)
.foreachRDD(rdd => {
if (rdd.count == 0) {
logInfo("\n------ No cdh role logs in this time interval ------\n")
}
else {
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
logInfo("\n------ 数据统计记录为：" + rdd.count() + " ------\n")
修改后的代码
val recordsDStream = KafkaUtils.createDirectStream[String, String](
ssc,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
)
recordsDStream
.window(windowLength, slideInterval)
.foreachRDD(rdd => {
if (rdd.count == 0) {
logInfo("\n------ No cdh role logs in this time interval ------\n")
}
else {
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
val handledRdd = rdd.map(record => {
val logJson = record.value()
if (
logJson.contains("INFO") || logJson.contains("WARN") || logJson.contains("ERROR") || logJson.contains("DEBUG") || logJson.contains("FATAL")
) {
// 解析Json
CDHLogParser.parseJson(logJson, totals, errors)
}
else {
null
}
})
.filter(_ != null)
logInfo("\n------ 数据统计记录为：" + handledRdd.count() + " ------\n")
继续运行，又报错了
19/08/01 18:00:42 WARN BlockManager: Putting block rdd_0_0 failed due to exception java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = alert_own_made_log, partition = 2, offset = 4352, CreateTime = -1, checksum = 3767506672, serialized key size = -1, serialized value size = 407, key = null, value = {"hostname":"hadoop004","servicename":"namenode","time": "2019-08-01 16:29:06", "logtype": "INFO", "loginfo":" org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:Deleted BP-540448286-127.0.0.1-1560823920268 blk_1073742063_1239 file /data/aaron/app/hadoop-2.6.0-cdh5.7.0/tmp/dfs/data/current/BP-540448286-127.0.0.1-1560823920268/current/finalized/subdir0/subdir0/blk_1073742063"})).
19/08/01 18:00:42 WARN BlockManager: Putting block rdd_0_2 failed due to exception java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = alert_own_made_log, partition = 0, offset = 4353, CreateTime = -1, checksum = 3767506672, serialized key size = -1, serialized value size = 407, key = null, value = {"hostname":"hadoop004","servicename":"namenode","time": "2019-08-01 16:29:06", "logtype": "INFO", "loginfo":" org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:Deleted BP-540448286-127.0.0.1-1560823920268 blk_1073742063_1239 file /data/aaron/app/hadoop-2.6.0-cdh5.7.0/tmp/dfs/data/current/BP-540448286-127.0.0.1-1560823920268/current/finalized/subdir0/subdir0/blk_1073742063"})).
19/08/01 18:00:42 WARN BlockManager: Putting block rdd_0_1 failed due to exception java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = alert_own_made_log, partition = 1, offset = 4353, CreateTime = -1, checksum = 1820730753, serialized key size = -1, serialized value size = 407, key = null, value = {"hostname":"hadoop004","servicename":"namenode","time": "2019-08-01 17:22:40", "logtype": "INFO", "loginfo":" org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:Deleted BP-540448286-127.0.0.1-1560823920268 blk_1073742063_1239 file /data/aaron/app/hadoop-2.6.0-cdh5.7.0/tmp/dfs/data/current/BP-540448286-127.0.0.1-1560823920268/current/finalized/subdir0/subdir0/blk_1073742063"})).
19/08/01 18:00:42 WARN BlockManager: Block rdd_0_0 could not be removed as it was not found on disk or in memory
19/08/01 18:00:42 WARN BlockManager: Block rdd_0_1 could not be removed as it was not found on disk or in memory
19/08/01 18:00:42 WARN BlockManager: Block rdd_0_2 could not be removed as it was not found on disk or in memory
19/08/01 18:00:42 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:
- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = alert_own_made_log, partition = 2, offset = 4352, CreateTime = -1, checksum = 3767506672, serialized key size = -1, serialized value size = 407, key = null, value = {"hostname":"hadoop004","servicename":"namenode","time": "2019-08-01 16:29:06", "logtype": "INFO", "loginfo":" org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:Deleted BP-540448286-127.0.0.1-1560823920268 blk_1073742063_1239 file /data/aaron/app/hadoop-2.6.0-cdh5.7.0/tmp/dfs/data/current/BP-540448286-127.0.0.1-1560823920268/current/finalized/subdir0/subdir0/blk_1073742063"}))
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:379)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1109)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/08/01 18:00:42 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
解决方案
把window注释掉就好了
那么窗口怎么去管理Offset呢
https://blog.csdn.net/young_0609/article/details/96839025
问题5
Exception in thread "streaming-job-executor-23" java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createStarted()Lcom/google/common/base/Stopwatch;
at org.influxdb.impl.InfluxDBImpl.ping(InfluxDBImpl.java:187)
at org.influxdb.impl.InfluxDBImpl.version(InfluxDBImpl.java:210)
at com.onlinelog.www.OnLineLogAnalysisV3$$anonfun$main$1.apply(OnLineLogAnalysisV3.scala:89)
at com.onlinelog.www.OnLineLogAnalysisV3$$anonfun$main$1.apply(OnLineLogAnalysisV3.scala:60)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
解决方案
[root@hadoop004 app]# find ./ -name *guava*
./maxwell-1.22.1/lib/guava-18.0.jar
./apache-flume-1.9.0-bin/lib/guava-11.0.2.jar
./apache-flume-1.9.0-bin/docs/apidocs/org/apache/flume/source/shaded/guava
./hive-1.1.0-cdh5.7.0/lib/guava-14.0.1.jar
./apache-flume-1.7.0-bin/lib/guava-11.0.2.jar
./azkaban-solo-server-0.1.0-SNAPSHOT/lib/guava-21.0.jar
./apache-maven-3.3.9/lib/guava-18.0.jar
./hadoop-2.6.0-cdh5.7.0/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0/share/hadoop/tools/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0/share/hadoop/yarn/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0/share/hadoop/hdfs/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/guava-11.0.2.jar
./hbase-1.2.0-cdh5.12.0/lib/guava-12.0.1.jar
./apache-flume-1.6.0-cdh5.7.0-bin/lib/guava-11.0.2.jar
./elasticsearch-6.6.2/modules/x-pack-security/guava-19.0.jar
./elasticsearch-6.6.2/modules/x-pack-watcher/guava-16.0.1.jar
./spark-2.3.3-bin-2.6.0-cdh5.7.0/jars/jersey-guava-2.22.2.jar
./spark-2.3.3-bin-2.6.0-cdh5.7.0/jars/guava-14.0.1.jar
./hadoop-2.6.0-cdh5.7.0.bak/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0.bak/share/hadoop/tools/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0.bak/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0.bak/share/hadoop/yarn/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0.bak/share/hadoop/hdfs/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0.bak/share/hadoop/common/lib/guava-11.0.2.jar
./hadoop-2.6.0-cdh5.7.0.bak/tmp/nm-local-dir/filecache/203/guava-14.0.1.jar
./hadoop-2.6.0-cdh5.7.0.bak/tmp/nm-local-dir/filecache/203/.tmp_guava-14.0.1.jar.crc
./hadoop-2.6.0-cdh5.7.0.bak/tmp/nm-local-dir/filecache/147/jersey-guava-2.22.2.jar
./hadoop-2.6.0-cdh5.7.0.bak/tmp/nm-local-dir/filecache/147/.tmp_jersey-guava-2.22.2.jar.crc
./zeppelin-0.8.1-bin-all/interpreter/alluxio/guava-15.0.jar
./zeppelin-0.8.1-bin-all/interpreter/hbase/guava-11.0.2.jar
./zeppelin-0.8.1-bin-all/interpreter/python/guava-19.0.jar
./zeppelin-0.8.1-bin-all/interpreter/lens/guava-13.0.1.jar
./zeppelin-0.8.1-bin-all/interpreter/elasticsearch/guava-18.0.jar
./zeppelin-0.8.1-bin-all/interpreter/bigquery/guava-jdk5-17.0.jar
./zeppelin-0.8.1-bin-all/interpreter/cassandra/guava-16.0.1.jar
./zeppelin-0.8.1-bin-all/interpreter/scio/guava-14.0.1.jar
./zeppelin-0.8.1-bin-all/interpreter/file/jersey-guava-2.22.2.jar
./zeppelin-0.8.1-bin-all/interpreter/pig/guava-11.0.jar
./zeppelin-0.8.1-bin-all/lib/guava-20.0.jar
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava-parent
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava-parent/20.0/guava-parent-20.0.pom
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava-parent/20.0/guava-parent-20.0.pom.sha1
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava/20.0/guava-20.0.pom.sha1
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava/20.0/guava-20.0.jar
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava/20.0/guava-20.0.pom
./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava/20.0/guava-20.0.jar.sha1
./kafka_2.11-0.10.0.0/libs/jersey-guava-2.22.2.jar
./kafka_2.11-0.10.0.0/libs/guava-18.0.jar
[root@hadoop004 app]# mv ./hadoop-2.6.0-cdh5.7.0/share/hadoop/yarn/lib/guava-11.0.2.jar ./hadoop-2.6.0-cdh5.7.0/share/hadoop/yarn/lib/guava-11.0.2.jar.bak
[root@hadoop004 app]# cp ./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava/20.0/guava-20.0.jar ./hadoop-2.6.0-cdh5.7.0/share/hadoop/yarn/lib/
[root@hadoop004 app]# mv ./spark-2.3.3-bin-2.6.0-cdh5.7.0/jars/guava-14.0.1.jar ./spark-2.3.3-bin-2.6.0-cdh5.7.0/jars/guava-14.0.1.jar.bak
[root@hadoop004 app]# cp ./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava/20.0/guava-20.0.jar ./spark-2.3.3-bin-2.6.0-cdh5.7.0/jars/
[root@hadoop004 app]# mv ./hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/guava-11.0.2.jar ./hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/guava-11.0.2.jar.bak
[root@hadoop004 app]# cp ./zeppelin-0.8.1-bin-all/local-repo/com/google/guava/guava/20.0/guava-20.0.jar ./hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/
问题6
运行spark-submit，又报错了
解决方案
https://www.iteblog.com/archives/1300.html
spark-submit \
--master yarn \
--name "OnLineLogAnalysisV3" \
--conf "spark.scheduler.mode=FAIR" \
--conf "spark.sql.codegen=true" \
--num-executors 3 \
--executor-cores 1 \
--executor-memory 2G \
--jars /data/aaron/software/mysql-connector-java-5.1.27-bin.jar \
--class com.onlinelog.www.OnLineLogAnalysisV3 \
/data/aaron/lib/loganalysis/0801/loganalysis-1.0-SNAPSHOT-jar-with-dependencies.jar
改为
spark-submit \
--master yarn \
--name "OnLineLogAnalysisV3" \
--conf "spark.scheduler.mode=FAIR" \
--conf "spark.sql.codegen=true" \
--num-executors 3 \
--executor-cores 1 \
--executor-memory 2G \
--driver-class-path /data/aaron/software/mysql-connector-java-5.1.27-bin.jar \
--class com.onlinelog.www.OnLineLogAnalysisV3 \
/data/aaron/lib/loganalysis/0801/loganalysis-1.0-SNAPSHOT-jar-with-dependencies.jar
4807
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
