Spark
Joseph25
这个作者很懒,什么都没留下…
展开
-
Spark:failed to launch: nice -n 0 /opt/spark/bin/spark-class org.apache.spark.deploy.worker.
node03: failed to launch: nice -n 0 /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://node01:7077node03: full log in /opt/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node03.out原创 2022-09-03 16:28:03 · 2436 阅读 · 0 评论 -
Spark:spark2.4.0安装
软件准备:Index of /dist/spark,选择跟hadoop集成的版本1,解压:tar -zxvf spark-2.4.0-bin-hadoop2.6.tgz mv spark-2.4.0-bin-hadoop2.6 spark vim /etc/profile.d/bigdata-etc.shexport SPARK_HOME=/opt/sparkexport PATH=$PATH:$SPARK_HOME/bin:$S......原创 2022-05-31 16:44:19 · 1171 阅读 · 0 评论 -
Spark:spark on yarn任务java.nio.channels.ClosedChannelException
原因:给节点分配的内存少,yarn kill了spark application解决:配置yarn-site.xml<property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value></property><property> <name>yarn.nodemanager.vmem-che......转载 2022-04-02 13:47:42 · 322 阅读 · 0 评论 -
Spark:“main” java.lang.IllegalArgumentException: Illegal pattern component: XXX 报错
Spark2.3.0解决Exception in thread “main” java.lang.IllegalArgumentException: Illegal pattern component: XXX 报错出错的调用代码此问题出现在调用spark.read.json或者csv的时候出现。res.write.mode(“append”).json(“c://out”)123说明maven升级的时候,没有自动加载完整依赖包,jsonAPI对于timeStampFormat有特殊需.转载 2021-04-16 15:46:42 · 499 阅读 · 0 评论 -
SparkStream与Kafka连接java.nio.channels.ClosedChannelException
Caused by: java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) at ...原创 2019-03-06 11:24:34 · 591 阅读 · 0 评论 -
RDD之flatMap与Map对比
定义首先我们列出flatMap与Map的定义,可参考RDD API def map[U](f: (T) ⇒ U)(implicit arg0: ClassTag[U]): RDD[U] Return a new RDD by applying a function to all elements of this RDD. def flat...转载 2019-01-04 14:46:50 · 182 阅读 · 0 评论