Hadoop HA 模式下运行spark 程序

最新推荐文章于 2022-09-17 23:31:44 发布

sysmedia

最新推荐文章于 2022-09-17 23:31:44 发布

阅读量1k

点赞数

分类专栏： spark 文章标签： spark

本文链接：https://blog.csdn.net/sysmedia/article/details/71170414

版权

spark 专栏收录该内容

38 篇文章 1 订阅

订阅专栏

（1）将Hadoop的hdfs-site.xml 和core-site.xml文件复制到spark/conf目录下

（2）追加如下内容到 spark-defaults.conf文件

spark.files file:///home/hadoop/spark/conf/hdfs-site.xml,file:///home/hadoop/spark/conf/core-site.xml

spark.files file:///home/hadoop/spark/conf/hdfs-site.xml,file:///home/hadoop/spark/conf/core-site.xml

如果不加这个，会有如下问题发生：

Java.lang.IllegalArgumentException: java.NET.UnknownHostException: mycluster
   at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
   at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:231)
   at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:139)
   at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510)
   at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)

（3）读取hdfs中的lzo文件，并且分片来执行

[html] view plain copy print ?

import org.apache.hadoop.io._
import com.hadoop.mapreduce._
val data = sc.newAPIHadoopFile[LongWritable, Text, LzoTextInputFormat]("hdfs://mycluster/user/hive/warehouse/logs_app_nginx/logdate=20160322/loghost=70/var.log.nginx.access_20160322.log.70.lzo")
data.count()

import org.apache.hadoop.io._
import com.hadoop.mapreduce._
val data = sc.newAPIHadoopFile[LongWritable, Text, LzoTextInputFormat]("hdfs://mycluster/user/hive/warehouse/logs_app_nginx/logdate=20160322/loghost=70/var.log.nginx.access_20160322.log.70.lzo")
data.count()

（4）读取hdfs中的通配符表示的目录和子目录下文件，并且分片来执行

[html] view plain copy print ?

import org.apache.hadoop.io._
import com.hadoop.mapreduce._
val dirdata = sc.newAPIHadoopFile[LongWritable, Text, LzoTextInputFormat]("hdfs://mycluster/user/hive/warehouse/logs_app_nginx/logdate=20160322/loghost=*/")
dirdata.count()