spark Hadoop 高可用模式下读写hdfs

第一种，通过配置文件

  val sc = new SparkContext()

    sc.hadoopConfiguration.set("fs.defaultFS", "hdfs://ns");
    sc.hadoopConfiguration.set("dfs.nameservices", "ns");
    sc.hadoopConfiguration.set("dfs.ha.namenodes.ns", "nn1,nn2");
    sc.hadoopConfiguration.set("dfs.namenode.rpc-address.ns.nn1", "hadoop01:9000");
    sc.hadoopConfiguration.set("dfs.namenode.rpc-address.ns.nn2", "hadoop02:9000");
    sc.hadoopConfiguration.set("dfs.client.failover.proxy.provider.ns", "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");

第二种，通过Java代码

   val conf = new SparkConf().setAppName("Spark Word Count") 
    val sc = new SparkContext()
    sc.hadoopConfiguration.addResource("ns/core-site.xml")
    sc.hadoopConfiguration.addResource("ns/hdfs-site.xml")

需要引入hadoop-hdfs包hadoop-client包