Spark 集成hadoop ha

spark集成hadoop ha

1.复制core-site.xml + hdfs-site.xml到spark/conf目录下

xcall cp /soft/hadoop/etc/hadoop/core-site.xml /soft/spark/conf/
xcall cp /soft/hadoop/etc/hadoop/hdfs-site.xml /soft/spark/conf/

2.分发文件到spark所有work节点
3.启动spark集群
4.启动spark-shell,连接spark集群上
        $>spark-shell --master spark://s201:7077
        $scala>sc.textFile("hdfs://mycluster/user/centos/test.txt").collect();