Spark本地测试要点

1.hive元数据访问地址，一般走thrift协议

System.setProperty("hive.metastore.uris","thrift://ip:9083")

2.hive数据访问目录，HDFS协议路径

System.setProperty("spark.sql.warehouse.dir","hdfs://ip:8020/user/hive/warehouse")

3.本地模式，流处理必须是2个及以上线程

System.setProperty("spark.master","local[*]")

4.hive本地缓存目录

System.setProperty("hive.exec.scratchdir","file:///C:\tmp\hive")

5.hadoop本地目录

System.setProperty("hadoop.home.dir","D:\apache\hadoop-2.7.1")

6.RDD默认并行度

System.setProperty("spark.default.parallelism","1")

7.SQL默认并行度

System.setProperty("spark.sql.shuffle.partitions","1")

8.hadoop用户名

System.setProperty("HADOOP_USER_NAME","admin")