Spark 初始化对象

Spark 1+ 和 Spark 2+ 初始化SparkContext有所区别，现列出Spark 1.5.1 和 Spark 2+中初始化sc方式：

1. Spark 2+

先创建一个SparkSession对象:

通过config方法配置自定义设置，可以写多个config设置

val spark = SparkSession.builder() // Creates a [[SparkSession.Builder]] for constructing a [[SparkSession]].  
  .appName("Word Count") // Sets a name for the application, which will be shown in the Spark web UI. If no application name is set, a randomly generated name will be used.
  .config("spark.some.config.option", "some-value") //Sets a config option. Options set using this method are automatically propagated to both `SparkConf` and SparkSession's own configuration.
  .enableHiveSupport() // 启用HiveContext
  .config("spark.sql.parquet.writeLegacyFormat", true) //use the same convention as Hive for writing the Parquet data
  .getOrCreate()  //Gets an existing [[SparkSession]] or, if there is no existing one, creates a new  one based on the options set in this builder.
  

val sc = spark.sparkContext

import spark.implicits._ 
spark.sql("select 123")  # 调用sql

注意：

Spark sql Insert 数据时可能报错：

Failed with exception java.io.IOException:org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file hdfs

原因及解决：

config("spark.sql.parquet.writeLegacyFormat", true) 

Root Cause:
This issue is caused because of different parquet conventions used in Hive and Spark. In Hive, the decimal datatype is represented as fixed bytes (INT 32). In Spark 1.4 or later the default convention is to use the Standard Parquet representation for decimal data type. As per the Standard Parquet representation based on the precision of the column datatype, the underlying representation changes.
eg: DECIMAL can be used to annotate the following types: int32: for 1 <= precision <= 9 int64: for 1 <= precision <= 18; precision < 10 will produce a warning

Hence this issue happens only with the usage of datatypes which have different representations in the different Parquet conventions. If the datatype is DECIMAL (10,3), both the conventions represent it as INT32, hence we won't face an issue. If you are not aware of the internal representation of the datatypes it is safe to use the same convention used for writing while reading. With Hive, you do not have the flexibility to choose the Parquet convention. But with Spark, you do.

2. Spark 1.5.0