从 json 文件读数据
json 文件内容示例
{"id":1, "name":"leo", "age":18}
{"id":2, "name":"jack", "age":19}
{"id":3, "name":"marry", "age":17}
从 json 文件读取
SparkSession spark = SparkSession.builder()
.appName("QSH")
.master("local[*]")
.config("spark.some.config.option", "some-value").getOrCreate();
Dataset<Row> df = spark.read().json("D:\\Study_Space\\SparkSQLTest\\json\\student.json");
df.show();
从MySQL数据库读取数据
从MySQL数据库读取
SparkConf sc = new SparkConf().setAppName("QSH").setMaster("local");
SparkContext sparkContext = new SparkContext(sc);
SQLContext sqlContext = new SQLContext(sparkContext);
String url = "jdbc:mysql://localhost:3306/qsh";
String table_Name = "tb_student";
Properties prop = new Properties();
prop.put("username", "root");
prop.put("password", "");
prop.put("driver", "com.mysql.jdbc.Driver");
Dataset<Row> ds = sqlContext.read().jdbc(url, table_Name, prop);
ds.show();
输出效果
+------+-----+-----+
|stu_ID|class|score|
+------+-----+-----+
| 1| A| 20|
| 2| A| 30|
| 3| A| 70|
| 4| B| 60|
| 5| B| 70|
| 6| B| 80|
+------+-----+-----+
读取后数据类型皆为 Dataset<Row>