Spark3读S3上文件
一、启动Spark-shell
- 本地启动spark-shell
./spark-shell
二、读文件
val rdd =spark.sparkContext.textFile(“s3a://opticsprod/dwh/ods_prod/software_1h/datetime=2020070919/test.parquet”);
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.0
/_/
Using Scala version 2.12.10 (Java HotSpot(TM