DaiSnow的博客

1.读取cvs文件数据写入parquet文件 val spark =SparkSession.builder().appName("test").master("local").getOrCreate() val file = "hdfs://clusters/test/demo.csv" val frame = spark.read.option("header","true").csv(file) val newfile = file.split("\\.")(0)+".parque

JAVA API实现hdfs的文件操作需要三个依赖包hadoop-common、hadoop-hdfs和hadoop-client。<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.6.0</version></dependency><de

我们知道在虚拟机中通过命令可以实现Linux本地文件log1.txt上传到HDFS中，以及从HDFS中下载文件。#上传文件（向HDFS中写文件）hdfs dfs -put /opt/log.txt /hdfs/log/log1.txt或hdfs dfs -copyFromLocal /opt/log.txt /hdfs/log/log2.txt#下载文件（从HDFS中读文件）hdfs dfs -get /hdfs/log/log.txt /opt/log1.txt或hdfs dfs -cop