hadoop
WEI_69
这个作者很懒,什么都没留下…
展开
-
HDFS的shell操作(大数据技术原理与应用实验内容)
#1hdfs dfs -test -e remote.txt echo $? hdfs dfs -appendToFile local.txt remote.txt #追加到原文件末尾 hdfs dfs -copyFromLocal -f local.txt remote.txt #覆盖原来文件,第一种命令形式 hdfs dfs -cp -f file:///home/hadoop/l...原创 2019-12-03 22:23:19 · 1398 阅读 · 0 评论 -
spark的RDD操作2
scala> val rawRDDA=sc.parallelize(List("!! bb ## cc","%% cc bb %%","cc && ++ aa"),3)rawRDDA: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[46] at parallelize at <console>:...原创 2019-12-03 22:20:08 · 171 阅读 · 0 评论 -
spark的RDD操作1
scala> val textFile=sc.textFile("file:///usr/local/spark/README.md")textFile: org.apache.spark.rdd.RDD[String] = file:///usr/local/spark/README.md MapPartitionsRDD[79] at textFile at <console&g...原创 2019-12-03 22:16:27 · 257 阅读 · 0 评论 -
Hadoop(3.2.0) 伪分布式wordcount
hadoop wordcount bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar grep /input output 'dfs[a-z.]+'hdfs dfs -lshdfs dfs put ./input inputhdfs dfs mkdir input原创 2019-09-16 09:31:36 · 277 阅读 · 0 评论 -
mapreduce运行wordcount
环境阿里轻量应用服务器hadoop3.2.0java1.8主程序//package WordCount; import java.io.IOException;import java.util.Iterator;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;...原创 2019-11-29 20:40:12 · 175 阅读 · 0 评论