spark on hdfs spark处理hdfs上的文件简单的wordcount

最新推荐文章于 2023-12-26 17:20:57 发布

JokerCao

最新推荐文章于 2023-12-26 17:20:57 发布

阅读量3.3k

点赞数

分类专栏： spark 文章标签： hadoop2.2.0 spark hdfs

本文链接：https://blog.csdn.net/sunflower_cao/article/details/26469553

版权

本文通过Spark Shell演示了如何在Hadoop HDFS上读取文件并执行WordCount操作。首先，进入Spark的bin目录启动Spark Shell，接着加载HDFS中的文件，然后使用flatMap、map和reduceByKey函数进行单词拆分、计数，最后收集结果。相比于MapReduce，Spark的代码更简洁，执行速度更快。

摘要由CSDN通过智能技术生成

进入spark/bin 目录输入spark-shell 进入spark shell模式

[hadoop@localhost bin]$ spark-shell
14/05/21 14:04:59 INFO spark.HttpServer: Starting HTTP Server
14/05/21 14:04:59 INFO server.Server: jetty-7.x.y-SNAPSHOT
14/05/21 14:04:59 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:40562
Welcome to
      ____              __
     / __/__ ___ _____/ /__
    _\ \/ _ \/ _ `/ __/ '_/
   /___/ .__/\_,_/_/ /_/\_\   version 0.9.1
      /_/

Using Scala version 2.10.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_55)
Type in expressions to have them evaluated.
Type :help for more information.
14/05/21 14:05:05 INFO slf4j.Slf4jLogger: Slf4jLogger started
14/05/21 14:05:05 INFO Remoting: Starting remoting
14/05/21 14:05:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@localhost:57902]
14/05/21 14:05:05 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@localhost:57902]
14/05/21 14:05:05 INFO spark.SparkEnv: Registering BlockManagerMaster
14/05/21 14:05:05 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140521140505-3f53
14/05/21 14:05:05 INFO storage.MemoryStore: MemoryStore started with capacity 294.9 MB.
14/05/21 14:05:06 INFO network.ConnectionManager: Bound socket to port 48084 with id = ConnectionManagerId(localhost,48084)
14/05/21 14:05:06 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/05/21 14:05:06 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager localhost:48084 with 294.9 MB RAM
14/05/21 14:05:06 INFO storage.BlockManagerMaster: Registered BlockManager
14/05/21 14:05:06 INFO spark.HttpServer: Starting HTTP Server
14/05/21 14:05:06 INFO server.Server: jetty-7.x.y-SNAPSHOT
14/05/21 14:05:06 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48508
14/05/21 14:05:06 INFO broadcast.HttpBroadcast: Broadcast server started at http://localhost:48508
14/05/21 14:05:06 INFO spark.SparkEnv: Registering MapOutputTracker
14/05/21 14:05:06 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-862d3bfc-8485-4adb-86a3-efd09c90fc03
14/05/21 14:05:06 INFO s