spark操作hdfs统计单词实例 for Eclipse

最新推荐文章于 2024-07-15 01:22:05 发布
最新推荐文章于 2024-07-15 01:22:05 发布
阅读量586 收藏 1
点赞数
分类专栏： 云平台 hadoop scala 分布式开发 存储
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/hsg77/article/details/82502497
版权

spark操作hdfs统计单词实例 for Eclipse

开发环境

eclipse 4.8.0 javaEE
安装scala插件
help/Eclipase Marketplace/find scala/Scala IDE4.7.x /install

引用库build paths…
JRE 1.8.181
scala 2.11.11

解压spark-2.3.1-bin-hadoop2.7.tgz文件
到D:\cwgis\apps\spark\
引入D:\cwgis\apps\spark\jars\目录下所有*.jar文件到
Java Build Path/Libraries/Add External JARs…

新建项目Scala Wizards/Scala Project
项目名称：SparkWordCount

新建SparkWordCount.scala文件

import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
object SparkWordCount {
     def main(args:Array[String]):Unit={
        //if(args.length<2){
        //   println("Usage:<in><out>");
        //   return;
        //}
        var in_path="";  //args(0);
        var out_path=""; //args(1);

        in_path="hdfs://192.168.145.180:8020/spark/hellospark";
        out_path="hdfs://192.168.145.180:8020/spark/output";
        println("hello spark");
        val conf=new SparkConf().setAppName("SparkWordCount");
        //set spark master url  
        conf.setMaster("local[*]");
        conf.set("spark.driver.allowMultipleContexts","true");
        //
        val sc=new SparkContext(conf);
        val textRDD=sc.textFile(in_path);
        val result=textRDD.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey((a,b)=>a+b)
        result.saveAsTextFile(out_path);
        //println(result.collect());
        println(in_path);
        println(out_path);
        println("hello end");
     }
}

执行情况

spark/hellospark文件内容：

hello spark
hello world
hello spark!

spark/output文件内容：

(spark!,1)
(hello,3)
(world,1)
(spark,1)

执行过程如下所示：

hello spark
2018-09-07 20:27:46 INFO  SparkContext:54 - Running Spark version 2.3.1
2018-09-07 20:27:46 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-07 20:27:46 INFO  SparkContext:54 - Submitted application: SparkWordCount
2018-09-07 20:27:46 INFO  SecurityManager:54 - Changing view acls to: hsg
2018-09-07 20:27:46 INFO  SecurityManager:54 - Changing modify acls to: hsg
2018-09-07 20:27:46 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-09-07 20:27:46 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-09-07 20:27:46 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hsg); groups with view permissions: Set(); users  with modify permissions: Set(hsg); groups with modify permissions: Set()
2018-09-07 20:27:46 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 1623.
2018-09-07 20:27:46 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-09-07 20:27:46 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-09-07 20:27:46 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-09-07 20:27:46 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-09-07 20:27:46 INFO  DiskBlockManager:54 - Created local directory at C:\Users\hsg\AppData\Local\Temp\blockmgr-2184be5b-b56c-4e63-a47f-a6bee53a2cce
2018-09-07 20:27:46 INFO  MemoryStore:54 - MemoryStore started with capacity 1987.5 MB
2018-09-07 20:27:46 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2018-09-07 20:27:46 INFO  log:192 - Logging initialized @1353ms
2018-09-07 20:27:46 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
2018-09-07 20:27:47 INFO  Server:414 - Started @1408ms
2018-09-07 20:27:47 INFO  AbstractConnector:278 - Started ServerConnector@bcef303{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-07 20:27:47 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@531f4093{/jobs,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@403f0a22{/jobs/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@503ecb24{/jobs/job,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6995bf68{/jobs/job/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5143c662{/stages,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77825085{/stages/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3568f9d2{/stages/stage,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5b1ebf56{/stages/stage/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@294a6b8e{/stages/pool,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b1d6571{/stages/pool/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1b835480{/storage,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3549bca9{/storage/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4f25b795{/storage/rdd,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6fb365ed{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e950bcf{/environment,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@16414e40{/environment/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@74bada02{/executors,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@525575{/executors/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@46dffdc3{/executors/threadDump,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5a709816{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@78383390{/static,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31bcf236{/,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b3ed2f0{/api,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a12c404{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1941a8ff{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://hsgpc:4040
2018-09-07 20:27:47 INFO  Executor:54 - Starting executor ID driver on host localhost
2018-09-07 20:27:47 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 1664.
2018-09-07 20:27:47 INFO  NettyBlockTransferService:54 - Server created on hsgpc:1664
2018-09-07 20:27:47 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-09-07 20:27:47 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, hsgpc, 1664, None)
2018-09-07 20:27:47 INFO  BlockManagerMasterEndpoint:54 - Registering block manager hsgpc:1664 with 1987.5 MB RAM, BlockManagerId(driver, hsgpc, 1664, None)
2018-09-07 20:27:47 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, hsgpc, 1664, None)
2018-09-07 20:27:47 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, hsgpc, 1664, None)
2018-09-07 20:27:47 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6722db6e{/metrics/json,null,AVAILABLE,@Spark}
2018-09-07 20:27:47 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 236.7 KB, free 1987.3 MB)
2018-09-07 20:27:47 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 22.9 KB, free 1987.2 MB)
2018-09-07 20:27:47 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on hsgpc:1664 (size: 22.9 KB, free: 1987.5 MB)
2018-09-07 20:27:47 INFO  SparkContext:54 - Created broadcast 0 from textFile at SparkWordCount.scala:22
2018-09-07 20:27:48 INFO  FileInputFormat:249 - Total input paths to process : 1
2018-09-07 20:27:48 INFO  deprecation:1173 - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
2018-09-07 20:27:48 INFO  FileOutputCommitter:108 - File Output Committer Algorithm version is 1
2018-09-07 20:27:48 INFO  SparkContext:54 - Starting job: runJob at SparkHadoopWriter.scala:78
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Registering RDD 3 (map at SparkWordCount.scala:23)
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Got job 0 (runJob at SparkHadoopWriter.scala:78) with 2 output partitions
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Final stage: ResultStage 1 (runJob at SparkHadoopWriter.scala:78)
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Parents of final stage: List(ShuffleMapStage 0)
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Missing parents: List(ShuffleMapStage 0)
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at SparkWordCount.scala:23), which has no missing parents
2018-09-07 20:27:48 INFO  MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 4.8 KB, free 1987.2 MB)
2018-09-07 20:27:48 INFO  MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.8 KB, free 1987.2 MB)
2018-09-07 20:27:48 INFO  BlockManagerInfo:54 - Added broadcast_1_piece0 in memory on hsgpc:1664 (size: 2.8 KB, free: 1987.5 MB)
2018-09-07 20:27:48 INFO  SparkContext:54 - Created broadcast 1 from broadcast at DAGScheduler.scala:1039
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at SparkWordCount.scala:23) (first 15 tasks are for partitions Vector(0, 1))
2018-09-07 20:27:48 INFO  TaskSchedulerImpl:54 - Adding task set 0.0 with 2 tasks
2018-09-07 20:27:48 INFO  TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, ANY, 7880 bytes)
2018-09-07 20:27:48 INFO  TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, ANY, 7880 bytes)
2018-09-07 20:27:48 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2018-09-07 20:27:48 INFO  Executor:54 - Running task 1.0 in stage 0.0 (TID 1)
2018-09-07 20:27:48 INFO  HadoopRDD:54 - Input split: hdfs://192.168.145.180:8020/spark/hellospark:0+18
2018-09-07 20:27:48 INFO  HadoopRDD:54 - Input split: hdfs://192.168.145.180:8020/spark/hellospark:18+18
2018-09-07 20:27:48 INFO  Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 1147 bytes result sent to driver
2018-09-07 20:27:48 INFO  TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 300 ms on localhost (executor driver) (1/2)
2018-09-07 20:27:48 INFO  Executor:54 - Finished task 1.0 in stage 0.0 (TID 1). 1104 bytes result sent to driver
2018-09-07 20:27:48 INFO  TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 301 ms on localhost (executor driver) (2/2)
2018-09-07 20:27:48 INFO  TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool 
2018-09-07 20:27:48 INFO  DAGScheduler:54 - ShuffleMapStage 0 (map at SparkWordCount.scala:23) finished in 0.358 s
2018-09-07 20:27:48 INFO  DAGScheduler:54 - looking for newly runnable stages
2018-09-07 20:27:48 INFO  DAGScheduler:54 - running: Set()
2018-09-07 20:27:48 INFO  DAGScheduler:54 - waiting: Set(ResultStage 1)
2018-09-07 20:27:48 INFO  DAGScheduler:54 - failed: Set()
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Submitting ResultStage 1 (MapPartitionsRDD[5] at saveAsTextFile at SparkWordCount.scala:24), which has no missing parents
2018-09-07 20:27:48 INFO  MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 72.3 KB, free 1987.2 MB)
2018-09-07 20:27:48 INFO  MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 26.1 KB, free 1987.1 MB)
2018-09-07 20:27:48 INFO  BlockManagerInfo:54 - Added broadcast_2_piece0 in memory on hsgpc:1664 (size: 26.1 KB, free: 1987.4 MB)
2018-09-07 20:27:48 INFO  SparkContext:54 - Created broadcast 2 from broadcast at DAGScheduler.scala:1039
2018-09-07 20:27:48 INFO  DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at saveAsTextFile at SparkWordCount.scala:24) (first 15 tasks are for partitions Vector(0, 1))
2018-09-07 20:27:48 INFO  TaskSchedulerImpl:54 - Adding task set 1.0 with 2 tasks
2018-09-07 20:27:48 INFO  TaskSetManager:54 - Starting task 0.0 in stage 1.0 (TID 2, localhost, executor driver, partition 0, ANY, 7649 bytes)
2018-09-07 20:27:48 INFO  TaskSetManager:54 - Starting task 1.0 in stage 1.0 (TID 3, localhost, executor driver, partition 1, ANY, 7649 bytes)
2018-09-07 20:27:48 INFO  Executor:54 - Running task 0.0 in stage 1.0 (TID 2)
2018-09-07 20:27:48 INFO  Executor:54 - Running task 1.0 in stage 1.0 (TID 3)
2018-09-07 20:27:48 INFO  ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks out of 2 blocks
2018-09-07 20:27:48 INFO  ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 2 blocks
2018-09-07 20:27:48 INFO  ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 5 ms
2018-09-07 20:27:48 INFO  ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 5 ms
2018-09-07 20:27:48 INFO  FileOutputCommitter:108 - File Output Committer Algorithm version is 1
2018-09-07 20:27:48 INFO  FileOutputCommitter:108 - File Output Committer Algorithm version is 1
2018-09-07 20:27:49 INFO  FileOutputCommitter:535 - Saved output of task 'attempt_20180907202748_0005_m_000001_0' to hdfs://192.168.145.180:8020/spark/output/_temporary/0/task_20180907202748_0005_m_000001
2018-09-07 20:27:49 INFO  SparkHadoopMapRedUtil:54 - attempt_20180907202748_0005_m_000001_0: Committed
2018-09-07 20:27:49 INFO  Executor:54 - Finished task 1.0 in stage 1.0 (TID 3). 1502 bytes result sent to driver
2018-09-07 20:27:49 INFO  TaskSetManager:54 - Finished task 1.0 in stage 1.0 (TID 3) in 696 ms on localhost (executor driver) (1/2)
2018-09-07 20:27:49 INFO  FileOutputCommitter:535 - Saved output of task 'attempt_20180907202748_0005_m_000000_0' to hdfs://192.168.145.180:8020/spark/output/_temporary/0/task_20180907202748_0005_m_000000
2018-09-07 20:27:49 INFO  SparkHadoopMapRedUtil:54 - attempt_20180907202748_0005_m_000000_0: Committed
2018-09-07 20:27:49 INFO  Executor:54 - Finished task 0.0 in stage 1.0 (TID 2). 1459 bytes result sent to driver
2018-09-07 20:27:49 INFO  TaskSetManager:54 - Finished task 0.0 in stage 1.0 (TID 2) in 716 ms on localhost (executor driver) (2/2)
2018-09-07 20:27:49 INFO  TaskSchedulerImpl:54 - Removed TaskSet 1.0, whose tasks have all completed, from pool 
2018-09-07 20:27:49 INFO  DAGScheduler:54 - ResultStage 1 (runJob at SparkHadoopWriter.scala:78) finished in 0.734 s
2018-09-07 20:27:49 INFO  DAGScheduler:54 - Job 0 finished: runJob at SparkHadoopWriter.scala:78, took 1.260756 s
2018-09-07 20:27:49 INFO  SparkHadoopWriter:54 - Job job_20180907202748_0005 committed.
hdfs://192.168.145.180:8020/spark/hellospark
hdfs://192.168.145.180:8020/spark/output
hello end
2018-09-07 20:27:49 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2018-09-07 20:27:49 INFO  AbstractConnector:318 - Stopped Spark@bcef303{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-07 20:27:49 INFO  SparkUI:54 - Stopped Spark web UI at http://hsgpc:4040
2018-09-07 20:27:49 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-09-07 20:27:49 INFO  MemoryStore:54 - MemoryStore cleared
2018-09-07 20:27:49 INFO  BlockManager:54 - BlockManager stopped
2018-09-07 20:27:49 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2018-09-07 20:27:49 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-09-07 20:27:49 INFO  SparkContext:54 - Successfully stopped SparkContext
2018-09-07 20:27:49 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-09-07 20:27:49 INFO  ShutdownHookManager:54 - Deleting directory C:\Users\hsg\AppData\Local\Temp\spark-97dc8724-e958-4db6-a005-5e365dcdcdba

—the–end—

hsg77
关注
专栏目录
[Spark应用]-- 实现单词统计（数据源在hdfs上）
欢迎来到我的博客，一起探索代码里的世界！
06-04 1866
实现如下 package com.scala.my import org.apache.spark.streaming.StreamingContext import org.apache.spark.SparkConf import org.apache.spark.streaming.Durations /**  * scala版本的hdfs上wordCount  */ object H...
HiveSpark整合原理与代码实例讲解
最新发布
程序员光剑
07-16 819
Hive-Spark整合原理与代码实例讲解 1. 背景介绍 1.1 问题的由来 随着大数据时代的到来，处理海量数据的需求日益增加，Hadoop生态系统中的Hive和Spark因其强大的数据处理能力而被广泛使用。Hive提供了SQL查询
Spark Streaming 简单实例一统计hdfs文件单词
sunbin11220904的博客
08-23 181
使用Spark Streaming统计hdfs文件单词 代码 package test01 import org.apache.spark.SparkConf import org.apache.spark.streaming.dstream.DStream import org.apache.spark.streaming.{Seconds, StreamingContext} import test01.Demo01.ssc object HDFSInputStreamDemo extends Ap
【博学谷学习记录】超强总结，用心分享|Spark的入门案例-从HDFS上读取文件并实现排序
Sweet~~~
07-03 436
输入 i 进入插入模式添加以下内容 : hadoop hive zookeeper hive hadoop上传到HDFS。
用Spark实现简单的单词统计
墨玉浮白的博客
04-18 1794
用Scala实现 RDD（可以简单理解为是一个list集合，里面放的就是读到的一行一行的数据）是spark中非常核心的内容，只有通过SparkContext才能创建出来RDD。 package com.husky.spark import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} object SparkWordCount { def main(args: Array[String]): Un
提交第一个Spark统计文件单词数程序，配合hadoop hdfs
weixin_34146410的博客
11-03 465
2019独角兽企业重金招聘Python工程师标准>>> ...
eclipse集成hadoop+spark+hive开发源码实例
05-02
在Windows操作系统中，Eclipse是一款广泛使用的Java集成开发环境（IDE），它对于开发分布式计算框架如Hadoop、Spark以及数据仓库工具Hive来说，是非常有用的。本教程将深入讲解如何在Eclipse中集成这些组件，以创建...
Spark SQL原理与代码实例讲解
程序员光剑
07-15 1795
随着大数据时代的到来，海量数据的处理和分析成为了许多企业和组织面临的巨大挑战。传统的数据库管理系统难以满足大规模数据处理的需求，而 Hadoop 等分布式计算框架的出现为解决这一问题提供了新的思路。Spark 作为新一代的分布式计算框架，凭借其高效的内存计算和易用的编程接口，迅速成为了大数据处理领域的热门技术。而 Spark SQL 作为 Spark 生态系统中用于结构化数据处理的核心组件，为用户提供了类似 SQL 的查询语言和 DataFrame API，使得用户能够更加方便地进行数据分析和处理。
SparkStreaming与Storm集成实例
程序员光剑
06-08 1059
SparkStreaming与Storm集成实例 1.背景介绍 在大数据处理领域，实时数据处理变得越来越重要。Spark Streaming和Apache Storm是两种广泛使用的实时数据处理框架。Spark Streaming是基于Apache Spark的扩展，提供了高
Hadoop--eclipse写MapReduce代码在Hadoop上执行单词统计
weixin_33805743的博客
07-25 266
一、需要的jar包 Hadoop-2.4.1\share\hadoop\hdfs\hadoop-hdfs-2.4.1.jar hadoop-2.4.1\share\hadoop\hdfs\lib\所有jar包 hadoop-2.4.1\share\hadoop\common\hadoop-common-2.4.1.jar hadoop-2.4.1\share\hadoop\common\l...
Spark最全操作完整示例代码
08-03
最全Spark操作完整示例代码-------是基于java的。 包含所有的spark常用算子操作和ml以及mlib、sparkstreaming、sparkSQL操作的示例DEMO。 内附有详细说明,由于内容过大删除了两个jar包,需要自己去下载,spark的安装包下的有这两个包，导入项目即可。懒得改maven项目了。 适合有一定基础和已经工作的人员下载学习。 个人Github: https://github.com/huangyueranbbc
Spark打包好的代码实现WordCount词频统计
weixin_53989498的博客
09-16 1236
Spark打包好的代码实现WordCount词频统计！
Java实现Spark词配对Wordcount计数
煊琦的CSDN小窝
05-24 844
需求 实现过程 开启hadoop和spark jar包处理 HDFS文件设置 spark-submit程序 查询结果 附加代码需求  使用Spark实现对一个文档中的每一行的单词进行词配对计数，要求去标点符号，将大写符号统一转化成为小写单词。
Spark基础学习笔记：SparkSQL案例
py20010218的博客
06-18 290
Spark基础学习笔记：SparkSQL案例
spark 统计单词_三十二、SparkSQL实现单词统计wordcount
weixin_36439285的博客
02-01 394
SparkSQL实现wordcount1.数据集假如有以下数据集: actors.txt风清扬 令狐冲任我行 东方不败 灭绝师太乔峰 段誉 虚竹 王语嫣 木婉清 阿紫阿朱 马大元 马夫人 慕容复 鸠摩智风清扬 令狐冲任盈盈 岳不群岳不群 岳灵珊任盈盈 岳不群岳灵珊 慕容复乔峰 段誉2.用SparkSQL实现单词统计package com.dvtn.spark.sql.wordcountimpor...
Spark学习笔记：基于HDFS的实时计算WordCount
lrxcmwy2的博客
09-16 356
基于HDFS实时计算WordCount
Spark SQL案例分析：词频统计
m0_63797754的博客
06-19 1041
Spark SQL 词频案例分析
spark 集群单词统计_Spark教程
weixin_31904867的博客
01-28 574
转载请私信。禁止无授权转载1.spark安装利用ambari安装整个集群 参考Ambari及其HDP集群安装及其配置教程2.Spark工作原理spark工作流程提交任务到spark集群后,spark会从数据存储系统如HDFS读取数据,生成RDD,RDD进行数据分区交由各个节点进行处理,每个节点处理完数据后的结果存储在内存中，同时可将该结果交由下一个节点继续处理.最后将最终结果写回HDFS/Mysq...
使用Spark编写一个简单的word count单词统计及Spark基本架构及运行原理
Mr_wang0916的博客
05-30 979
1、Spark基本架构及原理 Spark:spark只是一个计算框架，它的能力是在现有数据的基础上提供一个高性能的计算引擎，然后提供一些上层的处理工具比如做数据查询的Spark SQL、做机器学习的MLlib等；而hadoop的功能则更加全面，它是包括了数据存储（HDFS）、任务计划和集群资源管理（YARN）以及离线并行计算（MapReduce）的一整套技术栈。 Spark架构示意图 Spark...
HDFS与Spark统计分析实践：使用Vaadin和Highcharts
"Spark统计图表在已有Hadoop集群环境下的操作，涉及HDFS的权限控制、JDBC驱动的处理以及HDFS API的使用，利用Vaadin框架和Highcharts插件展示图表。" 在已有的Hadoop集群环境中进行Spark统计图表的操作，首先需要...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值