spark-shell 启动 以及 例子

最新推荐文章于 2024-06-01 16:17:06 发布
最新推荐文章于 2024-06-01 16:17:06 发布
阅读量7.3k 收藏 1
点赞数 1
分类专栏： spark 文章标签： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/hzdxw/article/details/51703461
版权
[root@cdh1 hadoop]# spark-shell
bash: spark-shell: command not found
[root@cdh1 hadoop]# source /etc/profile
[root@cdh1 hadoop]# spark-shell
16/06/17 10:07:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/06/17 10:07:59 INFO spark.SecurityManager: Changing view acls to: root
16/06/17 10:07:59 INFO spark.SecurityManager: Changing modify acls to: root
16/06/17 10:07:59 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/06/17 10:07:59 INFO spark.HttpServer: Starting HTTP Server
16/06/17 10:07:59 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/06/17 10:07:59 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48030
16/06/17 10:07:59 INFO util.Utils: Successfully started service 'HTTP class server' on port 48030.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0
      /_/


Using Scala version 2.10.4 (Java HotSpot(TM) Client VM, Java 1.7.0_67)
Type in expressions to have them evaluated.
Type :help for more information.
16/06/17 10:08:08 WARN util.Utils: Your hostname, cdh1 resolves to a loopback address: 127.0.0.1; using 192.168.0.103 instead (on interface eth3)
16/06/17 10:08:08 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
16/06/17 10:08:08 INFO spark.SparkContext: Running Spark version 1.4.0
16/06/17 10:08:09 INFO spark.SecurityManager: Changing view acls to: root
16/06/17 10:08:09 INFO spark.SecurityManager: Changing modify acls to: root
16/06/17 10:08:09 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/06/17 10:08:10 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/06/17 10:08:10 INFO Remoting: Starting remoting
16/06/17 10:08:11 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.0.103:46302]
16/06/17 10:08:11 INFO util.Utils: Successfully started service 'sparkDriver' on port 46302.
16/06/17 10:08:11 INFO spark.SparkEnv: Registering MapOutputTracker
16/06/17 10:08:11 INFO spark.SparkEnv: Registering BlockManagerMaster
16/06/17 10:08:11 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-4e24ee66-6db3-41f1-9fe5-12da91e71fc3/blockmgr-36b2f8b4-bb92-4759-9a4e-90ae9c128577
16/06/17 10:08:11 INFO storage.MemoryStore: MemoryStore started with capacity 267.3 MB
16/06/17 10:08:11 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-4e24ee66-6db3-41f1-9fe5-12da91e71fc3/httpd-aeb3a7a4-3947-4f11-9504-8ff175a289b2
16/06/17 10:08:11 INFO spark.HttpServer: Starting HTTP Server
16/06/17 10:08:11 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/06/17 10:08:11 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:40375
16/06/17 10:08:11 INFO util.Utils: Successfully started service 'HTTP file server' on port 40375.
16/06/17 10:08:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/06/17 10:08:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/06/17 10:08:13 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/06/17 10:08:13 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/06/17 10:08:13 INFO ui.SparkUI: Started SparkUI at http://192.168.0.103:4040
16/06/17 10:08:13 INFO executor.Executor: Starting executor ID driver on host localhost
16/06/17 10:08:13 INFO executor.Executor: Using REPL class URI: http://192.168.0.103:48030
16/06/17 10:08:13 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42970.
16/06/17 10:08:13 INFO netty.NettyBlockTransferService: Server created on 42970
16/06/17 10:08:13 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/06/17 10:08:13 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:42970 with 267.3 MB RAM, BlockManagerId(driver, localhost, 42970)
16/06/17 10:08:13 INFO storage.BlockManagerMaster: Registered BlockManager
16/06/17 10:08:14 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/06/17 10:08:15 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
16/06/17 10:08:17 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/06/17 10:08:17 INFO metastore.ObjectStore: ObjectStore, initialize called
16/06/17 10:08:18 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/06/17 10:08:18 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/06/17 10:08:18 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/06/17 10:08:19 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/06/17 10:08:21 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/06/17 10:08:21 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".
16/06/17 10:08:23 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/06/17 10:08:23 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/06/17 10:08:25 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/06/17 10:08:25 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/06/17 10:08:26 INFO metastore.ObjectStore: Initialized ObjectStore
16/06/17 10:08:26 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
16/06/17 10:08:28 INFO metastore.HiveMetaStore: Added admin role in metastore
16/06/17 10:08:28 INFO metastore.HiveMetaStore: Added public role in metastore
16/06/17 10:08:28 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
16/06/17 10:08:29 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
16/06/17 10:08:29 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.


scala> val textFile = sc.textFile("file:///spark.test.txt");
16/06/17 10:48:39 WARN util.SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
16/06/17 10:48:39 INFO storage.MemoryStore: ensureFreeSpace(85352) called with curMem=0, maxMem=280248975
16/06/17 10:48:39 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 83.4 KB, free 267.2 MB)
16/06/17 10:48:40 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=85352, maxMem=280248975
16/06/17 10:48:40 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.5 KB, free 267.2 MB)
16/06/17 10:48:40 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:42970 (size: 19.5 KB, free: 267.2 MB)
16/06/17 10:48:40 INFO spark.SparkContext: Created broadcast 0 from textFile at <console>:21
textFile: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:21


scala> textFile.count();
16/06/17 10:49:24 INFO mapred.FileInputFormat: Total input paths to process : 1
16/06/17 10:49:24 INFO spark.SparkContext: Starting job: count at <console>:24
16/06/17 10:49:24 INFO scheduler.DAGScheduler: Got job 0 (count at <console>:24) with 1 output partitions (allowLocal=false)
16/06/17 10:49:24 INFO scheduler.DAGScheduler: Final stage: ResultStage 0(count at <console>:24)
16/06/17 10:49:24 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/06/17 10:49:24 INFO scheduler.DAGScheduler: Missing parents: List()
16/06/17 10:49:24 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at textFile at <console>:21), which has no missing parents
16/06/17 10:49:24 INFO storage.MemoryStore: ensureFreeSpace(2976) called with curMem=105351, maxMem=280248975
16/06/17 10:49:24 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 267.2 MB)
16/06/17 10:49:24 INFO storage.MemoryStore: ensureFreeSpace(1755) called with curMem=108327, maxMem=280248975
16/06/17 10:49:24 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1755.0 B, free 267.2 MB)
16/06/17 10:49:24 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:42970 (size: 1755.0 B, free: 267.2 MB)
16/06/17 10:49:24 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874
16/06/17 10:49:24 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at textFile at <console>:21)
16/06/17 10:49:24 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/06/17 10:49:24 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1393 bytes)
16/06/17 10:49:24 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
16/06/17 10:49:25 INFO rdd.HadoopRDD: Input split: file:/spark.test.txt:0+331
16/06/17 10:49:25 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
16/06/17 10:49:25 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
16/06/17 10:49:25 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
16/06/17 10:49:25 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
16/06/17 10:49:25 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
16/06/17 10:49:25 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1830 bytes result sent to driver
16/06/17 10:49:25 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 927 ms on localhost (1/1)
16/06/17 10:49:25 INFO scheduler.DAGScheduler: ResultStage 0 (count at <console>:24) finished in 0.963 s
16/06/17 10:49:25 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/06/17 10:49:25 INFO scheduler.DAGScheduler: Job 0 finished: count at <console>:24, took 1.495097 s
res0: Long = 7


scala> val linesWithSpark = textFile.filter(line => line.contains("spark"));
linesWithSpark: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at filter at <console>:23


scala> textFile.filter(line => line.contains("spark")).count();
16/06/17 10:51:17 INFO spark.SparkContext: Starting job: count at <console>:24
16/06/17 10:51:17 INFO scheduler.DAGScheduler: Got job 1 (count at <console>:24) with 1 output partitions (allowLocal=false)
16/06/17 10:51:17 INFO scheduler.DAGScheduler: Final stage: ResultStage 1(count at <console>:24)
16/06/17 10:51:17 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/06/17 10:51:17 INFO scheduler.DAGScheduler: Missing parents: List()
16/06/17 10:51:17 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[3] at filter at <console>:24), which has no missing parents
16/06/17 10:51:17 INFO storage.MemoryStore: ensureFreeSpace(3200) called with curMem=110082, maxMem=280248975
16/06/17 10:51:17 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.1 KB, free 267.2 MB)
16/06/17 10:51:17 INFO storage.MemoryStore: ensureFreeSpace(1866) called with curMem=113282, maxMem=280248975
16/06/17 10:51:17 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1866.0 B, free 267.2 MB)
16/06/17 10:51:17 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:42970 (size: 1866.0 B, free: 267.2 MB)
16/06/17 10:51:17 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874
16/06/17 10:51:17 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[3] at filter at <console>:24)
16/06/17 10:51:17 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/06/17 10:51:17 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, PROCESS_LOCAL, 1393 bytes)
16/06/17 10:51:17 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 1)
16/06/17 10:51:17 INFO rdd.HadoopRDD: Input split: file:/spark.test.txt:0+331
16/06/17 10:51:17 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 1). 1830 bytes result sent to driver
16/06/17 10:51:17 INFO scheduler.DAGScheduler: ResultStage 1 (count at <console>:24) finished in 0.023 s
16/06/17 10:51:17 INFO scheduler.DAGScheduler: Job 1 finished: count at <console>:24, took 0.042199 s
res1: Long = 5


scala> 16/06/17 10:51:17 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 21 ms on localhost (1/1)
16/06/17 10:51:17 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 




scala> textFile.first();
16/06/17 10:51:50 INFO spark.SparkContext: Starting job: first at <console>:24
16/06/17 10:51:50 INFO scheduler.DAGScheduler: Got job 2 (first at <console>:24) with 1 output partitions (allowLocal=true)
16/06/17 10:51:50 INFO scheduler.DAGScheduler: Final stage: ResultStage 2(first at <console>:24)
16/06/17 10:51:50 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/06/17 10:51:50 INFO scheduler.DAGScheduler: Missing parents: List()
16/06/17 10:51:50 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[1] at textFile at <console>:21), which has no missing parents
16/06/17 10:51:50 INFO storage.MemoryStore: ensureFreeSpace(3136) called with curMem=115148, maxMem=280248975
16/06/17 10:51:50 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.1 KB, free 267.2 MB)
16/06/17 10:51:50 INFO storage.MemoryStore: ensureFreeSpace(1813) called with curMem=118284, maxMem=280248975
16/06/17 10:51:50 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1813.0 B, free 267.2 MB)
16/06/17 10:51:50 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:42970 (size: 1813.0 B, free: 267.2 MB)
16/06/17 10:51:50 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:874
16/06/17 10:51:50 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[1] at textFile at <console>:21)
16/06/17 10:51:50 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
16/06/17 10:51:50 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, PROCESS_LOCAL, 1393 bytes)
16/06/17 10:51:50 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2)
16/06/17 10:51:50 INFO rdd.HadoopRDD: Input split: file:/spark.test.txt:0+331
16/06/17 10:51:50 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 2). 1850 bytes result sent to driver
16/06/17 10:51:50 INFO scheduler.DAGScheduler: ResultStage 2 (first at <console>:24) finished in 0.014 s
16/06/17 10:51:50 INFO scheduler.DAGScheduler: Job 2 finished: first at <console>:24, took 0.040965 s
res2: String = hello java hello hadoop hello hive hehe spar1 hh jjj kk


scala> 16/06/17 10:51:50 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 19 ms on localhost (1/1)
16/06/17 10:51:50 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 




scala> textFile.count();
16/06/17 10:52:13 INFO spark.SparkContext: Starting job: count at <console>:24
16/06/17 10:52:13 INFO scheduler.DAGScheduler: Got job 3 (count at <console>:24) with 1 output partitions (allowLocal=false)
16/06/17 10:52:13 INFO scheduler.DAGScheduler: Final stage: ResultStage 3(count at <console>:24)
16/06/17 10:52:13 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/06/17 10:52:13 INFO scheduler.DAGScheduler: Missing parents: List()
16/06/17 10:52:13 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[1] at textFile at <console>:21), which has no missing parents
16/06/17 10:52:13 INFO storage.MemoryStore: ensureFreeSpace(2976) called with curMem=120097, maxMem=280248975
16/06/17 10:52:13 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 2.9 KB, free 267.1 MB)
16/06/17 10:52:13 INFO storage.MemoryStore: ensureFreeSpace(1755) called with curMem=123073, maxMem=280248975
16/06/17 10:52:13 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1755.0 B, free 267.1 MB)
16/06/17 10:52:13 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:42970 (size: 1755.0 B, free: 267.2 MB)
16/06/17 10:52:13 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:874
16/06/17 10:52:13 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[1] at textFile at <console>:21)
16/06/17 10:52:13 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
16/06/17 10:52:13 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, PROCESS_LOCAL, 1393 bytes)
16/06/17 10:52:13 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 3)
16/06/17 10:52:13 INFO rdd.HadoopRDD: Input split: file:/spark.test.txt:0+331
16/06/17 10:52:13 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 3). 1830 bytes result sent to driver
16/06/17 10:52:13 INFO scheduler.DAGScheduler: ResultStage 3 (count at <console>:24) finished in 0.004 s
16/06/17 10:52:13 INFO scheduler.DAGScheduler: Job 3 finished: count at <console>:24, took 0.019111 s
res3: Long = 7


scala> 16/06/17 10:52:13 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 9 ms on localhost (1/1)
16/06/17 10:52:13 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 




scala> var size = textFile.map(line=>line.split(" ").size)
size: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[4] at map at <console>:23


scala> size.collect();
16/06/17 10:53:30 INFO spark.SparkContext: Starting job: collect at <console>:26
16/06/17 10:53:30 INFO scheduler.DAGScheduler: Got job 4 (collect at <console>:26) with 1 output partitions (allowLocal=false)
16/06/17 10:53:30 INFO scheduler.DAGScheduler: Final stage: ResultStage 4(collect at <console>:26)
16/06/17 10:53:30 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/06/17 10:53:30 INFO scheduler.DAGScheduler: Missing parents: List()
16/06/17 10:53:30 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[4] at map at <console>:23), which has no missing parents
16/06/17 10:53:30 INFO storage.MemoryStore: ensureFreeSpace(3392) called with curMem=124828, maxMem=280248975
16/06/17 10:53:30 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.3 KB, free 267.1 MB)
16/06/17 10:53:30 INFO storage.MemoryStore: ensureFreeSpace(1967) called with curMem=128220, maxMem=280248975
16/06/17 10:53:30 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1967.0 B, free 267.1 MB)
16/06/17 10:53:30 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:42970 (size: 1967.0 B, free: 267.2 MB)
16/06/17 10:53:30 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:874
16/06/17 10:53:30 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[4] at map at <console>:23)
16/06/17 10:53:30 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
16/06/17 10:53:30 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, PROCESS_LOCAL, 1393 bytes)
16/06/17 10:53:30 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 4)
16/06/17 10:53:30 INFO rdd.HadoopRDD: Input split: file:/spark.test.txt:0+331
16/06/17 10:53:30 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 4). 1803 bytes result sent to driver
16/06/17 10:53:30 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 10 ms on localhost (1/1)
16/06/17 10:53:30 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
16/06/17 10:53:30 INFO scheduler.DAGScheduler: ResultStage 4 (collect at <console>:26) finished in 0.009 s
16/06/17 10:53:30 INFO scheduler.DAGScheduler: Job 4 finished: collect at <console>:26, took 0.019113 s
res4: Array[Int] = Array(11, 8, 8, 8, 8, 8, 8)


scala> size.reduce((a, b)=>if (a > b) a else b)
16/06/17 10:53:53 INFO spark.SparkContext: Starting job: reduce at <console>:26
16/06/17 10:53:53 INFO scheduler.DAGScheduler: Got job 5 (reduce at <console>:26) with 1 output partitions (allowLocal=false)
16/06/17 10:53:53 INFO scheduler.DAGScheduler: Final stage: ResultStage 5(reduce at <console>:26)
16/06/17 10:53:53 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/06/17 10:53:53 INFO scheduler.DAGScheduler: Missing parents: List()
16/06/17 10:53:53 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[4] at map at <console>:23), which has no missing parents
16/06/17 10:53:53 INFO storage.MemoryStore: ensureFreeSpace(3368) called with curMem=130187, maxMem=280248975
16/06/17 10:53:53 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.3 KB, free 267.1 MB)
16/06/17 10:53:53 INFO storage.MemoryStore: ensureFreeSpace(1944) called with curMem=133555, maxMem=280248975
16/06/17 10:53:53 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1944.0 B, free 267.1 MB)
16/06/17 10:53:53 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:42970 (size: 1944.0 B, free: 267.2 MB)
16/06/17 10:53:53 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:874
16/06/17 10:53:53 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[4] at map at <console>:23)
16/06/17 10:53:53 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
16/06/17 10:53:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, PROCESS_LOCAL, 1393 bytes)
16/06/17 10:53:53 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 5)
16/06/17 10:53:53 INFO rdd.HadoopRDD: Input split: file:/spark.test.txt:0+331
16/06/17 10:53:53 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 5). 1908 bytes result sent to driver
16/06/17 10:53:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 10 ms on localhost (1/1)
16/06/17 10:53:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
16/06/17 10:53:53 INFO scheduler.DAGScheduler: ResultStage 5 (reduce at <console>:26) finished in 0.005 s
16/06/17 10:53:53 INFO scheduler.DAGScheduler: Job 5 finished: reduce at <console>:26, took 0.017716 s
res5: Int = 11


scala> val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b);
wordCounts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[7] at reduceByKey at <console>:23


scala> 16/06/17 10:54:28 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:42970 in memory (size: 1866.0 B, free: 267.2 MB)
16/06/17 10:54:28 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on localhost:42970 in memory (size: 1813.0 B, free: 267.2 MB)
16/06/17 10:54:28 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on localhost:42970 in memory (size: 1755.0 B, free: 267.2 MB)
16/06/17 10:54:28 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on localhost:42970 in memory (size: 1967.0 B, free: 267.2 MB)
16/06/17 10:54:28 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on localhost:42970 in memory (size: 1944.0 B, free: 267.2 MB)




scala> wordCounts.collect();
16/06/17 10:55:12 INFO spark.SparkContext: Starting job: collect at <console>:26
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Registering RDD 6 (map at <console>:23)
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Got job 6 (collect at <console>:26) with 1 output partitions (allowLocal=false)
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Final stage: ResultStage 7(collect at <console>:26)
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 6)
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 6)
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 6 (MapPartitionsRDD[6] at map at <console>:23), which has no missing parents
16/06/17 10:55:12 INFO storage.MemoryStore: ensureFreeSpace(4104) called with curMem=110082, maxMem=280248975
16/06/17 10:55:12 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.0 KB, free 267.2 MB)
16/06/17 10:55:12 INFO storage.MemoryStore: ensureFreeSpace(2297) called with curMem=114186, maxMem=280248975
16/06/17 10:55:12 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.2 KB, free 267.2 MB)
16/06/17 10:55:12 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:42970 (size: 2.2 KB, free: 267.2 MB)
16/06/17 10:55:12 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:874
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[6] at map at <console>:23)
16/06/17 10:55:12 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
16/06/17 10:55:12 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, localhost, PROCESS_LOCAL, 1382 bytes)
16/06/17 10:55:12 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 6)
16/06/17 10:55:12 INFO rdd.HadoopRDD: Input split: file:/spark.test.txt:0+331
16/06/17 10:55:12 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 6). 2001 bytes result sent to driver
16/06/17 10:55:12 INFO scheduler.DAGScheduler: ShuffleMapStage 6 (map at <console>:23) finished in 0.097 s
16/06/17 10:55:12 INFO scheduler.DAGScheduler: looking for newly runnable stages
16/06/17 10:55:12 INFO scheduler.DAGScheduler: running: Set()
16/06/17 10:55:12 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 7)
16/06/17 10:55:12 INFO scheduler.DAGScheduler: failed: Set()
16/06/17 10:55:12 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 100 ms on localhost (1/1)
16/06/17 10:55:12 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Missing parents for ResultStage 7: List()
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Submitting ResultStage 7 (ShuffledRDD[7] at reduceByKey at <console>:23), which is now runnable
16/06/17 10:55:12 INFO storage.MemoryStore: ensureFreeSpace(2288) called with curMem=116483, maxMem=280248975
16/06/17 10:55:12 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 2.2 KB, free 267.2 MB)
16/06/17 10:55:12 INFO storage.MemoryStore: ensureFreeSpace(1377) called with curMem=118771, maxMem=280248975
16/06/17 10:55:12 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 1377.0 B, free 267.2 MB)
16/06/17 10:55:12 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:42970 (size: 1377.0 B, free: 267.2 MB)
16/06/17 10:55:12 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:874
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (ShuffledRDD[7] at reduceByKey at <console>:23)
16/06/17 10:55:12 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 1 tasks
16/06/17 10:55:12 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, localhost, PROCESS_LOCAL, 1165 bytes)
16/06/17 10:55:12 INFO executor.Executor: Running task 0.0 in stage 7.0 (TID 7)
16/06/17 10:55:12 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/06/17 10:55:12 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms
16/06/17 10:55:12 INFO executor.Executor: Finished task 0.0 in stage 7.0 (TID 7). 1248 bytes result sent to driver
16/06/17 10:55:12 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 94 ms on localhost (1/1)
16/06/17 10:55:12 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 
16/06/17 10:55:12 INFO scheduler.DAGScheduler: ResultStage 7 (collect at <console>:26) finished in 0.087 s
16/06/17 10:55:12 INFO scheduler.DAGScheduler: Job 6 finished: collect at <console>:26, took 0.541458 s
res6: Array[(String, Int)] = Array((spark,4), (hive,7), (hadoop,7), (hehe,7), (hh,1), (spar1,1), (kk,1), (hello,21), (java,7), (spark2,1), (jjj,1), (spk,1))


scala> :quit
Stopping spark context.
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
16/06/17 10:56:32 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
16/06/17 10:56:32 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.0.103:4040
16/06/17 10:56:32 INFO scheduler.DAGScheduler: Stopping DAGScheduler
16/06/17 10:56:32 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/06/17 10:56:32 INFO util.Utils: path = /tmp/spark-4e24ee66-6db3-41f1-9fe5-12da91e71fc3/blockmgr-36b2f8b4-bb92-4759-9a4e-90ae9c128577, already present as root for deletion.
16/06/17 10:56:32 INFO storage.MemoryStore: MemoryStore cleared
16/06/17 10:56:32 INFO storage.BlockManager: BlockManager stopped
16/06/17 10:56:32 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
16/06/17 10:56:32 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/06/17 10:56:32 INFO spark.SparkContext: Successfully stopped SparkContext
16/06/17 10:56:32 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/06/17 10:56:32 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/06/17 10:56:32 INFO util.Utils: Shutdown hook called
16/06/17 10:56:32 INFO util.Utils: Deleting directory /tmp/spark-3e11f3a4-cff9-45f3-8bb8-c98195c49bad
16/06/17 10:56:33 INFO util.Utils: Deleting directory /tmp/spark-b630d853-78dd-4f24-bdc2-683717e162d6
16/06/17 10:56:33 INFO util.Utils: Deleting directory /tmp/spark-4e24ee66-6db3-41f1-9fe5-12da91e71fc3
16/06/17 10:56:33 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
[root@cdh1 hadoop]# cat /spark.test.txt
hello java hello hadoop hello hive hehe spar1 hh jjj kk
hello java hello hadoop hello hive hehe spark2
hello java hello hadoop hello hive hehe spk
hello java hello hadoop hello hive hehe spark
hello java hello hadoop hello hive hehe spark
hello java hello hadoop hello hive hehe spark
hello java hello hadoop hello hive hehe spark
[root@cdh1 hadoop]# 
5icode.top
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
spark-3.2.0-bin-hadoop3-without-hive
04-24
你可以通过解压这个压缩包，配置相应的环境变量，然后启动 Spark Shell 或者 Spark Submit 来运行你的应用程序。需要注意的是，对于没有 Hive 支持的 Spark，你可能需要使用其他方式来实现数据仓库的功能，比如使用 ...
spark-sql: command not found 问题解决
aruewds的博客
08-07 2054
发现SPARK_HOME和PATH已经配置了，但是仍然出现该问题。
启动spark-shell后出现not found错误
PanPan的专栏
10-09 1万+
我曾遇到了类似于这样的问题： error:not found:value sqlContext error:not found:value sc 解决方案： 1.查看ifconfig，看ip地址是否和你在/etc/hosts中的ip地址一致，如果不一致，请修改/etc/hosts中的ip地址。 2.查看spark-env.sh中配置的SPARK_MASTER_IP是否和ifconfig中
【scau大数据技术与原理2】综合性实验Spark集群的安装和使用——安装启动spark shell篇
最新发布
weixin_53762564的博客
06-01 919
Spark是一个分布式计算框架，常用于大数据处理。本次实验中，首先设计一个包含主节点和从节点的Spark集群架构，并在CentOS的Linux环境下进行搭建。通过下载并解压Spark安装包，配置环境变量和集群参数，部署Spark集群。接着言编写Spark应用程序，并将其打包通过spark-submit命令将应用程序提交到Spark集群中运行，观察任务分配和执行进度，验证集群的性能和正确性。
如何解决“hadoop：未找到命令”？
weixin_72700161的博客
02-04 1022
原因是未将hadoop命令添加至执行路径。
bash脚本中的“command not found”
平凡之路
02-05 3531
直接上图片吧，下图是文件“QC1.sh”中的一部分内容（$bit_10是）： 运行后报错
安装spark，命令行输入spark-shell，返回找不到指定的路径，解决
Remmine的博客
05-09 5656
第一次装pyspark，在装完jdk，pyspark，hadoop和winutils之后在cmd中输入spark-shell来测试spark安装，返回了找不到指定的路径。 试了些可能的错误最后找到了问题。 这是spark-shell文件： 用到的是环境变量里的SPARK_HOME，所以即便path里面用的全路径也要创建SPARK_HOME变量。 然后JAVA_HOME和SPARK_HOME都要创建在用户环境变量，一开始我创建在了系统环境变量，这是我遇到找不到路径的原因。 别的可能的原因： 1
spark安装和编程实践简要概括
11-01
- 启动Spark Shell：通过`bin/spark-shell`命令启动交互式Spark环境。Shell会自动创建`sc`（SparkContext）和`spark`（SparkSession）对象。 - 加载数据：在Shell中，你可以使用`sc.textFile()`方法加载本地或HDFS...
spark1.2.1常用模式部署运行
03-02
在 Spark 1.2.1 中，可以使用以下命令启动 Spark Shell： ``` ./bin/spark-shell --master local[2] ``` 这将启动 Spark Shell 并连接到本地模式。然后，可以使用 Scala 语言编写 Spark 应用程序，并执行各种数据...
Spark开发环境搭建
06-15
val conf = new SparkConf().setAppName("Spark shell").setMaster("local[2]") val sc = new SparkContext(conf) val file = spark.sparkContext.textFile("file:///usr/app/wc.txt") val wordCounts = file....
Scala基础(12)Spark命令参数及独立应用程序
01-20
例如，如果你有4个CPU核心，你可以运行`./bin/spark-shell --master local[4]`来启动Spark Shell，并使用这4个核心模拟Spark集群。 另一个重要参数是`--jars`，它允许你指定需要添加到类路径（CLASSPATH）中的JAR包...
linux 命令行报bash command not found的解决办法
北国有鱼的博客
05-04 5万+
命令行报bash:.....:command not found的解决办法（几乎所有命令） 命令行输入命令执行后报“bash:....:command not found”这是由于系统PATH设置问题，PATH没有设置对，系统就无法找到精确命令了。 1、在命令行中输入：export PATH=/usr/bin:/usr/sbin:/bin:/sbin 这样可以保证命令行命令暂时可以使用...
安装spark时输入spark-shell报错 “系统找不到指定的路径“
xfysq_的博客
12-16 3228
安装spark时输入spark-shell报错 “系统找不到指定的路径“
提示-bash: command not found的解决方法集锦
热门推荐
想你依然心痛的博客
06-10 7万+
3、export PATH=“$PATH:/phpstudy/mysql/bin” #在该文件中加入 command not found 的应用 bin 目录,这里我是将 mysql 的 bin 放到这里，因为我是 mysqld / mysqldump / mysqladmin 命令用不了。第三个这和Windows是相同的道理，都是环境变量惹的祸，就是说你的 命令的 执行文件不在 /usr/bin 或者 /etc/init.d 中，他就找不到 执行该命令的 文件来执行。1.你将命令的名称拼错了。
第一次安装Hive遇到的一些小问题
BadMoodvip的博客
11-26 435
在安装hive时，安装完MySQL、启动Hadoop集群后，使用bin/hive启动Hive时报错 2020-11-26 16:53:03,705 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(589)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 2020-11-26 16
Spark 总结项目一---\r: 未找到命令警告 & 脚本jar执行报错Error:Unable to access jarfile
修行的博客
10-16 963
八、 linux脚本执行：\r: 未找到命令警告 & Error:Unable to access jarfile 第一个\r: 未找到命令警告，是因为你所执行的脚本是在win环境下编辑后放在集群上执行的sh脚本，此时需要安装dos2unix执行 原因分析：在windows下编写的shell.sh脚本，上传到服务器，存在多余的空格或换行，导致报错 解决办法，安装dos2unix 安装dos2unix yum install dos2unix 使用dos2unix对脚本转换 dos2unix x
spark学习5：spark-shell
hzp666的博客
05-19 7530
1.spark提供了交互式接口 spark-shell spark-shell 实现了用户可以逐行输入代码，进行操作的功能。 即可以不像Java一样，编写完所有代码，然后编译才能运行 spark-shell 支持 Scala交互环境 和 python交互环境 在学习测试中可以使用spark-shell 进行API学习 2.进入Scala交互环境 在spark安装目录的根目录下，有个bin目录中有个 spark_shell 2.1 在本机生成spark指挥所，创建sparkCo...
Spark Shell 的使用
12-16 3万+
现在我们已经执行了行动操作，执行这些操作需要从磁盘读取数据，Spark在处理这些操作的时候，会把数据缓存起来，后续不管对该RDD执行转换操作还是行动操作，都将直接从内存读取，而不需要和磁盘进行交互。如果数据已经存在外部文件系统，例如本地文件系统，HDFS，HBase，Cassandra，S3 等，可以使用这种方式，即调用 SparkContext 的 textFile 方法，并把文件目录或者路径作为参数。操作，RDD并不会立即缓存，直到执行行动（Action）操作，数据才会真正缓存在内存中。
spark-shell示例
jarth的专栏
11-09 1003
Scala 行数统计 命令及结果示例 # spark-shell scala> val lines  = sc.textFile("README.md") lines: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1] at textFile at :24 scala> lines.count
spark启动spark-shell报错
04-29
spark-shell --driver-memory 4g --executor-memory 4g ``` 2. 错误：`java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveContext` 解决方法：这个错误通常是由于缺少Hive相关的依赖库引起的。...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值