Spark 1.2 Standalone Mode 单机安装

最新推荐文章于 2024-09-14 13:42:55 发布
最新推荐文章于 2024-09-14 13:42:55 发布
阅读量8.5k 收藏 2
点赞数
分类专栏： Spark 文章标签： spark scala
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wind520/article/details/43458925
版权

1：spark下载，解压

[jifeng@jifeng01 hadoop]$ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.2.0-bin-hadoop1.tgz
--2015-02-03 21:50:25--  http://d3kbcqa49mib13.cloudfront.net/spark-1.2.0-bin-hadoop1.tgz
正在解析主机 d3kbcqa49mib13.cloudfront.net... 54.192.150.88, 54.230.149.156, 54.230.150.31, ...
Connecting to d3kbcqa49mib13.cloudfront.net|54.192.150.88|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：203977086 (195M) [application/x-compressed]
Saving to: `spark-1.2.0-bin-hadoop1.tgz'

100%[===========================================================================>] 203,977,086  410K/s   in 7m 55s  

2015-02-03 21:58:21 (419 KB/s) - `spark-1.2.0-bin-hadoop1.tgz' saved [203977086/203977086]


[jifeng@jifeng01 hadoop]$ tar zxf spark-1.2.0-bin-hadoop1.tgz


2：scala下载

在jifeng04下载并复制到jifeng01

[jifeng@jifeng04 hadoop]$ wget http://downloads.typesafe.com/scala/2.11.4/scala-2.11.4.tgz?_ga=1.248348352.61371242.1418807768
--2015-02-03 21:55:35--  http://downloads.typesafe.com/scala/2.11.4/scala-2.11.4.tgz?_ga=1.248348352.61371242.1418807768
正在解析主机 downloads.typesafe.com... 54.230.151.79, 54.230.151.80, 54.230.151.162, ...
Connecting to downloads.typesafe.com|54.230.151.79|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：26509669 (25M) [application/octet-stream]
Saving to: `scala-2.11.4.tgz?_ga=1.248348352.61371242.1418807768'

100%[===========================================================================>] 26,509,669   320K/s   in 71s     

2015-02-03 21:56:48 (366 KB/s) - `scala-2.11.4.tgz?_ga=1.248348352.61371242.1418807768' saved [26509669/26509669]

[jifeng@jifeng04 hadoop]$ ls
hadoop-1.2.1  scala-2.11.4.tgz?_ga=1.248348352.61371242.1418807768  tmp
[jifeng@jifeng04 hadoop]$ mv scala-2.11.4.tgz\?_ga\=1.248348352.61371242.1418807768 scala-2.11.4.tgz
[jifeng@jifeng04 hadoop]$ tar zxf scala-2.11.4.tgz 
[jifeng@jifeng04 hadoop]$ ls
hadoop-1.2.1  scala-2.11.4  scala-2.11.4.tgz  tmp
[jifeng@jifeng04 hadoop]$ scp ./scala-2.11.4.tgz jifeng@jifeng01.sohudo.com:/home/jifeng/hadoop
scala-2.11.4.tgz                                                                   100%   25MB  25.3MB/s   00:01    
[jifeng@jifeng04 hadoop]$

jifeng01 解压 

[jifeng@jifeng01 hadoop]$ tar zxf scala-2.11.4.tgz

3：配置scala

配置scala环境变量

export SCALA_HOME=$HOME/hadoop/scala-2.11.4
export PATH=$PATH:$ANT_HOME/bin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$HADOOP_HOME/bin:$MAHOUT_HOME/bin:$SCALA_HOME/bin

[jifeng@jifeng01 ~]$ vi ~/.bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
export JAVA_HOME=$HOME/jdk1.7.0_45
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=$HOME/hadoop/hadoop-1.2.1
export HADOOP_CONF_DIR=$HOME/hadoop/hadoop-1.2.1/conf
export ANT_HOME=$HOME/apache-ant-1.9.4
export HBASE_HOME=$HOME/hbase-0.94.21
export SQOOP_HOME=$HOME/sqoop-1.99.3-bin-hadoop100
export CATALINA_HOME=$SQOOP_HOME/server
export LOGDIR=$SQOOP_HOME/logs

export MAHOUT_HOME=$HOME/hadoop/mahout-distribution-0.9
export MAHOUT_CONF_DIR=$HOME/hadoop/mahout-distribution-0.9/conf

export SCALA_HOME=$HOME/hadoop/scala-2.11.4
export PATH=$PATH:$ANT_HOME/bin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$HADOOP_HOME/bin:$MAHOUT_HOME/bin:$SCALA_HOME/bin
~                                                                                                                    
~                                                                                                                    
".bash_profile" 28L, 889C 已写入
[jifeng@jifeng01 ~]$ source ~/.bash_profile
source ~/.bash_profile 配置文件马上生效

查看版本

[jifeng@jifeng01 hadoop]$ scala -version
Scala code runner version 2.11.4 -- Copyright 2002-2013, LAMP/EPFL

[jifeng@jifeng01 hadoop]$ scala
Welcome to Scala version 2.11.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_45).
Type in expressions to have them evaluated.
Type :help for more information.

scala>

4：配置spark环境变量

加入环境变量

export SPARK_HOME=$HOME/hadoop/spark-1.2.0-bin-hadoop1
export PATH=$PATH:$SPARK_HOME/bin

[jifeng@jifeng01 ~]$ vi ~/.bash_profile    

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
export JAVA_HOME=$HOME/jdk1.7.0_45
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=$HOME/hadoop/hadoop-1.2.1
export HADOOP_CONF_DIR=$HOME/hadoop/hadoop-1.2.1/conf
export ANT_HOME=$HOME/apache-ant-1.9.4
export HBASE_HOME=$HOME/hbase-0.94.21
export SQOOP_HOME=$HOME/sqoop-1.99.3-bin-hadoop100
export CATALINA_HOME=$SQOOP_HOME/server
export LOGDIR=$SQOOP_HOME/logs

export MAHOUT_HOME=$HOME/hadoop/mahout-distribution-0.9
export MAHOUT_CONF_DIR=$HOME/hadoop/mahout-distribution-0.9/conf

export SCALA_HOME=$HOME/hadoop/scala-2.11.4
export SPARK_HOME=$HOME/hadoop/spark-1.2.0-bin-hadoop1
export PATH=$PATH:$ANT_HOME/bin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$HADOOP_HOME/bin:$MAHOUT_HOME/bin:$SCALA_HOME/bin
export PATH=$PATH:$SPARK_HOME/bin
".bash_profile" 30L, 978C 已写入                                                                   
[jifeng@jifeng01 ~]$ source ~/.bash_profile

5：配置spark

复制配置文件 cp spark-env.sh.template spark-env.sh

[jifeng@jifeng01 ~]$ cd hadoop
[jifeng@jifeng01 hadoop]$ cd spark-1.2.0-bin-hadoop1
[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ ls
bin  conf  data  ec2  examples  lib  LICENSE  NOTICE  python  README.md  RELEASE  sbin
[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ cd conf
[jifeng@jifeng01 conf]$ ls
fairscheduler.xml.template  metrics.properties.template  spark-defaults.conf.template
log4j.properties.template   slaves.template              spark-env.sh.template
[jifeng@jifeng01 conf]$ cp spark-env.sh.template spark-env.sh

在spark-env.sh最后添加下面

export SCALA_HOME=/home/jifeng/hadoop/scala-2.11.4
export SPARK_MASTER_IP=jifeng01.sohudo.com
export SPARK_WORKER_MEMORY=2G
export JAVA_HOME=/home/jifeng/jdk1.7.0_45

# Generic options for the daemons used in the standalone deploy mode
# - SPARK_CONF_DIR      Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - SPARK_LOG_DIR       Where log files are stored.  (Default: ${SPARK_HOME}/logs)
# - SPARK_PID_DIR       Where the pid file is stored. (Default: /tmp)
# - SPARK_IDENT_STRING  A string representing this instance of spark. (Default: $USER)
# - SPARK_NICENESS      The scheduling priority for daemons. (Default: 0)
export SCALA_HOME=/home/jifeng/hadoop/scala-2.11.4
export SPARK_MASTER_IP=jifeng01.sohudo.com
export SPARK_WORKER_MEMORY=2G
export JAVA_HOME=/home/jifeng/jdk1.7.0_45
"spark-env.sh" 54L, 3378C 已写入                                                                   
[jifeng@jifeng01 conf]$ cd ..

6：启动

[jifeng@jifeng01 sbin]$ cd ..
[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/jifeng/hadoop/spark-1.2.0-bin-hadoop1/sbin/../logs/spark-jifeng-org.apache.spark.deploy.master.Master-1-jifeng01.sohudo.com.out


通过 http://jifeng01:8080/ 看到对应界面

7：启动worker

sbin/start-slaves.sh park://jifeng01:7077

[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ sbin/start-slaves.sh park://jifeng01:7077
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/jifeng/hadoop/spark-1.2.0-bin-hadoop1/sbin/../logs/spark-jifeng-org.apache.spark.deploy.worker.Worker-1-jifeng01.sohudo.com.out

[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ sbin/start-slaves.sh park://jifeng01.sohudo.com:7077
localhost: org.apache.spark.deploy.worker.Worker running as process 10273. Stop it first.
[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$

看界面的变化


8：测试

进入交互模式

[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ master=spark://jifeng01.sohudo.com:7077 ./bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
15/02/03 23:05:32 INFO spark.SecurityManager: Changing view acls to: jifeng
15/02/03 23:05:32 INFO spark.SecurityManager: Changing modify acls to: jifeng
15/02/03 23:05:32 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/02/03 23:05:32 INFO spark.HttpServer: Starting HTTP Server
15/02/03 23:05:32 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/02/03 23:05:32 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:46617
15/02/03 23:05:32 INFO util.Utils: Successfully started service 'HTTP class server' on port 46617.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.2.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
15/02/03 23:05:36 INFO spark.SecurityManager: Changing view acls to: jifeng
15/02/03 23:05:36 INFO spark.SecurityManager: Changing modify acls to: jifeng
15/02/03 23:05:36 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/02/03 23:05:37 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/02/03 23:05:37 INFO Remoting: Starting remoting
15/02/03 23:05:37 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@jifeng01.sohudo.com:41533]
15/02/03 23:05:37 INFO util.Utils: Successfully started service 'sparkDriver' on port 41533.
15/02/03 23:05:37 INFO spark.SparkEnv: Registering MapOutputTracker
15/02/03 23:05:37 INFO spark.SparkEnv: Registering BlockManagerMaster
15/02/03 23:05:37 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20150203230537-f1cb
15/02/03 23:05:37 INFO storage.MemoryStore: MemoryStore started with capacity 265.4 MB
15/02/03 23:05:37 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-0e198e02-fea8-49df-a1ab-8dd08e988cab
15/02/03 23:05:37 INFO spark.HttpServer: Starting HTTP Server
15/02/03 23:05:37 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/02/03 23:05:37 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:52156
15/02/03 23:05:37 INFO util.Utils: Successfully started service 'HTTP file server' on port 52156.
15/02/03 23:05:38 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/02/03 23:05:38 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/02/03 23:05:38 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
15/02/03 23:05:38 INFO ui.SparkUI: Started SparkUI at http://jifeng01.sohudo.com:4040
15/02/03 23:05:38 INFO executor.Executor: Using REPL class URI: http://10.5.4.54:46617
15/02/03 23:05:38 INFO util.AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@jifeng01.sohudo.com:41533/user/HeartbeatReceiver
15/02/03 23:05:38 INFO netty.NettyBlockTransferService: Server created on 39115
15/02/03 23:05:38 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/02/03 23:05:38 INFO storage.BlockManagerMasterActor: Registering block manager localhost:39115 with 265.4 MB RAM, BlockManagerId(<driver>, localhost, 39115)
15/02/03 23:05:38 INFO storage.BlockManagerMaster: Registered BlockManager
15/02/03 23:05:38 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.

scala>

测试

输入命令：

val file=sc.textFile("hdfs://jifeng01.sohudo.com:9000/user/jifeng/in/test1.txt")
val count=file.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey(_+_)
count.collect()

scala> val file=sc.textFile("hdfs://jifeng01.sohudo.com:9000/user/jifeng/in/test1.txt")
15/02/03 23:07:25 INFO storage.MemoryStore: ensureFreeSpace(32768) called with curMem=0, maxMem=278302556
15/02/03 23:07:25 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 32.0 KB, free 265.4 MB)
15/02/03 23:07:25 INFO storage.MemoryStore: ensureFreeSpace(5070) called with curMem=32768, maxMem=278302556
15/02/03 23:07:25 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 5.0 KB, free 265.4 MB)
15/02/03 23:07:25 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:39115 (size: 5.0 KB, free: 265.4 MB)
15/02/03 23:07:25 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
15/02/03 23:07:25 INFO spark.SparkContext: Created broadcast 0 from textFile at <console>:12
file: org.apache.spark.rdd.RDD[String] = hdfs://jifeng01.sohudo.com:9000/user/jifeng/in/test1.txt MappedRDD[1] at textFile at <console>:12

scala> val count=file.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey(_+_)
15/02/03 23:07:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/02/03 23:07:38 WARN snappy.LoadSnappy: Snappy native library not loaded
15/02/03 23:07:38 INFO mapred.FileInputFormat: Total input paths to process : 1
count: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at <console>:14

scala> count.collect()
15/02/03 23:07:44 INFO spark.SparkContext: Starting job: collect at <console>:17
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Registering RDD 3 (map at <console>:14)
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Got job 0 (collect at <console>:17) with 3 output partitions (allowLocal=false)
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Final stage: Stage 1(collect at <console>:17)
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 0)
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Missing parents: List(Stage 0)
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Submitting Stage 0 (MappedRDD[3] at map at <console>:14), which has no missing parents
15/02/03 23:07:44 INFO storage.MemoryStore: ensureFreeSpace(3584) called with curMem=37838, maxMem=278302556
15/02/03 23:07:44 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.5 KB, free 265.4 MB)
15/02/03 23:07:44 INFO storage.MemoryStore: ensureFreeSpace(2543) called with curMem=41422, maxMem=278302556
15/02/03 23:07:44 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.5 KB, free 265.4 MB)
15/02/03 23:07:44 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:39115 (size: 2.5 KB, free: 265.4 MB)
15/02/03 23:07:44 INFO storage.BlockManagerMaster: Updated info of block broadcast_1_piece0
15/02/03 23:07:44 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:838
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from Stage 0 (MappedRDD[3] at map at <console>:14)
15/02/03 23:07:44 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 3 tasks
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1310 bytes)
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, ANY, 1310 bytes)
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, ANY, 1310 bytes)
15/02/03 23:07:44 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
15/02/03 23:07:44 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2)
15/02/03 23:07:44 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1)
15/02/03 23:07:44 INFO rdd.HadoopRDD: Input split: hdfs://jifeng01.sohudo.com:9000/user/jifeng/in/test1.txt:6+6
15/02/03 23:07:44 INFO rdd.HadoopRDD: Input split: hdfs://jifeng01.sohudo.com:9000/user/jifeng/in/test1.txt:0+6
15/02/03 23:07:44 INFO rdd.HadoopRDD: Input split: hdfs://jifeng01.sohudo.com:9000/user/jifeng/in/test1.txt:12+1
15/02/03 23:07:44 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1897 bytes result sent to driver
15/02/03 23:07:44 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1897 bytes result sent to driver
15/02/03 23:07:44 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1897 bytes result sent to driver
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 164 ms on localhost (1/3)
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 160 ms on localhost (2/3)
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 160 ms on localhost (3/3)
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Stage 0 (map at <console>:14) finished in 0.185 s
15/02/03 23:07:44 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
15/02/03 23:07:44 INFO scheduler.DAGScheduler: looking for newly runnable stages
15/02/03 23:07:44 INFO scheduler.DAGScheduler: running: Set()
15/02/03 23:07:44 INFO scheduler.DAGScheduler: waiting: Set(Stage 1)
15/02/03 23:07:44 INFO scheduler.DAGScheduler: failed: Set()
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Missing parents for Stage 1: List()
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Submitting Stage 1 (ShuffledRDD[4] at reduceByKey at <console>:14), which is now runnable
15/02/03 23:07:44 INFO storage.MemoryStore: ensureFreeSpace(2112) called with curMem=43965, maxMem=278302556
15/02/03 23:07:44 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.1 KB, free 265.4 MB)
15/02/03 23:07:44 INFO storage.MemoryStore: ensureFreeSpace(1546) called with curMem=46077, maxMem=278302556
15/02/03 23:07:44 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1546.0 B, free 265.4 MB)
15/02/03 23:07:44 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:39115 (size: 1546.0 B, free: 265.4 MB)
15/02/03 23:07:44 INFO storage.BlockManagerMaster: Updated info of block broadcast_2_piece0
15/02/03 23:07:44 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from Stage 1 (ShuffledRDD[4] at reduceByKey at <console>:14)
15/02/03 23:07:44 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 3 tasks
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 3, localhost, PROCESS_LOCAL, 1056 bytes)
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 4, localhost, PROCESS_LOCAL, 1056 bytes)
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 5, localhost, PROCESS_LOCAL, 1056 bytes)
15/02/03 23:07:44 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 3)
15/02/03 23:07:44 INFO executor.Executor: Running task 2.0 in stage 1.0 (TID 5)
15/02/03 23:07:44 INFO executor.Executor: Running task 1.0 in stage 1.0 (TID 4)
15/02/03 23:07:44 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
15/02/03 23:07:44 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 3 blocks
15/02/03 23:07:44 INFO storage.ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
15/02/03 23:07:44 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 8 ms
15/02/03 23:07:44 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 7 ms
15/02/03 23:07:44 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 8 ms
15/02/03 23:07:44 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 3). 820 bytes result sent to driver
15/02/03 23:07:44 INFO executor.Executor: Finished task 2.0 in stage 1.0 (TID 5). 820 bytes result sent to driver
15/02/03 23:07:44 INFO executor.Executor: Finished task 1.0 in stage 1.0 (TID 4). 990 bytes result sent to driver
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 3) in 44 ms on localhost (1/3)
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 1.0 (TID 5) in 44 ms on localhost (2/3)
15/02/03 23:07:44 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 4) in 45 ms on localhost (3/3)
15/02/03 23:07:44 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Stage 1 (collect at <console>:17) finished in 0.048 s
15/02/03 23:07:44 INFO scheduler.DAGScheduler: Job 0 finished: collect at <console>:17, took 0.366077 s
res0: Array[(String, Int)] = Array((hadoop,1), (hello,1))

scala>

看到控制台有如下结果：res0: Array[(String, Int)] = Array((hadoop,1), (hello,1))

同时也可以将结果保存到HDFS上


scala>count.saveAsTextFile("hdfs://jifeng01.sohudo.com:9000/user/jifeng/out/test1.txt")


9：停止

./sbin/stop-master.sh

[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ ./sbin/stop-master.sh 
stopping org.apache.spark.deploy.master.Master


wind520
关注
专栏目录
spark(2) -- spark运行模式及Spark local模式安装与Spark Standalone集群
erainm
10-01 968
 Spark 框架编写的应用程序可以运行在本地模式(Local Mode)、集群模式(Cluster Mode) 和云服务(Cloud)，方便开发测试和生产部署。 概述 本地模式:Local Mode 将Spark 应用程序中任务Task运行在一个本地JVM Process进程中，通常开发测试使用。 本地模式LocalMode运行:启动一个JVM进程，里面运行Task任务，Task任务并行运行数目依据分 配CPU Core核数，在Spark应用中，一个Task任务运行，需要1Core CPU。 1)
spark-单机部署（三）
weixin_45025143的博客
11-18 2343
Spark的windows、linux单机部署 1、windows单机部署   相信大多数同学用的都是windows系统，没错，我也是windows系统，主要是因为公司电脑安装了深信服的办公软件，但是这个不支持linux系统，所有我们公司都强制被使用win10。为了调试方便，还是习惯在电脑上装上单机Spark，好在Spark也支持win10。 废话不多说，我们开始哈。。。 1.1、安装Java环境   这里我们安装Java8，哎，这会是不是有同学问了现在Java15都有了我们为啥还用Java8了？❓❓❓
Spark Standalone 单机版部署
10-17
Spark standalone 单机版部署，看了网上很多方法，事实证明都是错误的，本人亲身经历，在导师的指导下，成功配置成功单机版。
安装spark1.3.1单机环境
apple01010105的博客
04-27 158
本文介绍安装spark单机环境的方法，可用于测试及开发。主要分成以下4部分： （1）环境准备 （2）安装scala （3）安装spark （4）验证安装情况 1、环境准备 （1）配套软件版本要求：Spark runs on Java 6+ and Python 2.6+. For the Scala API, Spark 1.3.1 uses Scala ...
spark单机安装
最新发布
Aronup
09-14 383
【代码】spark单机安装。
Spark Standalone Mode
dh15303249535的博客
07-18 125
因为Spark与Hadoop是关联的，所以在安装Spark前应该根据已安装的Hadoop版本来选择待安装的Sqark版本，要不然就会报“Server IPC version X cannot communicate with client version Y”的错误。 我安装的Hadoop版本为Hadoop2.4.0(下载)，选择的Spark版本为spark-1.2.0-bin-ha...
Spark Standalone Mode 安装
yang7551735的专栏
01-10 380
Spark Standalone Mode 安装
Spark 安装 Standalone模式 (单机与集群)
Be yourself.
03-29 2513
前言 今天我们了解下Spark的安装. Spark的安装模式有很多模式, 主要包括 Spark Standalone、Running Spark on YARN、Running Spark on Kubernetes、Running Spark on Mesos. 其中YARN/Mesos/Kubernetes都是资源的调度器. 本章我们将简单的介绍下Spark Standalone在单机与集群模...
Spark-3.0.0 重点归纳
m0_67991849的博客
06-23 1470
spark重点归纳
【人工智能】Hadoop和Spark使用教程
大雨淅淅的博客
07-29 919
Hadoop和Spark是大数据处理领域的两个重要工具，它们各自具有独特的功能和优势，经常一起使用以构建高效的数据处理和分析系统。以下是一个关于Hadoop和Spark使用的基本教程，旨在提供从安装、配置到基本应用开发的概览。
04-Spark入门
PenguinLittle的博客
03-01 1946
04-Spark入 熟悉spark相关概念 搭建spark集群 编写简单的spark应用程序 2.spark概述 2.1.什么是spark spark是基于内存的快速、通用、可扩展的大数据分析计算引擎。它的计算速度非常快。但是仅仅只涉及到数据的计算，并没有涉及到数据的存储。 2.2.为什么要学习spark 运行速度比mapred uce快很多 2.3.spark特点 速度快（比mapreduce在内存中快100倍，在磁盘中快10倍） spark中的job中间结果可以不落地，
spark单机版安装
maodun
03-31 682
单机版安装，只需要jdk和spark， 不需要hadoop。
Spark安装单机版
RJSong
03-14 584
1、下载地址      https://archive.apache.org/dist/spark/spark-2.1.0/2、通过FTP上传至Linux服务器某目录文件下    3、解压缩      tar -zxvf spark-2.1.0-bin-without-hadoop.tgz.gz4、重命名      mv spark-2.1.0-bin-without-hadoop spark5、...
Spark 单机版（不依赖Hadoop）
qq_43250374的博客
12-03 4883
Spark 单机版 Spark 还久没有配置了，做个配置记录一下 Spark 单机版，不依赖Hadoop，只需安装JDK，Scala。 安装步骤Spark 单机版简介一、前期准备二、安装步骤1.环境依赖2.Spark 文件修改3.启动测试 简介 查看网上好多安装Spark的步骤都太长了，不够简约，再此做个简约的步骤记录 一、前期准备 我使用版本的是 ： jdk1.8.0_141 scala-2.11.12 spark-2.3.2-bin-hadoop2.7 二、安装步骤 1.环境依赖 全部解压后在
Spark单机安装
欢迎订阅 -- 小源笔迹
04-30 438
spark安装，Spark支持运行在Yarn或者Mesos上，同时也提供自己的资源管理系统，即Standalone模式．为了方便本地开发，Spark提供了Local运行模式．不同的环境的开发代码是相同的，只是在提交到不同的调度系统时参数不同而已．
spark-standalone模式怎么安装配置
03-28
1. 下载Spark安装包 首先，需要下载Spark的安装包。可以从Spark官网下载，也可以从Apache镜像站下载。 2. 解压安装包 将Spark安装包解压到某个目录下，例如/opt/spark。 3. 配置Spark环境变量 打开.bashrc文件，添加如下内容： export SPARK_HOME=/opt/spark export PATH=$SPARK_HOME/bin:$PATH 4. 配置Spark Standalone模式 在Spark安装目录下，进入conf目录，复制一份模板文件spark-env.sh.template，并将复制出来的文件重命名为spark-env.sh。 在spark-env.sh文件中设置以下变量： export SPARK_MASTER_HOST=<master节点IP地址> export SPARK_MASTER_PORT=<master节点端口，默认为7077> export SPARK_MASTER_WEBUI_PORT=<master节点Web端口，默认为8080> export SPARK_WORKER_CORES=<每个Worker节点的CPU核数，默认为1> export SPARK_WORKER_MEMORY=<每个Worker节点的内存，默认为1g> export SPARK_WORKER_INSTANCES=<Worker节点数量，默认为1> 5. 启动Spark Standalone集群 在Master节点上执行以下命令启动Spark Standalone集群： $SPARK_HOME/sbin/start-master.sh 在Worker节点上执行以下命令启动Worker节点： $SPARK_HOME/sbin/start-worker.sh spark://<master节点IP地址>:<master节点端口> 6. 访问Spark Web界面 在浏览器中访问Master节点的Web界面，地址为http://<master节点IP地址>:<master节点Web端口>，可以查看Spark集群的状态和任务情况。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值