Spark 1.2 集群环境安装

最新推荐文章于 2023-02-21 11:26:01 发布
最新推荐文章于 2023-02-21 11:26:01 发布
阅读量1.1k 收藏
点赞数
分类专栏： Spark 文章标签： spark 集群
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wind520/article/details/43494761
版权

我是在单机环境下修改下配置完成的集群模式

单机安装查看：http://blog.csdn.net/wind520/article/details/43458925

参考官网配置：http://spark.apache.org/docs/latest/spark-standalone.html

1：修改slaves配置

[jifeng@jifeng01 conf]$ cp slaves.template slaves
[jifeng@jifeng01 conf]$ vi slaves


# A Spark Worker will be started on each of the machines listed below.
jifeng02.sohudo.com
jifeng03.sohudo.com
jifeng04.sohudo.com
                                                                                                                  
"slaves" 4L, 131C 已写入                                                                           
[jifeng@jifeng01 conf]$ cat slaves
# A Spark Worker will be started on each of the machines listed below.
jifeng02.sohudo.com
jifeng03.sohudo.com
jifeng04.sohudo.com

2：复制文件到其它节点 

[jifeng@jifeng01 hadoop]$ scp -r ./spark-1.2.0-bin-hadoop1 jifeng@jifeng02.sohudo.com:/home/jifeng/hadoop/
[jifeng@jifeng01 hadoop]$ scp -r ./spark-1.2.0-bin-hadoop1 jifeng@jifeng03.sohudo.com:/home/jifeng/hadoop/
[jifeng@jifeng01 hadoop]$ scp -r ./spark-1.2.0-bin-hadoop1 jifeng@jifeng04.sohudo.com:/home/jifeng/hadoop/
[jifeng@jifeng01 hadoop]$ scp -r ./scala-2.11.4 jifeng@jifeng02.sohudo.com:/home/jifeng/hadoop/
[jifeng@jifeng01 hadoop]$ scp -r ./scala-2.11.4 jifeng@jifeng03.sohudo.com:/home/jifeng/hadoop/
[jifeng@jifeng01 hadoop]$ scp -r ./scala-2.11.4 jifeng@jifeng04.sohudo.com:/home/jifeng/hadoop/

3：启动

  • sbin/start-all.sh - Starts both a master and a number of slaves as described above.

[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/jifeng/hadoop/spark-1.2.0-bin-hadoop1/sbin/../logs/spark-jifeng-org.apache.spark.deploy.master.Master-1-jifeng01.sohudo.com.out
jifeng04.sohudo.com: starting org.apache.spark.deploy.worker.Worker, logging to /home/jifeng/hadoop/spark-1.2.0-bin-hadoop1/sbin/../logs/spark-jifeng-org.apache.spark.deploy.worker.Worker-1-jifeng04.sohudo.com.out
jifeng02.sohudo.com: starting org.apache.spark.deploy.worker.Worker, logging to /home/jifeng/hadoop/spark-1.2.0-bin-hadoop1/sbin/../logs/spark-jifeng-org.apache.spark.deploy.worker.Worker-1-jifeng02.sohudo.com.out
jifeng03.sohudo.com: starting org.apache.spark.deploy.worker.Worker, logging to /home/jifeng/hadoop/spark-1.2.0-bin-hadoop1/sbin/../logs/spark-jifeng-org.apache.spark.deploy.worker.Worker-1-jifeng03.sohudo.com.out
[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$


4：web管理页面

spark集群的web管理页面访问：http://jifeng01.sohudo.com:8080/

5：启动spark-shell控制台

./spark-shell

[jifeng@jifeng01 spark-1.2.0-bin-hadoop1]$ cd bin
[jifeng@jifeng01 bin]$ ./spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
15/02/04 22:11:10 INFO spark.SecurityManager: Changing view acls to: jifeng
15/02/04 22:11:10 INFO spark.SecurityManager: Changing modify acls to: jifeng
15/02/04 22:11:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/02/04 22:11:10 INFO spark.HttpServer: Starting HTTP Server
15/02/04 22:11:10 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/02/04 22:11:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:57780
15/02/04 22:11:10 INFO util.Utils: Successfully started service 'HTTP class server' on port 57780.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.2.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
15/02/04 22:11:15 INFO spark.SecurityManager: Changing view acls to: jifeng
15/02/04 22:11:15 INFO spark.SecurityManager: Changing modify acls to: jifeng
15/02/04 22:11:15 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/02/04 22:11:15 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/02/04 22:11:15 INFO Remoting: Starting remoting
15/02/04 22:11:15 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@jifeng01.sohudo.com:48769]
15/02/04 22:11:15 INFO util.Utils: Successfully started service 'sparkDriver' on port 48769.
15/02/04 22:11:15 INFO spark.SparkEnv: Registering MapOutputTracker
15/02/04 22:11:15 INFO spark.SparkEnv: Registering BlockManagerMaster
15/02/04 22:11:15 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20150204221115-20b0
15/02/04 22:11:15 INFO storage.MemoryStore: MemoryStore started with capacity 265.4 MB
15/02/04 22:11:15 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-6d24f541-4ce1-4e12-9be2-4296791bd73f
15/02/04 22:11:15 INFO spark.HttpServer: Starting HTTP Server
15/02/04 22:11:15 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/02/04 22:11:15 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:42647
15/02/04 22:11:15 INFO util.Utils: Successfully started service 'HTTP file server' on port 42647.
15/02/04 22:11:16 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/02/04 22:11:16 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/02/04 22:11:16 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
15/02/04 22:11:16 INFO ui.SparkUI: Started SparkUI at http://jifeng01.sohudo.com:4040
15/02/04 22:11:16 INFO executor.Executor: Using REPL class URI: http://10.5.4.54:57780
15/02/04 22:11:16 INFO util.AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@jifeng01.sohudo.com:48769/user/HeartbeatReceiver
15/02/04 22:11:16 INFO netty.NettyBlockTransferService: Server created on 58669
15/02/04 22:11:16 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/02/04 22:11:16 INFO storage.BlockManagerMasterActor: Registering block manager localhost:58669 with 265.4 MB RAM, BlockManagerId(<driver>, localhost, 58669)
15/02/04 22:11:16 INFO storage.BlockManagerMaster: Registered BlockManager
15/02/04 22:11:16 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.

scala>


6：Spark shell管理页面

进入后还可打开访问web控制页面，访问http://jifeng01.sohudo.com:4040/


7：测试

输入命令：

val file=sc.textFile("hdfs://jifeng01.sohudo.com:9000/user/jifeng/in/test1.txt")

读取文件
val count=file.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey(_+_)

统计单词数量
count.collect()

collect命令提交并执行job

8：查看执行结果


9：退出shell

:quit

scala> :quit
Stopping spark context.
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/kill,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/static,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors/threadDump/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors/threadDump,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/environment/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/environment,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/rdd/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/rdd,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/pool/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/pool,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/jobs/job/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/jobs/job,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/jobs/json,null}
15/02/04 22:28:51 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/jobs,null}
15/02/04 22:28:51 INFO ui.SparkUI: Stopped Spark web UI at http://jifeng01.sohudo.com:4040
15/02/04 22:28:51 INFO scheduler.DAGScheduler: Stopping DAGScheduler
15/02/04 22:28:52 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
15/02/04 22:28:52 INFO storage.MemoryStore: MemoryStore cleared
15/02/04 22:28:52 INFO storage.BlockManager: BlockManager stopped
15/02/04 22:28:52 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
15/02/04 22:28:52 INFO spark.SparkContext: Successfully stopped SparkContext
15/02/04 22:28:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/02/04 22:28:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/02/04 22:28:52 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
[jifeng@jifeng01 bin]$

10：停止集群

sbin/stop-all.sh 


wind520
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
CDH 安装 大数据组件 报错
Masli的博客
02-17 2873
使用CDH来安装大数据组件的时候，当安装到YARN、Hbase、Spark的时候报错，主要原因是YARN、Hbase、Spark在HDFS里面没有权限去创建文件 Spark SecurityManager: authentication disabled; ui acls disabled; users with view permissions : Set(spark); groups with...
sparkstreaming整合kafka运行报错
flashtown的博客
11-30 800
20/11/30 14:24:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(flashtown); groups with view permissions: Set(); users with modify permissions: Set(flashtown); groups with modify permis..
Spark问题
急速虫子
07-09 683
15/07/09 17:50:36 INFO SecurityManager: Changing view acls to: dev001, 15/07/09 17:50:36 INFO SecurityManager: Changing modify acls to: dev001, 15/07/09 17:50:36 INFO SecurityManager: SecurityManager:
Docker 搭建Spark集群
纸上得来终觉浅，绝知此事要躬行
09-01 1787
文章转自：https://blog.csdn.net/lizongti/article/details/99674153 Docker 搭建Spark集群 环境准备 依赖 安装Docker Local模式（without Docker） 安装JDK 安装Scala 安装Spark 测试 Standalone 模式（without Docker） 更改hostname 更改配置 ssh免密登录 关闭防火墙 启动Spark集群 访问集群web 测.
Spark打包成jar在shell能成功，但以spark-submit却运行失败
qq_43719388的博客
10-28 1521
有的错误确实很莫名其妙 2020-10-28 15:13:21,199 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with
spark1.2集群搭建文档
05-05
spark集群搭建文档,版本号：spark-1.2.1-bin-hadoop2.3.tgz
Linux环境Hadoop2.6+Hbase1.2集群安装部署
03-01
在构建大数据处理环境时，Linux环境下的Hadoop2.6+Hbase1.2集群安装部署是基础步骤，而Spark分布式集群的搭建则是提升数据处理效率的关键。这些技术的组合使用，可以为大规模数据处理提供高效、可靠的解决方案。 ...
spark2.1集群安装
11-02
### Spark 2.1 集群安装及配置详解 #### 一、规划与环境准备 根据提供的信息，“cancer01”被指定为Master节点，“cancer02”、“cancer03”、“cancer04”以及“cancer05”作为Worker节点。这些节点将共同构成一...
SparkR安装部署及数据分析实例
02-01
##### 1.2 rJava安装 **1.2.1 rJava介绍** rJava是一个强大的工具，它提供了一种在R中调用Java的方法，同时也支持从Java调用R的功能。通过JNI (Java Native Interface) 实现R与Java之间的交互。 **1.2.2 rJava...
spark三种模式部署安装（基于Anaconda3实现spark编程）
最新发布
03-25
**Standalone模式**是Spark自带的一种简单的集群管理模式，适用于中小型集群环境。在此模式下，用户需要手动搭建Master和Worker节点，并通过配置文件指定Master节点的地址。通过这种方式，可以在多个节点之间分配...
Apache Tomcat 8 (8.0.20) - Security Manager HOW-TO.
12-01
- Apache Tomcat 8 (8.0.20) - Security Manager HOW-TO
Caused by: java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclus
易者悠也
03-31 1万+
[hjr@master bin]$ ./spark-shell Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true 16/03/3
spark集群：无法指定被请求的地址,Service 'Driver' could not bind on a random free port.
窃光者
07-10 8578
19/07/09 17:00:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify pe...
org.apache.spark.SparkException: Task not serializable
热门推荐
Shockang的博客
07-25 3万+
报错原因解析如果出现“org.apache.spark.SparkException: Task not serializable”错误，一般是因为在 map 、 filter 等的参数使用了外部的变量，但是这个变量不能序列化(不是说不可以引用外部变量，只是要做好序列化工作)。
authentication disabled； ui acls disabled； users with view permissions: Set(han)；
qq_65303368的博客
02-21 1376
刚把数据上传到hdfs准备用spark测试一下看能不能读取结果在创建执行环境时遇到了这个报错。翻译一下好像是权限问题。
spark-shell on yarn 出错解决【启动命令bin/spark-shell --master yarn-client出现错误，类ExecutorLauncher 找不到】
Thinking
12-01 1万+
文章来源：http://www.dataguru.cn/thread-331456-1-1.html 今天想要将spark-shell 在yarn-client的状态下 结果出错： [python] view plaincopy [hadoop@localhost spark-1.0.1-bin-hadoop2]$ bin/spark-shell
问题 Spark Yarn集群模式 exitCode = 13
zhangxiping
03-25 7123
背景 今天一同事在提交任务时遇到一个很奇葩的问题，使用的是集群模式提交的spark任务 下面是运行的日志： Log Length: 19060 20/03/25 14:43:03 INFO util.SignalUtils: Registered signal handler for TERM 20/03/25 14:43:03 INFO util.SignalUtils: Registered...
spark on yarn 的那些坑
qq_32253371的博客
10-25 1889
在公司6个节点的测试集群运行得好好的，结果也很正常，然后放上60个节点的预生产环境，我勒个擦，搞了我两天，主要是生产环境的那些家伙不配合，一个问题搞得拖啊拖 ，首先是安全认证问题，截取一两个有意义的吧：       1.没有认证 Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initi
spark 启动报错 -- 主机别名
hzdxw的专栏
07-03 1万+
spark 启动报错 -- 主机别名
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值