Centos6 下 spark1.6.3_hadoop2.6 分布式集群环境搭建

最新推荐文章于 2021-12-15 01:07:17 发布
最新推荐文章于 2021-12-15 01:07:17 发布
阅读量824 收藏 1
点赞数
分类专栏： Hadoop 文章标签： 分布式 hadoop spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/sakulamartain/article/details/121033401
版权

Scala 安装

下载 ，解压

下载 Scala-2.11.8.tgz，

修改环境变量文件 /etc/profile

添加以下内容

export SCALA_HOME=/root/workspace/software/scala-2.10.4

export PATH=$PATH:$SCALA_HOME/bin1

source 使之生效

source /etc/profile

验证 Scala 安装

[root@racnode2 bin]# scala

Welcome to Scala 2.11.8 (JavaHotSpot(TM) 64-Bit Server VM, Java 1.8.0_144).

Type in expressions for evaluation. Ortry :help.

scala> var a = "helloworld!"

a: String = hello world!

scala> :q

[root@racnode2 bin]#

Spark 安装

下载，解压

下载 Spark-1.6.3-bin-hadoop2.6.tgz，下载地址Downloads | Apache Spark。

修改环境变量文件 /etc/profile, 添加以下内容。

export SPARK_HOME=/usr/spark-1.6.3-bin-hadoop2.6

export PATH=$PATH:XXX其他软件的环境变量:$SPARK_HOME/bin# 在最后添加:$SPARK_HOME/bin

source 使之生效

source /etc/profile

Spark 配置

spark-env.sh

进入 Spark 安装目录下的 /usr/spark1.6.3_hadoop2.6/conf 目录， 拷贝 spark-env.sh.template 到 spark-env.sh。

cp spark-env.sh.template spark-env.sh

编辑 spark-env.sh，在其中添加以下配置信息：

export SCALA_HOME=/usr/scala-2.11.8

export JAVA_HOME=/usr/jdk1.8.0_144

export HADOOP_HOME=/usr/local/hadoop

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export SPARK_MASTER_IP=192.168.52.128

export SPARK_LOCAL_DIRS=/usr/spark-1.6.3-bin-hadoop2.6

export SPARK_WORKER_MEMORY=4g

JAVA_HOME 指定 Java 安装目录； 
SCALA_HOME 指定 scala 安装目录； 
SPARK_MASTER_IP 指定 Spark 集群 Master 节点的 IP 地址； 
SPARK_WORKER_MEMORY 指定的是 Worker 节点能够分配给 Executors 的最大内存大小； 
HADOOP_CONF_DIR 指定 Hadoop 集群配置文件目录。 
SPARK_WORKER_MEMORY 我这边的机器内存32g，我设置内存为20g,更加自己的情况修改。

Tips:这里的JDK要部署1.7版本以上的，否则会报如下错误：

[root@racnode2 sbin]# ./start-master.sh

starting org.apache.spark.deploy.master.Master, logging to/usr/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-racnode2.hadoop.out

failed to launch org.apache.spark.deploy.master.Master:

        atjava.lang.ClassLoader.loadClass(ClassLoader.java:247)

  Could not find the main class:org.apache.spark.launcher.Main.  Programwill exit.

full log in/usr/spark-1.6.3-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-racnode2.hadoop.out

这个错误的原因就是当前用户在尝试使用低版本的jdk运行或者编译高版本的Java程序，报无法找到class的error，遇到该错误，下载7或者8的jdk，然后重新部署，并更新环境变量信息即可，这里的环境变量涉及到两个地方的配置：/etc/profile和/usr/spark1.6.3_hadoop2.6/conf/ spark-env.sh,

启动 Spark 集群

启动 Hadoop 集群

[root@racnode2conf]# start-all.sh

[root@racnode2conf]# jps

2457 NameNode

2971 NodeManager

2877ResourceManager

2549 DataNode

3681 Jps

2689SecondaryNameNode

[root@racnode2conf]#

启动 Spark 集群

[root@racnode2 sbin]# ./start-all.sh

This script is Deprecated. Instead usestart-dfs.sh and start-yarn.sh

Starting namenodes on [racnode2.hadoop]

racnode2.hadoop: starting namenode,logging to /usr/local/hadoop/logs/hadoop-root-namenode-racnode2.hadoop.out

racnode2.hadoop: starting datanode,logging to /usr/local/hadoop/logs/hadoop-root-datanode-racnode2.hadoop.out

racnode1.hadoop: ssh: connect to hostracnode1.hadoop port 22: No route to host

Starting secondary namenodes [0.0.0.0]

0.0.0.0: reverse mapping checkinggetaddrinfo for 5ulkyfcsae9bnha [127.0.0.1] failed - POSSIBLE BREAK-IN ATTEMPT!

0.0.0.0: starting secondarynamenode,logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-racnode2.hadoop.out

starting yarn daemons

starting resourcemanager, logging to/usr/local/hadoop/logs/yarn-root-resourcemanager-racnode2.hadoop.out

racnode2.hadoop: starting nodemanager,logging to /usr/local/hadoop/logs/yarn-root-nodemanager-racnode2.hadoop.out

racnode1.hadoop: ssh: connect to hostracnode1.hadoop port 22: No route to host

[root@racnode2 sbin]# jps

4662 NameNode

4934 SecondaryNameNode

5673 Worker

6255 Jps

5170 NodeManager

4753 DataNode

5609 Master

5077 ResourceManager

[root@racnode2 sbin]#

可以看到多了一个新进程 Master和Worker 进程。

浏览器查看 Spark 集群信息

访问：http://masterIP:8080,如下图：

  

 

使用 spark-shell

运行 spark-shell，可以进入 Spark 的 shell 控制台，如下： 
[root@racnode2 bin]# ./spark-shell

17/09/07 19:40:51 INFOspark.SecurityManager: Changing view acls to: root

17/09/07 19:40:51 INFOspark.SecurityManager: Changing modify acls to: root

17/09/07 19:40:51 INFO spark.SecurityManager:SecurityManager: authentication disabled; ui acls disabled; users with viewpermissions: Set(root); users with modify permissions: Set(root)

17/09/07 19:40:51 INFO spark.HttpServer:Starting HTTP Server

17/09/07 19:40:52 INFO server.Server:jetty-8.y.z-SNAPSHOT

17/09/07 19:40:52 INFOserver.AbstractConnector: Started SocketConnector@0.0.0.0:57628

17/09/07 19:40:52 INFO util.Utils:Successfully started service 'HTTP class server' on port 57628.

Welcome to

     ____              __

    / __/__  ___ _____/ /__

   _\ \/ _ \/ _ `/ __/  '_/

  /___/ .__/\_,_/_/ /_/\_\   version1.6.3

     /_/

Using Scala version 2.10.5 (JavaHotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)

Type in expressions to have themevaluated.

Type :help for more information.

17/09/07 19:40:59 INFOspark.SparkContext: Running Spark version 1.6.3

17/09/07 19:40:59 INFOspark.SecurityManager: Changing view acls to: root

17/09/07 19:40:59 INFOspark.SecurityManager: Changing modify acls to: root

17/09/07 19:40:59 INFOspark.SecurityManager: SecurityManager: authentication disabled; ui aclsdisabled; users with view permissions: Set(root); users with modifypermissions: Set(root)

17/09/07 19:41:00 INFO util.Utils:Successfully started service 'sparkDriver' on port 55184.

17/09/07 19:41:00 INFOslf4j.Slf4jLogger: Slf4jLogger started

17/09/07 19:41:01 INFO Remoting:Starting remoting

17/09/07 19:41:01 INFO Remoting:Remoting started; listening on addresses:[akka.tcp://sparkDriverActorSystem@192.168.52.128:55089]

17/09/07 19:41:01 INFO util.Utils:Successfully started service 'sparkDriverActorSystem' on port 55089.

17/09/07 19:41:01 INFO spark.SparkEnv:Registering MapOutputTracker

17/09/07 19:41:01 INFO spark.SparkEnv:Registering BlockManagerMaster

17/09/07 19:41:01 INFO storage.DiskBlockManager:Created local directory at/usr/spark-1.6.3-bin-hadoop2.6/blockmgr-b970925a-643c-4f8a-810d-df461eef8db5

17/09/07 19:41:01 INFOstorage.MemoryStore: MemoryStore started with capacity 517.4 MB

17/09/07 19:41:01 INFO spark.SparkEnv: RegisteringOutputCommitCoordinator

17/09/07 19:41:02 INFO server.Server:jetty-8.y.z-SNAPSHOT

17/09/07 19:41:02 INFOserver.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040

17/09/07 19:41:02 INFO util.Utils:Successfully started service 'SparkUI' on port 4040.

17/09/07 19:41:02 INFO ui.SparkUI:Started SparkUI at http://192.168.52.128:4040

17/09/07 19:41:02 INFOexecutor.Executor: Starting executor ID driver on host localhost

17/09/07 19:41:02 INFOexecutor.Executor: Using REPL class URI: http://192.168.52.128:57628

17/09/07 19:41:02 INFO util.Utils:Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 51665.

17/09/07 19:41:02 INFO netty.NettyBlockTransferService:Server created on 51665

17/09/07 19:41:02 INFOstorage.BlockManagerMaster: Trying to register BlockManager

17/09/07 19:41:02 INFOstorage.BlockManagerMasterEndpoint: Registering block manager localhost:51665with 517.4 MB RAM, BlockManagerId(driver, localhost, 51665)

17/09/07 19:41:02 INFOstorage.BlockManagerMaster: Registered BlockManager

17/09/07 19:41:03 INFO repl.SparkILoop:Created spark context..

Spark context available as sc.

17/09/07 19:41:05 INFO hive.HiveContext:Initializing execution hive, version 1.2.1

17/09/07 19:41:05 INFOclient.ClientWrapper: Inspected Hadoop version: 2.6.0

17/09/07 19:41:05 INFOclient.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims forHadoop version 2.6.0

17/09/07 19:41:06 INFOmetastore.HiveMetaStore: 0: Opening raw store with implemenationclass:org.apache.hadoop.hive.metastore.ObjectStore

17/09/07 19:41:06 INFOmetastore.ObjectStore: ObjectStore, initialize called

17/09/07 19:41:06 INFODataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown- will be ignored

17/09/07 19:41:06 INFODataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will beignored

17/09/07 19:41:07 WARNDataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or oneof dependencies)

17/09/07 19:41:07 WARNDataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or oneof dependencies)

17/09/07 19:41:10 INFOmetastore.ObjectStore: Setting MetaStore object pin classes withhive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

17/09/07 19:41:11 INFODataNucleus.Datastore: The class"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as"embedded-only" so does not have its own datastore table.

17/09/07 19:41:11 INFODataNucleus.Datastore: The class"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as"embedded-only" so does not have its own datastore table.

17/09/07 19:41:13 INFODataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"is tagged as "embedded-only" so does not have its own datastoretable.

17/09/07 19:41:13 INFODataNucleus.Datastore: The class"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as"embedded-only" so does not have its own datastore table.

17/09/07 19:41:13 INFOmetastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY

17/09/07 19:41:13 INFOmetastore.ObjectStore: Initialized ObjectStore

17/09/07 19:41:14 WARNmetastore.ObjectStore: Version information not found in metastore.hive.metastore.schema.verification is not enabled so recording the schemaversion 1.2.0

17/09/07 19:41:14 WARNmetastore.ObjectStore: Failed to get database default, returningNoSuchObjectException

17/09/07 19:41:14 INFO metastore.HiveMetaStore:Added admin role in metastore

17/09/07 19:41:14 INFOmetastore.HiveMetaStore: Added public role in metastore

17/09/07 19:41:15 INFOmetastore.HiveMetaStore: No user is added in admin role, since config is empty

17/09/07 19:41:15 INFOmetastore.HiveMetaStore: 0: get_all_databases

17/09/07 19:41:15 INFOHiveMetaStore.audit: ugi=root   ip=unknown-ip-addr     cmd=get_all_databases

17/09/07 19:41:15 INFOmetastore.HiveMetaStore: 0: get_functions: db=default pat=*

17/09/07 19:41:15 INFOHiveMetaStore.audit: ugi=root   ip=unknown-ip-addr     cmd=get_functions: db=default pat=*

17/09/07 19:41:15 INFODataNucleus.Datastore: The class"org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as"embedded-only" so does not have its own datastore table.

17/09/07 19:41:17 INFOsession.SessionState: Created HDFS directory: /tmp/hive/root

17/09/07 19:41:17 INFOsession.SessionState: Created local directory: /tmp/root

17/09/07 19:41:17 INFOsession.SessionState: Created local directory:/tmp/e25eb236-702d-41b8-b390-45185702d59a_resources

17/09/07 19:41:17 INFOsession.SessionState: Created HDFS directory:/tmp/hive/root/e25eb236-702d-41b8-b390-45185702d59a

17/09/07 19:41:17 INFOsession.SessionState: Created local directory:/tmp/root/e25eb236-702d-41b8-b390-45185702d59a

17/09/07 19:41:17 INFOsession.SessionState: Created HDFS directory:/tmp/hive/root/e25eb236-702d-41b8-b390-45185702d59a/_tmp_space.db

17/09/07 19:41:18 INFO hive.HiveContext:default warehouse location is /user/hive/warehouse

17/09/07 19:41:18 INFO hive.HiveContext:Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.

17/09/07 19:41:18 INFOclient.ClientWrapper: Inspected Hadoop version: 2.6.0

17/09/07 19:41:18 INFOclient.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims forHadoop version 2.6.0

17/09/07 19:41:19 INFOmetastore.HiveMetaStore: 0: Opening raw store with implemenationclass:org.apache.hadoop.hive.metastore.ObjectStore

17/09/07 19:41:19 INFOmetastore.ObjectStore: ObjectStore, initialize called

17/09/07 19:41:19 INFODataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown- will be ignored

17/09/07 19:41:19 INFODataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored

17/09/07 19:41:19 WARNDataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or oneof dependencies)

17/09/07 19:41:20 WARNDataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or oneof dependencies)

17/09/07 19:41:21 INFOmetastore.ObjectStore: Setting MetaStore object pin classes withhive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

17/09/07 19:41:23 INFODataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"is tagged as "embedded-only" so does not have its own datastoretable.

17/09/07 19:41:23 INFODataNucleus.Datastore: The class"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as"embedded-only" so does not have its own datastore table.

17/09/07 19:41:24 INFODataNucleus.Datastore: The class"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as"embedded-only" so does not have its own datastore table.

17/09/07 19:41:24 INFODataNucleus.Datastore: The class"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as"embedded-only" so does not have its own datastore table.

17/09/07 19:41:24 INFOmetastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY

17/09/07 19:41:24 INFO metastore.ObjectStore:Initialized ObjectStore

17/09/07 19:41:25 WARNmetastore.ObjectStore: Version information not found in metastore.hive.metastore.schema.verification is not enabled so recording the schemaversion 1.2.0

17/09/07 19:41:25 WARNmetastore.ObjectStore: Failed to get database default, returningNoSuchObjectException

17/09/07 19:41:25 INFOmetastore.HiveMetaStore: Added admin role in metastore

17/09/07 19:41:25 INFOmetastore.HiveMetaStore: Added public role in metastore

17/09/07 19:41:25 INFOmetastore.HiveMetaStore: No user is added in admin role, since config is empty

17/09/07 19:41:25 INFOmetastore.HiveMetaStore: 0: get_all_databases

17/09/07 19:41:25 INFOHiveMetaStore.audit: ugi=root   ip=unknown-ip-addr     cmd=get_all_databases

17/09/07 19:41:25 INFOmetastore.HiveMetaStore: 0: get_functions: db=default pat=*

17/09/07 19:41:25 INFOHiveMetaStore.audit: ugi=root   ip=unknown-ip-addr     cmd=get_functions: db=default pat=*

17/09/07 19:41:25 INFO DataNucleus.Datastore:The class "org.apache.hadoop.hive.metastore.model.MResourceUri" istagged as "embedded-only" so does not have its own datastore table.

17/09/07 19:41:26 INFOsession.SessionState: Created local directory:/tmp/51d85aba-c051-42a5-81eb-3e5b5f91ebb6_resources

17/09/07 19:41:26 INFOsession.SessionState: Created HDFS directory:/tmp/hive/root/51d85aba-c051-42a5-81eb-3e5b5f91ebb6

17/09/07 19:41:26 INFOsession.SessionState: Created local directory:/tmp/root/51d85aba-c051-42a5-81eb-3e5b5f91ebb6

17/09/07 19:41:26 INFOsession.SessionState: Created HDFS directory:/tmp/hive/root/51d85aba-c051-42a5-81eb-3e5b5f91ebb6/_tmp_space.db

17/09/07 19:41:26 INFO repl.SparkILoop:Created sql context (with Hive support)..

SQL context available as sqlContext.

scala> :q

Stopping spark context.

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/SQL/execution/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/SQL/execution,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/metrics/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/stages/stage/kill,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/executors/threadDump/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/executors/threadDump,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/executors/json,null}

17/09/07 19:56:47 INFO handler.ContextHandler:stopped o.s.j.s.ServletContextHandler{/executors,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/environment/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/storage/rdd/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/storage/rdd,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/storage/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/stages/pool,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/stages/stage/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/stages/stage,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/stages/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stoppedo.s.j.s.ServletContextHandler{/jobs/job/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}

17/09/07 19:56:47 INFOhandler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}

17/09/07 19:56:47 INFO ui.SparkUI:Stopped Spark web UI at http://192.168.52.128:4040

17/09/07 19:56:47 INFOspark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

17/09/07 19:56:48 INFOstorage.MemoryStore: MemoryStore cleared

17/09/07 19:56:48 INFOstorage.BlockManager: BlockManager stopped

17/09/07 19:56:48 INFOstorage.BlockManagerMaster: BlockManagerMaster stopped

17/09/07 19:56:48 INFOscheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!

17/09/07 19:56:48 INFOspark.SparkContext: Successfully stopped SparkContext

17/09/07 19:56:48 INFOremote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.

17/09/07 19:56:48 INFOremote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down;proceeding with flushing remote transports.

17/09/07 19:56:48 INFOremote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.

17/09/07 19:56:48 INFOutil.ShutdownHookManager: Shutdown hook called

17/09/07 19:56:48 INFOutil.ShutdownHookManager: Deleting directory/tmp/spark-f13fa31e-cff8-4504-a43f-5a33562917dd

17/09/07 19:56:48 INFOutil.ShutdownHookManager: Deleting directory/usr/spark-1.6.3-bin-hadoop2.6/spark-468e95b7-945c-4ecf-ad95-3fff574ea2f5

17/09/07 19:56:48 INFOutil.ShutdownHookManager: Deleting directory/tmp/spark-f910557c-1c90-4511-a11e-991499b28bc7

scala> var a = "helloworld!"

a: String = hello world!

scala> :q
至此，整个 Spark 分布式集群的搭建就到这里结束。

 

sakulamartain
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
spark-1.6.3-bin-hadoop2.6.tgz
08-27
spark-1.6.3-bin-hadoop2.6.tgz
spark 1.6 完全分布式平台搭建
weixin_30780649的博客
03-13 82
软件环境： scala-2.11.4.tgz spark-1.6.2-bin-hadoop2.6.tgz 操作步骤： 一. 安装scala 1. 解压scala (tar –zxvf filename) 2. 配置scala环境变量（.bashrc中添加SCALA_HOME, 在path中添加：$SCALA_HOME$/bin） 3...
DataNucleus JDO基础入门
从大数据到人工智能的博客
12-15 837
开发应用程序通常是一项复杂的任务，涉及许多组件。 开发所有这些组件可能非常耗时。 Java 数据对象 API (JDO) 旨在减少花费的一些时间，提供一个 API 以允许 Java 开发人员将面向对象的数据持久化到任何数据库中，并提供一种使用与开发人员相同的 Java 语法的查询语言。 DataNucleus JDO 提供了此 JDO 标准的实现，允许您（用户）将面向对象的数据持久保存到该标准的 RDBMS 数据存储中，而且还保存到范围广泛的其他数据存储中。 其中包括流行的map存储，例如 Cassandr
spark-1.6.0-bin-hadoop2.6.tgz
07-13
linux中搭建spark环境使用的spark-1.6.0-bin-hadoop2.6.tgz安装包
hadoop-2.6.0.tar.gz + spark-1.6.1-bin-hadoop2.6.tgz的集群搭建（单节点）（CentOS系统）
weixin_34318272的博客
03-23 352
                  前言 　　本人呕心沥血所写，经过好一段时间反复锤炼和整理修改。感谢所参考的博友们！同时，欢迎前来查阅赏脸的博友们收藏和转载，附上本人的链接。http://www.cnblogs.com/zlslch/p/5847528.html    关于几个疑问和几处心得！ a.用NAT，还是桥接，还是only-host模式？    答： host...
Hadoop安装教程_单机_伪分布式配置_CentOS6.4_Hadoop2.6
03-05
Hadoop安装教程_单机_伪分布式配置_CentOS6.4_Hadoop2.6
Hadoop伪分布式集群环境搭建
10-14
本篇将详细介绍如何在本地计算机上搭建Hadoop的伪分布式集群环境，这非常适合初学者进行学习和测试。 首先，我们需要了解Hadoop的组成部分。Hadoop主要由两个核心组件构成：HDFS（Hadoop Distributed File System）...
细细品味Hadoop_Hadoop集群CentOS安装配置.rar_hadoop
最新发布
09-24
在“细细品味Hadoop_Hadoop集群（第1期）_CentOS安装配置.pdf”文件中，你可以找到更详细的步骤和截图，以便更好地理解和实践这一过程。通过这个系列的学习，你将具备搭建和管理Hadoop集群的基本技能，为进一步探索...
小坨的Spark分布式集群环境搭建小笔记
01-09
文章目录写在前面搭建好Hadoop集群环境安装Spark（Master节点上操作）配置环境变量（Master节点上操作）Spark配置（Master节点上操作）配置Worker节点启动Spark集群（在Master节点上操作）关闭Spark集群（在Master...
Centos7+hadoop2.6.0分布式集群搭建
01-21
Centos7上搭建hadoop2.6.0分布式系统，教程超详细! 搭建过程中如果有什么疑问可以邮我1668544151@qq.com 单机搭建伪分布式也适用！
spark-1.6.2-bin-hadoop2.6.tgz压缩分卷1
02-05
这个工具包网上找了半天找不到，官网下载非常慢，下载了一夜终于下载到了，但是上传发现有限制，因此把它按分卷来压缩，有多个卷，此卷是卷1，共6卷
Spark standalone模式的安装（spark-1.6.1-bin-hadoop2.6.tgz）（master、slave1和slave2）
weixin_33817333的博客
03-28 200
   前期博客  Spark运行模式概述 Spark standalone简介与运行wordcount（master、slave1和slave2）               开篇要明白 　　（1）spark-env.sh 是环境变量配置文件 　　（2）spark-defaults.conf 　　（3）slaves 是从节点机器配置文件 　　（4）metrics.pr...
Spark1.6.0 on Hadoop-2.6.3 安装配置
yjh314的专栏
07-26 1684
Spark1.6.0 on Hadoop-2.6.3 安装配置 1. 配置hadoop n 下载hadoop mkdir /usr/local/bigdata/hadoop cd  /usr/local/bigdata/hadoop wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.3/hado
高可用spark安装与配置
m0_55769244的博客
10-22 569
（1）解压安装包到/usr/local下。 tar -zxf /opt/spark-1.6.3-bin-hadoop2.6.tgz -C /usr/local （2）进入/usr/local/spark-1.6.3-bin-hadoop2.6/conf。 cd /usr/local/spark-1.6.3-bin-hadoop2.6/conf 3）配置spark-env.sh文件，复制spark-env.sh.template文件,重命名为spark-env.sh
Spark1.6.0 Hadoop2.6.0 单机win7下配置（Intellij IDEA）
Crystal_Zero的博客
11-24 4648
1. 首先需要注意的Scala版本要与Spark相应版本对应 比如Spark1.6.0，官网上给出了依赖 Spark runs on Java 7+, Python 2.6+ and R 3.1+. For the Scala API, Spark 1.6.0 uses Scala 2.10. You will need to use a compatible Scala version (2.
Spark 1.6.2 + Hadoop 2.7.2 集群搭建
小狼无声的博客
07-16 3040
版本信息：Centos7 + Hadoop 2.7.2 + Spark 1.6.2 + Scala 2.11.8 Hadoop + Spark 集群搭建系列文章，建议按顺序参考： Hadoop & Spark 集群搭建 理念思想 Hadoop 2.7.2 集群搭建－预备工作 Hadoop 2.7.2 集群搭建 Spark 1.6.2 + Hadoop 2.7.2 集
spark-2.2.0-bin-hadoop2.6和spark-1.6.1-bin-hadoop2.6发行包自带案例全面详解（java、python、r和scala）之Basic包下的JavaTC.j...
weixin_34262482的博客
08-31 90
        不多说，直接上干货！     spark-1.6.1-bin-hadoop2.6里Basic包下的JavaTC.java     /* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. Se...
Linux搭建Hive On Spark环境(spark-1.6.3-without-hive+hadoop2.8.0+hive2.1.1)
eric_lee的专栏
01-08 2540
概述 1各个机器安装概况 2下载和解压缩各种包 3配置环境变量 4修改hostname文件         4.1在3台机器上执行hostname命令         4.2编辑hosts文件 5 Hadoop的安装和启动         5.1设置ssh免密码登录         5.2 创建一堆目录         5.3
CentOS6.4单机伪分布式安装Hadoop2.6教程
这个Hadoop安装教程提供了一条清晰的路径，让初学者能够在CentOS环境中搭建Hadoop环境，从而进一步学习和实践大数据处理技术。对于那些希望深入了解Hadoop分布式计算和大数据处理的人来说，这是一个很好的起点。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值