spark 启动

总用量 0

[wangshumin@CentOSNode1 logs]$ kill -9 32377
[wangshumin@CentOSNode1 logs]$ jps
29777 DFSZKFailoverController
29362 NameNode
29075 JournalNode
32547 Jps
31837 ResourceManager
[wangshumin@CentOSNode1 logs]$ start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/logs/spark-wangshumin-org.apache.spark.deploy.master.Master-1-CentOSNode1.out
[wangshumin@CentOSNode1 logs]$ ll
总用量 4
-rw-rw-r--. 1 wangshumin wangshumin 1534 3月  11 17:56 spark-wangshumin-org.apache.spark.deploy.master.Master-1-CentOSNode1.out
[wangshumin@CentOSNode1 logs]$ cat spark-wangshumin-org.apache.spark.deploy.master.Master-1-CentOSNode1.out 
Spark Command: /home/wangshumin/ekl/jdk1.8.0_162/bin/java -cp /home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/conf/:/home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar:/home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/wangshumin/hadoop-2.6.5/etc/hadoop/ -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip CentOSNode1 --port 7077 --webui-port 8080
========================================
18/03/11 17:56:08 INFO master.Master: Registered signal handlers for [TERM, HUP, INT]
18/03/11 17:56:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/03/11 17:56:10 INFO spark.SecurityManager: Changing view acls to: wangshumin
18/03/11 17:56:10 INFO spark.SecurityManager: Changing modify acls to: wangshumin
18/03/11 17:56:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(wangshumin); users with modify permissions: Set(wangshumin)
18/03/11 17:56:12 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
18/03/11 17:56:12 INFO master.Master: Starting Spark master at spark://CentOSNode1:7077
18/03/11 17:56:12 INFO master.Master: Running Spark version 1.6.2
18/03/11 17:56:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
18/03/11 17:56:12 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080
18/03/11 17:56:12 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.
18/03/11 17:56:12 INFO ui.MasterWebUI: Started MasterWebUI at http://192.168.72.141:8080
18/03/11 17:56:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
18/03/11 17:56:12 INFO server.AbstractConnector: Started SelectChannelConnector@CentOSNode1:6066
18/03/11 17:56:12 INFO util.Utils: Successfully started service on port 6066.
18/03/11 17:56:12 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
18/03/11 17:56:13 INFO master.Master: I have been elected leader! New state: ALIVE
[wangshumin@CentOSNode1 logs]$ 

drwxr-xr-x. 13 wangshumin wangshumin 222 3月  11 09:36 spark-1.6.2-bin-hadoop2.6

[wangshumin@CentOSNode3 spark]$ cd spark-1.6.2-bin-hadoop2.6/
[wangshumin@CentOSNode3 spark-1.6.2-bin-hadoop2.6]$ ll
总用量 1384
drwxr-xr-x. 2 wangshumin wangshumin    4096 3月  11 09:12 bin
-rw-r--r--. 1 wangshumin wangshumin 1343562 3月  11 09:12 CHANGES.txt
drwxr-xr-x. 2 wangshumin wangshumin     264 3月  11 13:15 conf
drwxr-xr-x. 3 wangshumin wangshumin      19 3月  11 09:12 data
drwxr-xr-x. 3 wangshumin wangshumin      79 3月  11 09:12 ec2
drwxr-xr-x. 3 wangshumin wangshumin      17 3月  11 09:12 examples
drwxr-xr-x. 2 wangshumin wangshumin     237 3月  11 09:16 lib
-rw-r--r--. 1 wangshumin wangshumin   17352 3月  11 09:12 LICENSE
drwxr-xr-x. 2 wangshumin wangshumin    4096 3月  11 09:12 licenses
drwxrwxr-x. 2 wangshumin wangshumin    4096 3月  11 12:11 logs
-rw-r--r--. 1 wangshumin wangshumin   23529 3月  11 09:12 NOTICE
drwxr-xr-x. 6 wangshumin wangshumin     119 3月  11 09:12 python
drwxr-xr-x. 3 wangshumin wangshumin      17 3月  11 09:12 R
-rw-r--r--. 1 wangshumin wangshumin    3359 3月  11 09:16 README.md
-rw-r--r--. 1 wangshumin wangshumin     120 3月  11 09:12 RELEASE
drwxr-xr-x. 2 wangshumin wangshumin    4096 3月  11 09:12 sbin
[wangshumin@CentOSNode3 spark-1.6.2-bin-hadoop2.6]$ ./sbin/start-slave.sh   spark://CentOSNode1:7077
starting org.apache.spark.deploy.worker.Worker, logging to /home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/logs/spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out
failed to launch org.apache.spark.deploy.worker.Worker:
full log in /home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/logs/spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out
[wangshumin@CentOSNode3 spark-1.6.2-bin-hadoop2.6]$ cat  /home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/logs/spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out
Spark Command: /home/wangshumin/ekl/jdk1.8.0_162/bin/java -cp /home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/conf/:/home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar:/home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/wangshumin/hadoop-2.6.5/etc/hadoop/ -Xms1g -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://CentOSNode1:7077
========================================
[wangshumin@CentOSNode3 spark-1.6.2-bin-hadoop2.6]$ cd   /home/wangshumin/spark/spark-1.6.2-bin-hadoop2.6/logs
[wangshumin@CentOSNode3 logs]$ ll
总用量 24
-rw-rw-r--. 1 wangshumin wangshumin 1225 3月  11 17:58 spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out
-rw-rw-r--. 1 wangshumin wangshumin 2435 3月  11 12:11 spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out.1
-rw-rw-r--. 1 wangshumin wangshumin 2439 3月  11 11:44 spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out.2
-rw-rw-r--. 1 wangshumin wangshumin 2439 3月  11 10:26 spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out.3
-rw-rw-r--. 1 wangshumin wangshumin 2439 3月  11 10:19 spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out.4
-rw-rw-r--. 1 wangshumin wangshumin 2439 3月  11 09:55 spark-wangshumin-org.apache.spark.deploy.worker.Worker-1-CentOSNode3.out.5
[wangshumin@CentOSNode3 logs]$