编写运行脚本
[hadoop@localhost bin]$ cat ./run_sparkPi_on_yarn.sh
export YARN_CONF_DIR=/home/hadoop/hadoop/etc/hadoop
export SPARK_JAR=/home/hadoop/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
spark-class org.apache.spark.deploy.yarn.Client --jar ../examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar
--class org.apache.spark.examples.SparkPi --args yarn-standalone
为其增加执行权限
[hadoop@localhost bin]$ chmod u+x run_sparkPi_on_yarn.sh
[hadoop@localhost bin]$ ll
total 792
-rw-rw-r--. 1 hadoop hadoop 2601 Mar 27 13:44 compute-classpath.cmd
-rwxrwxr-x. 1 hadoop hadoop 3330 Mar 27 13:44 compute-classpath.sh
-rw-rw-r--. 1 hadoop hadoop 5219 Jul 4 10:33 JavaSparkM-0.0.1-SNAPSHOT.jar
-rw-rw-r--. 1 hadoop hadoop 10283 Jul 1 17:23 JavaSparkT.jar
-rwxrwxr-x. 1 hadoop hadoop 2070 Mar 27 13:44 pyspark
-rw-rw-r--. 1 hadoop hadoop 1827 Mar 27 13:44 pyspark2.cmd
-rw-rw-r--. 1 hadoop hadoop 1000 Mar 27 13:44 pyspark.cmd
-rwxrwxr-x. 1 hadoop hadoop 3055 Mar 27 13:44 run-example
-rw-rw-r--. 1 hadoop hadoop 2046 Mar 27 13:44 run-example2.cmd
-rw-rw-r--. 1 hadoop hadoop 1012 Mar 27 13:44 run-example.cmd
-rwxrw-r--. 1 hadoop hadoop 181 May 30 10:35 run_graphx.sh
-rwxrw-r--. 1 hadoop hadoop 2155 Jul 1 17:25 run-java-example
-rwxrw-r--. 1 hadoop hadoop 410 Jul 4 10:42 run_java_wordcount_local.sh
-rwxrw-r--. 1 hadoop hadoop 440 Jul 4 10:39 run_java_wordcount.sh
-rwxrw-r--. 1 hadoop hadoop 352 May 22 11:05 run_sparkPi_on_yarn.sh
-rwxrw-r--. 1 hadoop hadoop 430 May 22 11:06 run_spark_wordcount.sh
-rw-rw-r--. 1 hadoop hadoop 698341 May 23 16:04 spam.data
-rwxrwxr-x. 1 hadoop hadoop 5151 Mar 27 13:44 spark-class
-rwxrwxr-x. 1 hadoop hadoop 3212 Mar 27 13:44 spark-class2.cmd
-rw-rw-r--. 1 hadoop hadoop 1010 Mar 27 13:44 spark-class.cmd
-rwxrwxr-x. 1 hadoop hadoop 3184 Mar 27 13:44 spark-shell
-rwxrwxr-x. 1 hadoop hadoop 941 Mar 27 13:44 spark-shell.cmd
-rw-rw-r--. 1 hadoop hadoop 5231 May 21 15:34 spark-wordcount-scala.jar
[hadoop@localhost bin]$
控制台log如下:
[hadoop@localhost bin]$ ./run_sparkPi_on_yarn.sh
14/07/29 18:03:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/29 18:03:07 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/07/29 18:03:08 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 1
14/07/29 18:03:08 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
queueApplicationCount = 3, queueChildQueueCount = 0
14/07/29 18:03:08 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 8192
14/07/29 18:03:08 INFO yarn.Client: Preparing Local resources
14/07/29 18:03:09 INFO yarn.Client: Uploading file:/home/hadoop/spark-0.9.1-bin-hadoop2/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar to hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1406524413406_0004/spark-examples_2.10-assembly-0.9.1.jar
14/07/29 18:03:10 INFO yarn.Client: Uploading file:/home/hadoop/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar to hdfs://localhost:9000/user/hadoop/.sparkStaging/application_1406524413406_0004/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
14/07/29 18:03:11 INFO yarn.Client: Setting up the launch environment
14/07/29 18:03:11 INFO yarn.Client: Setting up container launch context
14/07/29 18:03:11 INFO yarn.Client: Command for starting the Spark ApplicationMaster: $JAVA_HOME/bin/java -server -Xmx512m -Djava.io.tmpdir=$PWD/tmp org.apache.spark.deploy.yarn.ApplicationMaster --class org.apache.spark.examples.SparkPi --jar ../examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar --args 'yarn-standalone' --worker-memory 1024 --worker-cores 1 --num-workers 2 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
14/07/29 18:03:11 INFO yarn.Client: Submitting application to ASM
14/07/29 18:03:11 INFO impl.YarnClientImpl: Submitted application application_1406524413406_0004 to ResourceManager at /0.0.0.0:8032
14/07/29 18:03:12 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:13 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:14 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:15 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:16 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:17 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:18 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:19 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: N/A
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:20 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:21 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:22 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:23 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:24 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:25 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:26 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:27 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:28 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:29 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:30 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/
appUser: hadoop
14/07/29 18:03:31 INFO yarn.Client: Application report from ASM:
application identifier: application_1406524413406_0004
appId: 4
clientToAMToken: null
appDiagnostics:
appMasterHost: localhost
appQueue: default
appMasterRpcPort: 0
appStartTime: 1406628191709
yarnAppState: FINISHED
distributedFinalState: SUCCEEDED
appTrackingUrl: localhost:8088/proxy/application_1406524413406_0004/A
appUser: hadoop
这个过程中我们可以通过http://localhost:8088/cluster来监视任务的执行过程
这里已经结束了 通过点击applicationid我们可以进入更详细的画面:
通过下方的logs我们可以进入log页面查看log 程序的输出也会来这里给出
这个log是stderr 就是程序运行的log都会通过stderr输入
与之对应的是stdout是程序的执行结果:
这里给出了pi的大概值
Pi is roughly 3.13822同样的这些log可以在hadoop的logs目录中找到
我机器的路径是在
[hadoop@localhost logs]$ cd userlogs/
[hadoop@localhost userlogs]$ ls
application_1403063579463_0001 application_1403080077168_0006 application_1404437711537_0005
application_1403063579463_0002 application_1403080077168_0007 application_1404437711537_0006
application_1403074938465_0001 application_1403080077168_0009 application_1405671242399_0001
application_1403074938465_0002 application_1403080077168_0010 application_1406016955739_0001
application_1403074938465_0003 application_1403080077168_0011 application_1406016955739_0002
application_1403074938465_0004 application_1403080077168_0012 application_1406016955739_0003
application_1403074938465_0005 application_1403135013165_0001 application_1406524413406_0001
application_1403080077168_0001 application_1403136067016_0001 application_1406524413406_0002
application_1403080077168_0002 application_1403136067016_0002 application_1406524413406_0003
application_1403080077168_0003 application_1403136067016_0003 application_1406524413406_0004
application_1403080077168_0004 application_1403136067016_0004
application_1403080077168_0005 application_1404175010388_0001
[hadoop@localhost userlogs]$ pwd
/home/hadoop/hadoop/logs/userlogs
我们的applicationid为
application_1406524413406_0004 所以我们进入这个目录:
[hadoop@localhost application_1406524413406_0004]$ ll
total 12
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000001
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000002
drwx--x---. 2 hadoop hadoop 4096 Jul 29 18:03 container_1406524413406_0004_01_000003
[hadoop@localhost application_1406524413406_0004]$ pwd
/home/hadoop/hadoop/logs/userlogs/application_1406524413406_0004
然后就可以找到对应stdout和stderr了
[hadoop@localhost application_1406524413406_0004]$ cd container_1406524413406_0004_01_000001/
[hadoop@localhost container_1406524413406_0004_01_000001]$ ls
stderr stdout
[hadoop@localhost container_1406524413406_0004_01_000001]$