话接上回Spark学习笔记1-编译源代码,已经成功编译了spark on yarn
启动yarn
victor@victor-ubuntu:~/software/hadoop-2.2.0/sbin$ ./start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-namenode-victor-ubuntu.out
localhost: starting datanode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-datanode-victor-ubuntu.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-secondarynamenode-victor-ubuntu.out
starting yarn daemons
starting resourcemanager, logging to /home/victor/software/hadoop-2.2.0/logs/yarn-victor-resourcemanager-victor-ubuntu.out
localhost: starting nodemanager, logging to /home/victor/software/hadoop-2.2.0/logs/yarn-victor-nodemanager-victor-ubuntu.out
victor@victor-ubuntu:~/software/hadoop-2.2.0/sbin$ jps
22896 SecondaryNameNode
23383 Jps
22317 NameNode
22578 DataNode
23078 ResourceManager
23342 NodeManager
运行脚本,准备运行的是spark的计算Pi的例子。先来看下准备条件:1.spark客户端目录,那几个文件夹放编译好的jar包。2. 运行的shell
victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating$ pwd
/home/victor/software/incubator-spark-0.8.1-incubating
victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating/spark_compiled_client$ pwd
/home/victor/software/incubator-spark-0.8.1-incubating/spark_compiled_client
victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating/spark_compiled_client$ ll
total 28
drwxrwxr-x 5 victor victor 4096 3月 28 21:38 ./
drwxrwxr-x 24 victor victor 4096 3月 28 22:47 ../
drwxrwxr-x 3 victor victor 4096 3月 28 02:41 assembly/
drwxrwxr-x 2 victor victor 4096 12月 11 06:35 conf/
drwxrwxr-x 3 victor victor 4096 3月 28 02:42 examples/
-rwxr-xr-x 1 victor victor 4802 12月 11 06:35 spark-class*
#!/bin/sh
export YARN_CONF_DIR=/home/victor/software/hadoop-2.2.0/etc/hadoop
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0.jar \
./spark-class org.apache.spark.deploy.yarn.Client \
--jar ./examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar \
--class org.apache.spark.examples.JavaSparkPi \
--args yarn-standalone \
--num-workers 2 \
--master-memory 400m \
--worker-memory 512m \
--worker-cores 1
我的机器配置比较差,这里设置worker和master都比较保守。。。还是比较卡运行的时候。
执行截图:
<原创,转载请注明出处http://blog.csdn.net/oopsoom/article/details/22419597>