[Spark | Yarn | Hadoop] Spark Submit over Yarn

I use pre built package of spark 1.0.2 for Hadoop 2.4.1

edit conf/spark-env.sh

export HADOOP_CONF_DIR="/apache/hadoop/conf"
export YARN_CONF_DIR="/apache/hadoop/conf"
export SPARK_LIBRARY_PATH="/apache/hadoop/lib/native"
export SPARK_YARN_USER_ENV="CLASSPATH=/apache/hadoop/conf"

Submit a Pi Sample by the following command:

bin/spark-submit \
--master "yarn" \
--deploy-mode "cluster" \
--class "org.apache.spark.examples.SparkPi" \
--name "spark pi" \
--driver-memory "512m" \
--driver-library-path "/apache/hadoop/lib/native:/apache/hadoop/lib/native/Linux-amd64-64" \
--driver-class-path "/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-XXXX-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop/share/hadoop/common/lib/hadoop-<span style="font-family: Arial, Helvetica, sans-serif;">XXXX</span>-0.1-<span style="font-family: Arial, Helvetica, sans-serif;">XXXX</span>-2.jar" \
--jars "/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-XXXX-2.jar,/apache/hadoop/lib/hadoop-lzo-0.6.0.jar,/apache/hadoop/share/hadoop/common/lib/hadoop-XXXX-0.1-XXXX-2.jar" \
--executor-memory "512m" \
--verbose \
--executor-cores 1 \
--queue "queue-spark" \
--num-executors 3 \
"lib/spark-examples-1.0.2-hadoop2.4.1.jar"

Look at the detailed log for the job:

yarn logs -applicationId application_1412632629877_11612

Below is the job submission template:

bin/spark-submit \
    --master "yarn" \
    --deploy-mode "cluster" \
    --class <my-class> \
    --driver-library-path "/apache/hadoop/lib/native:/apache/hadoop/lib/native/Linux-amd64-64" \
<pre name="code" class="php">--driver-class-path "/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-XXXX-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop/share/hadoop/common/lib/hadoop-<span style="font-family: Arial, Helvetica, sans-serif;">XXXX</span>-0.1-<span style="font-family: Arial, Helvetica, sans-serif;">XXXX</span>-2.jar" \
--jars "/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-XXXX-2.jar,/apache/hadoop/lib/hadoop-lzo-0.6.0.jar,/apache/hadoop/share/hadoop/common/lib/hadoop-XXXX-0.1-XXXX-2.jar" \

--queue "queue-spark" \ <job jar> \ <main arguments>