spark样例测试

spark on yarn计算平台已经搭建完成,下面来利用这个平台运行下官方的样例,样例全部在/home/work/spark目录下执行

SparkPi

./bin/spark-submit –class org.apache.spark.examples.SparkPi –master yarn –deploy-mode cluster examples/jars/spark-examples*.jar 10

SparkPageRank

首先编写一个文件,内容如下:

1 2
1 3
1 4
2 1
3 1
1 5
2 5

将其添加到hdfs文件系统中hadoop fs -put -R test.txt /user/work/

./bin/spark-submit –class org.apache.spark.examples.SparkPageRank –master yarn –deploy-mode cluster examples/jars/spark-examples*.jar test.txt

计算结果如下

4 has rank: 0.250559973109555.
5 has rank: 0.35684113180533983.
2 has rank: 0.250559973109555.
3 has rank: 0.250559973109555.
1 has rank: 0.46884347608735455.

CustomReceiver

spark01中执行nc -lk 9999

spark00中执行./bin/spark-submit --class org.apache.spark.examples.streaming.CustomReceiver --master yarn --deploy-mode cluster examples/jars/spark-examples*.jar 10.19.145.213 9999