1.产生数据:
./bin/spark-submit \
--class org.apache.spark.examples.TeraGen \
--master spark://centos1:7077 \
/usr/local/spark-2.2.1/examples/jars/spark-examples_2.11-2.3.3.jar \
1g /terasort/spark/1G-input
2.检查是否产生数据:
hdfs dfs -ls /terasort/spark
3.使用terasort排序:
./bin/spark-submit \
--class org.apache.spark.examples.TeraSort \
--master spark://centos1:7077 \
/usr/local/spark-2.2.1/examples/jars/spark-examples_2.11-2.3.3.jar \
/terasort/spark/1G-input /terasort//spark/1G-output
4.检查是否产生排序数据:
hdfs dfs -ls /terasort/spark
5.使用TeraValidate 进行结果验证:
./bin/spark-submit \
--class org.apache.spark.examples.TeraValidate \
--master spark://centos1:7077 \
/usr/local/spark-2.2.1/examples/jars/spark-examples_2.11-2.3.3.jar \
/terasort/spark/1G-output
spark terasort 排序
最新推荐文章于 2024-06-22 09:38:45 发布