spark 常用脚本

2019独角兽企业重金招聘Python工程师标准>>>

spark 常用脚本 博客分类： spark

# Run on a Spark standalone cluster
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://207.184.161.138:7077 \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \

 

 

spark-class org.apache.spark.deploy.master.Master

 

spark-class  org.apache.spark.deploy.worker.Worker  spark://9YCPFQ5M05Y29A8:7077

 

spark-shell --master spark://9YCPFQ5M05Y29A8:7077

 

spark-submit --class spark.JavaWordCount --master spark://9YCPFQ5M05Y29A8:7077 E:\wordCount.jar

 

 

spark-class  org.apache.spark.deploy.worker.Worker  spark://Lenovo-PC:7077

spark-submit --class spark.JavaWordCount --master spark://Lenovo-PC:7077   --executor-memory 512m  --total-executor-cores 6  E:\wordcount.jar

spark-class  org.apache.spark.deploy.worker.Worker  spark://Lenovo-PC:7077 --cores 2 --memory 800m

spark-class  org.apache.spark.deploy.worker.Worker  spark://Lenovo-PC:7077 --cores 2 --memory 800m

spark-shell --master spark://9YCPFQ5M05Y29A8:7077  --executor-memory 2000m  --total-executor-cores 5

 

 

转载于:https://my.oschina.net/xiaominmin/blog/1597247