spark常用命令总结

spark命令总结：

1. yarn application -list ： 列出所有正在执行的任务。
2. spark-submit 提交任务任务。以及各参数的含义。
   spark-submit \
   –master yarn \
   –jars required_jars/spark-streaming-kafka-0-8-assembly_2.11-2.4.3.jar \
   –deploy-mode cluster \
   –num-executors 1 \
   –executor-memory 1G \
   –driver-memory 1G \
   –executor-cores 4 \
   –queue root.algorithm \
   –conf spark.streaming.backpressure.initialRate=10 \
   –conf spark.executor.extraJavaOptions=" -Dfile.encoding=utf-8 " \
   –archives hdfs:///user/algorithm-dev/leo/tools/anaconda0904.zip#anaconda0904 \
   –conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./anaconda0904/anaconda3/bin/python3 \
   –py-files your_project_source_file.zip \
   main.py \
   pro