pyspark streaming部署

/usr/hdp/2.6.2.0-205/spark-2.3.0-bin-hadoop2.7/bin/spark-submit --master yarn --deploy-mode cluster --executor-cores 1 --executor-memory 1608M --num-executors 2 --driver-cores 1 --driver-memory 1608M --archives hdfs:///python/py3env_pick.zip --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./py3env_pick.zip/py3/bin/python --conf spark.executorEnv.PYSPARK_PYTHON=./py3env_pick.zip/py3/bin/python /home/yuezhiqian/DirectedKakfaSparkStreaming.py dev

1.用户异常，切换为root用户
2.包缺失
kafka-python py4j python-snappy
3 java kafka包缺失
spark-streaming-kafka-0-8-assembly_2.11-2.3.1

pip 不能使用
python -m ensurepip

python-snappy报错gcc找不到
sudo -y yum install snappy-devel