tensorflowonspark集群提交命令

最新推荐文章于 2024-03-21 09:58:28 发布

流光追不及

最新推荐文章于 2024-03-21 09:58:28 发布

阅读量180

点赞数

文章标签：深度学习 tensorflow 机器学习数据挖掘

本文链接：https://blog.csdn.net/zwdcdut/article/details/115260201

版权

tensorflowonspark集群提交命令

本地调试模型训练
spark-submit --master yarn
–deploy-mode cluster
–queue default
–executor-cores 1
–executor-memory 600M
–conf spark.yarn.executor.memoryOverhead=1500
–archives hdfs://node1:9000/user/xxxxx/py36.zip#Python
–conf spark.executorEnv.LD_LIBRARY_PATH="/sdb/hadoop-3.3.0/lib/native:$JAVA_HOME/jre/lib/amd64/server"
–conf spark.pyspark.python=./Python/py36/bin/python
–conf spark.pyspark.driver.python=./Python/py36/bin/python
–conf spark.yarn.maxAppAttempts=1
–conf spark.yarn.executor.memoryOverhead=256
–conf spark.task.cpus=1
–conf spark.cores.max=4
mnist_spark.py
–images_labels hdfs://node1:9000/user/xxxxx/data/train.csv
–cluster_size 2
–model_dir hdfs://node1:9000/user/xxxxx/model
–epochs 1
–mode train

spark-submit --master yarn
–deploy-mode cluster
–queue default
–executor-cores 1
–executor-memory 600M
–conf spark.yarn.executor.memoryOverhead=1500
–archives hdfs://node1:9000/user/xxxxx/py36.zip#Python
–conf spark.executorEnv.LD_LIBRARY_PATH="/sdb/hadoop-3.3.0/lib/native:$JAVA_HOME/jre/lib/amd64/server"
–conf spark.pyspark.python=./Python/py36/bin/python
–conf spark.pyspark.driver.python=./Python/py36/bin/python
–conf spark.yarn.maxAppAttempts=1
–conf spark.yarn.executor.memoryOverhead=256
–conf spark.task.cpus=1
–conf spark.cores.max=4
mnist_spark.py
–batch_size 1
–images_labels hdfs://node1:9000/user/xxxxx/data/train.csv
–cluster_size 2
–model_dir hdfs://node1:9000/user/xxxxx/model
–epochs 1
–mode predict
–output hdfs://node1:9000/user/xxxxx/pred_all
–output1 hdfs://node1:9000/user/xxxxx/predict

保存本地文件
spark-submit --master yarn
–deploy-mode cluster
–queue default
–executor-cores 1
–executor-memory 600M
–conf spark.executorEnv.LD_LIBRARY_PATH="/sdb/hadoop-3.3.0/lib/native:$JAVA_HOME/jre/lib/amd64/server"
–conf spark.pyspark.python=/sdb/tf/py36/bin/python
–conf spark.pyspark.driver.python=/sdb/tf/py36/bin/python
–conf spark.yarn.maxAppAttempts=1
–conf spark.yarn.executor.memoryOverhead=256
–conf spark.task.cpus=1
–conf spark.cores.max=4
mnist_spark.py
–batch_size 1
–images_labels hdfs://node1:9000/user/xxxxx/data/train1.csv
–cluster_size 2
–model_dir hdfs://node1:9000/user/xxxxx/model
–epochs 1
–mode predict
–output /sdb/tf/pred
–output1 hdfs://node1:9000/user/xxxxx/predict