Spark在window环境启动

最新推荐文章于 2024-03-11 16:59:50 发布

manitod

最新推荐文章于 2024-03-11 16:59:50 发布

阅读量248

点赞数 1

分类专栏：大数据文章标签： spark 大数据

本文链接：https://blog.csdn.net/manitod/article/details/135952034

版权

大数据专栏收录该内容

3 篇文章 0 订阅

订阅专栏

Spark在window环境启动时，常用命令总结

windows环境
-- 启动master
spark-class.cmd org.apache.spark.deploy.master.Master

-- 启动worker
spark-class.cmd org.apache.spark.deploy.worker.Worker spark://localhost:7077

-- 启动历史服务器
spark-class.cmd org.apache.spark.deploy.history.HistoryServer

spark-defaults.conf的配置
spark.eventLog.enabled true
spark.eventLog.dir file:///D:/Zpan/spark/log
spark.history.fs.logDirectory file:///D:/Zpan/spark/log

spark-env.cmd的配置
set SPARK_MASTER_HOST=localhost
set JAVA_HOME=%JAVA_HOME%
set HADOOP_HOME=%HADOOP_HOME%


spark-submit.cmd --class org.apache.spark.examples.SparkPi --master local E:\SoftWare\BigData\spark-3.2.3-bin-hadoop3.2\examples\jars\spark-examples_2.12-3.2.3.jar 100

spark-submit --class org.apache.spark.examples.SparkPi --master local E:\SoftWare\BigData\spark-3.2.3-bin-hadoop3.2\examples\jars\spark-examples_2.12-3.2.3.jar 100

spark-submit  --name 新增字段验证 --master local --deploy-mode client  --conf spark.driver.memory=1g --conf spark.executor.instances=1 --conf spark.executor.memory=1g --conf spark.executor.cores=1 --class com.sf.datawrangling.boot.App jars/data-wrangling.jar "{\"lines\":[{\"from\":\"51pfh6d7x6w000\",\"to\":\"1dy2ck98im1s00\"}],\"nodes\":[{\"category\":\"INPUT\",\"collectTimeStrategy\":{\"type\":\"ALL\"},\"connectionInfo\":{\"database\":\"competior\",\"datasourceId\":\"30\",\"host\":\"10.233.32.238\",\"password\":\"Changeme_123\",\"port\":3307,\"type\":\"MYSQL\",\"username\":\"root\"},\"id\":\"51pfh6d7x6w000\",\"items\":[{\"name\":\"t_test_data\",\"sql\":\"(select * from t_test_data ) t_test_data \"}],\"partitioningStrategy\":{\"partitionType\":\"NO_SPECIFIEFD_CONDITIONS\"},\"type\":\"JdbcInputNode\"},{\"category\":\"PROCESSOR\",\"fieldInfos\":[{\"filedName\":\"test\",\"filedType\":\"STRING\",\"filedValue\":\"test\"}],\"id\":\"1dy2ck98im1s00\",\"tableName\":\"\",\"type\":\"AddColumnProcessorNode\"}],\"taskId\":\"195\",\"taskName\":\"新增字段验证\"}"