spark 2.4 资源
下载地址
https://www.apache.org/dyn/closer.lua/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
文档地址
http://spark.apache.org/docs/latest/index.html
我的本地电脑是win10, 启用了ubuntun 系统, 所以下载后,将文件移动至Linux系统
下载后放
/root/spark-2.4.0-bin-hadoop2.7.tgz
解压
tar zxvf spark-2.4.0-bin-hadoop2.7.tgz
启动脚本目录
/root/spark-2.4.0-bin-hadoop2.7/sbin
主要的脚本有下面几个,只要把配置文件改好, 最常用的是start-all.sh, stop-all.sh
sbin/start-master.sh - Starts a master instance on the machine the script is executed on.
sbin/start-slaves.sh - Starts a slave instance on each machine specified in the conf/slaves file.
sbin/start-slave.sh - Starts a slave instance on the machine the script is executed on.
sbin/start-all.sh - Starts both a master and a number of slaves as described above.
sbin/stop-master.sh - Stops the master that was started via the sbin/start-master.sh script.
sbin/stop-slaves.sh - Stops all slave instances on the machines specified in the conf/slaves file.
sbin/stop-all.sh - Stops both the master and the slaves as described above.
配置文件修改 目录 /root/spark-2.4.0-bin-hadoop2.7/conf
复制一份配置模板
cp spark-env.sh.template spark-env.sh
这个配置文件可以配置的参数如下, 本地运行可以保持不修改任何参数
Environment Variable Meaning
SPARK_MASTER_HOST Bind the master to a specific hostname or IP address, for example a public one.
SPARK_MASTER_PORT Start the master on a different port (default: 7077).
SPARK_MASTER_WEBUI_PORT Port for the master web UI (default: 8080).
SPARK_MASTER_OPTS Configuration properties that apply only to the master in the form "-Dx=y" (default: none). See below for a list of possible options.
SPARK_LOCAL_DIRS Directory to use for "scratch" space in Spark, including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks.
SPARK_WORKER_CORES Total number of cores to allow Spark applications to use on the machine (default: all available cores).
SPARK_WORKER_MEMORY Total amount of memory to allow Spark applications to use on the machine, e.g. 1000m, 2g (default: total memory minus 1 GB); note that each application's individual memory is configured using its spark.executor.memory property.
SPARK_WORKER_PORT Start the Spark worker on a specific port (default: random).
SPARK_WORKER_WEBUI_PORT Port for the worker web UI (default: 8081).
SPARK_WORKER_DIR Directory to run applications in, which will include both logs and scratch space (default: SPARK_HOME/work).
SPARK_WORKER_OPTS Configuration properties that apply only to the worker in the form "-Dx=y" (default: none). See below for a list of possible options.
SPARK_DAEMON_MEMORY Memory to allocate to the Spark master and worker daemons themselves (default: 1g).
SPARK_DAEMON_JAVA_OPTS JVM options for the Spark master and worker daemons themselves in the form "-Dx=y" (default: none).
SPARK_DAEMON_CLASSPATH Classpath for the Spark master and worker daemons themselves (default: none).
SPARK_PUBLIC_DNS The public DNS name of the Spark master and workers (default: none).
启动 master
./sbin/start-master.sh
启动worker
./sbin/start-slave.sh spark://hostname:7077
hostname 换成自己的