在vmare上安装五台centos6.5的linux,两台用着spark的master节点,三台用做spark的worker节点,用zookeeper来配置spark的ha,两台master一台是alive,另一台是standby。spark的安装部署相当简单(这里不做介绍),配置ha是根据官网的配置如下:
Configuration
In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spark-env using this configuration:
System property | Meaning |
---|---|
spark.deploy.recoveryMode |
Set to ZOOKEEPER to enable standby Master recovery mode (default: NONE). |
spark.deploy.zookeeper.url |
The ZooKeeper cluster url (e.g., 192.168.1.100:2181,192.168.1.101:2181). |
spark.deploy.zookeeper.dir |
The directory in ZooKeeper to store recovery state (default: /spark). |
Possible gotcha: If you have multiple Masters in your cluster but fail to correctly configure the Masters to use ZooKeeper, the Masters will fail to discover each other and think they’re all leaders. This will not lead to a healthy cluster state (as all Masters will schedule independently).
配置好后,主备切换可以成功!
在openstack上安装五台centos6.6的linux,两台用着spark的master节点,三台用做spark的worker节点,用zookeeper来配置spark的ha,两台master一台是alive,另一台是standby。spark的安装部署相当简单(这里不做介绍),配置ha是根据官网的配置如下:
Configuration