Spark 搭建入门

docker搭建spark

docker pull singularities/spark

docker-compose.yml

version: "2"

services:
  master:
    image: singularities/spark
    command: start-spark master
    hostname: master
    ports:
      - "6066:6066"
      - "7070:7070"
      - "8080:8080"
      - "50070:50070"
  worker:
    image: singularities/spark
    command: start-spark worker master
    environment:
      SPARK_WORKER_CORES: 1
      SPARK_WORKER_MEMORY: 2g
    links:
      - master

参考：https://github.com/SingularitiesCR/spark-docker

执行docker-compose up即可启动一个单工作节点的standlone模式下运行的spark集群

http://localhost:8080/

​​

停止容器docker-compose stop

删除容器docker-compose rm

查看版本

docker exec -it f6734171c8b5 bash