基于Docker快速在notebook上运行spark命令

最新推荐文章于 2024-10-09 09:22:38 发布

KeepLearningBigData

最新推荐文章于 2024-10-09 09:22:38 发布

阅读量372

点赞数

分类专栏： docker spark 文章标签： docker spark 大数据

本文链接：https://blog.csdn.net/xubo245/article/details/130051261

版权

spark 同时被 2 个专栏收录

70 篇文章 27 订阅

订阅专栏

docker

3 篇文章 0 订阅

订阅专栏

1.下载镜像

docker pull jupyter/all-spark-notebook

2.运行镜像
创建目录：

mkdir -p /Users/xubo/Desktop/xubo/data/notebook:

运行notebook：

 docker run -d -p 8888:8888 -v /Users/xubo/Desktop/xubo/data/notebook:/home/xubo/work --restart always jupyter/all-spark-notebook

记录：

localhost:notebook xubo$ pwd
/Users/xubo/Desktop/xubo/data/notebook
localhost:notebook xubo$ docker run -d -p 8888:8888 -v /Users/xubo/Desktop/xubo/data/notebook:/home/xubo/work --restart always jupyter/all-spark-notebook
afd32f3e9d1d8c919735a10a2ad65b3f2012e942fbab70e80916c4cba62d3648

查找访问地址：

localhost:notebook xubo$ docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED          STATUS                    PORTS                                                 NAMES
afd32f3e9d1d   jupyter/all-spark-notebook   "tini -g -- start-no…"   29 seconds ago   Up 25 seconds (healthy)   4040/tcp, 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp   pensive_colden
ab245d51dce7   apache/spark                 "/opt/entrypoint.sh …"   7 minutes ago    Up 7 minutes                                                                    pedantic_wozniak
localhost:notebook xubo$ docker logs --tail 100   afd32f3e9d1d
Entered start.sh with args: jupyter lab
/usr/local/bin/start.sh: running hooks in /usr/local/bin/before-notebook.d as uid / gid: 1000 / 100
/usr/local/bin/start.sh: running script /usr/local/bin/before-notebook.d/spark-config.sh
/usr/local/bin/start.sh: done running hooks in /usr/local/bin/before-notebook.d
。。。
[I 2023-04-09 17:47:45.184 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-04-09 17:47:45.185 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2023-04-09 17:47:45.185 ServerApp] Jupyter Server 2.5.0 is running at:
[I 2023-04-09 17:47:45.185 ServerApp] http://afd32f3e9d1d:8888/lab?token=b0124468954b3dbd7f036d67e9fbca0184736f867acc0a1c
[I 2023-04-09 17:47:45.185 ServerApp]     http://127.0.0.1:8888/lab?token=b0124468954b3dbd7f036d67e9fbca0184736f867acc0a1c
[I 2023-04-09 17:47:45.185 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2023-04-09 17:47:45.191 ServerApp] 
    
    To access the server, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html
    Or copy and paste one of these URLs:
        http://afd32f3e9d1d:8888/lab?token=b0124468954b3dbd7f036d67e9fbca0184736f867acc0a1c
        http://127.0.0.1:8888/lab?token=b0124468954b3dbd7f036d67e9fbca0184736f867acc0a1c

3.启动Spark

打开notebook：
http://127.0.0.1:8888/lab?token=b0124468954b3dbd7f036d67e9fbca0184736f867acc0a1c
运行启动spark：
启动命令

 spark-shell

记录

(base) jovyan@afd32f3e9d1d:~$ spark-shell 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/04/09 17:53:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://afd32f3e9d1d:4040
Spark context available as 'sc' (master = local[*], app id = local-1681062818647).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.2
      /_/
         
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 17.0.6)
Type in expressions to have them evaluated.
Type :help for more information.

4.运行Spark

scala> spark.range(1000 * 1000 * 1000).count()
res0: Long = 1000000000                                                                                                                        

scala>

参考：
【1】 https://hub.docker.com/r/jupyter/all-spark-notebook