Spark之Ubuntu18上安装Spark单机环境

最新推荐文章于 2023-02-08 01:39:04 发布

weixin_34126215

最新推荐文章于 2023-02-08 01:39:04 发布

阅读量249

点赞数

文章标签：大数据运维 scala

原文链接：https://my.oschina.net/u/2518341/blog/3019889

版权

2019独角兽企业重金招聘Python工程师标准>>>

安装JDK，步骤略，

List-1

mjduan@mjduan-ubuntu:~$ java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

安装Scala，下载地址

List-2

...

#scala
export SCALA_HOME=/opt/software/tool/scala2.12
export PATH=$SCALA_HOME/bin:$PATH

List-3

mjduan@mjduan-ubuntu:~$ source ~/.bashrc 
mjduan@mjduan-ubuntu:~$ scala -version
Scala code runner version 2.12.8 -- Copyright 2002-2018, LAMP/EPFL and Lightbend, Inc.

安装Spark，下载地址

List-4

mjduan@mjduan-ubuntu:~$ tail -f ~/.bashrc 
......

#scala
export SCALA_HOME=/opt/software/tool/scala2.12
export PATH=$SCALA_HOME/bin:$PATH

#spark
export SPARK_HOME=/opt/software/tool/spark
export PATH=$SPARK_HOME/bin:$PATH

List-5 执行spark-shell可以看到命令行

mjduan@mjduan-ubuntu:~$ source ~/.bashrc 
mjduan@mjduan-ubuntu:~$ spark-shell 
2019-03-08 18:02:36 WARN  Utils:66 - Your hostname, mjduan-ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.43.214 instead (on interface wlp2s0)
2019-03-08 18:02:36 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2019-03-08 18:02:37 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.43.214:4040
Spark context available as 'sc' (master = local[*], app id = local-1552039363367).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

在浏览器中输入http://192.168.43.214:4040就可看到UI界面了。

转载于:https://my.oschina.net/u/2518341/blog/3019889