spark Java local测试,前天装了个spark，发个安装笔记，经本人亲自测试成功

该楼层疑似违规已被系统折叠 隐藏此楼查看此楼

环境：ubuntu17.10.1

jdkSE8 (测试9不行，坑啊)

hadoop 2.9.0

scala 2.10.7

spark 2.3.0

经测试成功

2018.04.10

允许所有用户访问显示器：

xhost +

编辑profile

sudo gedit /etc/profile

启动spark

./sbin/start-all.sh

看监听端口

netstat -tnpl

配置环境变量

sudo gedit /etc/profile

source /etc/profile

一般软件安装

sudo tar -zxf ~/下载/spark-1.6.0-bin-without-hadoop.tgz -C /usr/local/

sudo mv ./spark-1.6.0-bin-without-hadoop/ ./spark

sudo chown -R hadoop:hadoop ./spark

vbox增强不能运行:

sudo apt-get install gcc make perl

再装一次，重启

别忘了vbox常规选项，高级，打开双向剪贴功能

===========安装计划===========

java/hadoop/scala/spark全部安装在/usr/local/目录下

到各自网站下载安装包然后解压改名

===========java安装===========

java需要配置的环境变量

export JAVA_HOME=/usr/local/java8

export JRE_HOME=${JAVA_HOME}/jre

export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib

export PATH=${JAVA_HOME}/bin:$PATH

===========ssh安装===========

是否启动：ps -e | grep ssh

安装：sudo apt-get install openssh-server

启动：sudo /etc/init.d/ssh start

先登录并退出：ssh localhost

进入ssh目录：cd ~/.ssh/

产生秘钥：ssh-keygen -t rsa

加入授权：cat ./id_rsa.pub >> ./authorized_keys

登录测试不输入密码则为正确：ssh localhost

===========hadoop安装===========

配置三个文件：hadoop目录/etc/hadoop/

sudo gedit ./etc/hadoop/core-site.xml

fs.default.name

hdfs://localhost:9000

hadoop.tmp.dir

/home/tam/temp

/home/tam/temp没有则创建

sudo gedit ./etc/hadoop/hdfs-site.xml

dfs.replication

1

sudo gedit ./etc/hadoop/hadoop-env.sh

# The java implementation to use.

export JAVA_HOME=/usr/local/java8

export HADOOP_HOME=/usr/local/hadoop

export PATH=$PATH:/usr/local/hadoop/bin

系统环境加：

#hadoop

export PATH=/usr/local/hadoop/bin:/usr/local/hadoop/sbin:$PATH

重启

格式化：hadoop namenode -format

启动服务：/usr/local/hadoop/sbin/start-all.sh

看看：jps

停止服务：/usr/local/hadoop/sbin/stop-all.sh

===========spark安装===========

配置环境变量：

sudo gedit /etc/profile

#spark

export SPARK_HOME=/usr/local/spark

export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

source /etc/profile

生成env文件：cp conf/spark-env.sh.template conf/spark-env.sh

修改之：sudo gedit conf/spark-env.sh

export JAVA_HOME=/usr/local/java8

export SCALA_HOME=/usr/local/scala

export HADOOP_HOME=/usr/local/hadoop

#hadoop集群的配置文件的目录

export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop

#spark集群的Master节点的ip地址

export SPARK_MASTER_IP=192.168.100.109

#每个worker节点能够最大分配给exectors的内存大小

export SPARK_WORKER_MEMORY=1g

#每个worker节点所占有的CPU核数目

export SPARK_WORKER_CORES=1

#每台机器上开启的worker节点的数目

export SPARK_WORKER_INSTANCES=1

生成默认slaves文件：cp conf/slaves.template conf/slaves

修改slave文件：sudo gedit conf/slaves

启动spark：/usr/local/spark/sbin/start-all.sh

启动spark-shell：/usr/local/spark/bin/spark-shell