学习spark之基础环境搭建

1前期准备工作

1.1设置机器名：

使用vim /etc/sysconfig/network设置该机器名

NETWORKING=yes
HOSTNAME=hadoop1

1.2关闭selinux vim /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 

1.3修改ssh配置文件vim /etc/ssh/sshd_config 为了可以以ssh的方式登录

 47 RSAAuthentication yes
 48 PubkeyAuthentication yes
 49 AuthorizedKeysFile  .ssh/authorized_keys
 50 #AuthorizedKeysCommand none
 51 #AuthorizedKeysCommandRunAs nobody

1.4添加hadoop用户组以及用户，设置密码

groupadd -g 1000 hadoop
useradd -u 2000 -g hadoop hadoop

passwd hadoop

1.5jdk安装

注意 使用jdk8会在编译hadoop的过程中出现错误，务必使用jdk7或jdk6.

jdk7下载地址：http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html#jdk-7u80-oth-JPR

下载jdk7安装包并解压至目录/usr/local/sinasrv2/java/

配置环境变量，vim ~/.bash_profile：

export SINASRV=/usr/local/sinasrv2
export JAVA_HOME=$SINASRV/java/jdk1.7.0_80
export PATH=$JAVA_HOME/bin:$HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

使之生效：

source ~/.bash_profile 

2.安装Scala

2.1下载Scala安装包

下载地址：http://www.scala-lang.org/download/，本文所用的版本为2.11.8

2.2解压及配置

tar -xvf scala-2.11.8.tgz  -C /usr/local/sinasrv2/scala/

配置环境变量，vim ~/.bash_profile：

export SCALA_HOME=$SINASRV/scala/scala-2.11.8
export PATH=$SCALA_HOME/bin:$JAVA_HOME/bin:$HOME/bin:$PATH

使之生效：

source ~/.bash_profile 

3.配置ssh无密码登录

ssh-keygen -t rsa
cd ~/.ssh
cat id_rsa.pub >> authorized_keys
chmod 400 authorized_keys

4.答疑解惑

4.1 从hadoop用户chmod时提示：hadoop is not in the sudoers file.  This incident will be reported

解决：使用su命令进入root用户，chmod u+w /etc/sudoers，vim /etc/sudoers 添加hadoop ALL=(ALL) ALL