VM Environment Setup

VM Environment Setup

• Add Hadoop User and Groups
• Install and config JDK
• Install and config Scala
• Configure SSH logon without password
• Show [user@hostname ~] when su user

Add Hadoop User and Groups

Execute the following commands

mkdir -p /myhome/hadoop
groupadd -g 1000 hadoop
useradd -u 2000 -g hadoop -d /myhome/hadoop hadoop
chown -R hadoop:hadoop /myhome/hadoop
passwd hadoop

Check the group and user are correctly created

tail /etc/group
tail /etc/passwd
groups hadoop
id hadoop

Edit the home directroy of user

usermod -d /myhome/hadoop hadoop

Prepare the app and upload directories for hadoop

mkdir -p /app/hadoop
chown -R hadoop:hadoop /app/hadoop
mkdir /myhome/hadoop/upload
chown -R hadoop:hadoop /myhome/hadoop/upload

Grant sudo permission to hadoop

chmod u+w /etc/sudoers

Edit /etc/sudoers to add the following lines

hadoop  ALL=(ALL)       NOPASSWD: ALL

Install and config JDK

Add the following environment under /etc/profile

 export JAVA_HOME=/pcc/app/Linux_jdk1.7.0_x86_64
 export PATH=$JAVA_HOME/bin:$PATH
 export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

Verify after re-logon

 java -version

Install and config Scala

wget http://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz /myhome/hadoop/upload
wget http://downloads.lightbend.com/scala/2.10.6/scala-2.10.6.tgz /myhome/hadoop/upload
tar -zxvf scala-2.10.6.tgz -C /app

Edit /etc/profile with SCALA configuration

export SCALA_HOME=/app/scala-2.10.6
export PATH=$PATH:${SCALA_HOME}/bin

Verify after re-logon

scala -version

Configure SSH logon without password

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub user@server_name

Show [user@hostname ~] when su user

 cp /root/.bashrc /myhome/hadoop/
 [hadoop@qilibjtst3 ~]$ cat .bashrc
 # .bashrc

 # User specific aliases and functions

 alias rm='rm -i'
 alias cp='cp -i'
 alias mv='mv -i'

 # Source global definitions
 if [ -f /etc/bashrc ]; then
         . /etc/bashrc

 fi

 cp /root/.bash_profile /myhome/hadoop/
 [hadoop@qilibjtst3 ~]$ cat .bash_profile
 # .bash_profile

 # Get the aliases and functions
 if [ -f ~/.bashrc ]; then
         . ~/.bashrc
 fi

 # User specific environment and startup programs

 PATH=$PATH:$HOME/bin

 export PATH

Change the file permission after copy done

chown hadoop:hadoop /myhome/hadoop/.*