Setting up SSH for a Hadoop cluster

Setting up SSH for a Hadoop cluster

When setting up aHadoop cluster, you’ll need to designate one specific node as the master node.We need to define a means for the master node to remotely access every node inyour cluster. Hadoop uses pasphraseless SSH for this purpose. SSH utilizesstandard public key cryptography to create a pair of keys for user verification– one public, one private. The public key is stored locally on every node inthe cluster, and the master node sends the private key when attempting toaccess a remote machine. With both pieces of information, the target machinecan validate the login attempt.

We’ve been speakingin general terms of one node accessing another; more precisely this access isfrom a user account on one node to another user account on the target machine.For Hadoop, the accounts should have the same username on all of the nodes, andfor security purpose we recommend it being a user-level account. This accountis only for managing your hadoop cluster. Once the Cluster daemons are up andrunning, you’ll be able to run your actual MapReduce jobs from other accounts.

Check whether SSHis installed on your nodes, if not please install OpenSSH via a Linux packagemanager or by downloading the source directly.

$ which ssh

/usr/bin/ssh

$ which ssd

/usr/bin/which: nossd in(/usr/kerberos/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin)

$ which ssh-keygen

/usr/bin/ssh-keygen

Having verifiedthat SSH is correctly installed on all nodes of the cluster, We use ssh-keygenon the master node to generate an RSA key pair. Be certain to avoid entering apassphrase, or you’ll have to manually enter that phrase every time the masternode attempts to access another node.

$ ssh-keygen -t rsa

Generating 2048-bitrsa key pair

4 .oOo.oOo.oOO

Key generated.

2048-bit rsa,yj70978@retailvm1d, Mon Jul 22 2013 21:06:38 -0400

Passphrase :

Again :

Key is stored withNULL passphrase.

(You can ignore the following warning if youare generating hostkeys.)

This is not recommended.

Don't do this unless you know what you'redoing.

If file system protections fail (someone canaccess the keyfile),

or if the super-user is malicious, your keycan be used without

the deciphering effort.

Private key savedto /home/yj70978/.ssh2/id_rsa_2048_a

Public key saved to/home/yj70978/.ssh2/id_rsa_2048_a.pub

$ ls -rlt/home/yj70978/.ssh2/id_rsa_2048_a

-rw------- 1yj70978 retailfi 1539 Jul 22 21:07 /home/yj70978/.ssh2/id_rsa_2048_a

$ cat /home/yj70978/.ssh2/id_rsa_2048_a

---- BEGIN SSH2ENCRYPTED PRIVATE KEY ----

Subject: yj70978

Comment:"2048-bit rsa, yj70978@retailvm1d, Mon Jul 22 2013 21:06:38 -\

0400"

P2/56wAAA+wAAAA1aWYtbW9kbntzaWdue3JzYS1wa2NzMS1zaGExfSxlbmNyeXB0e3JzYS

1wa2NzMS1ub25lfX0AAAAEbm9uZQAAA58AAAObAAAAEQEAAQAAB/oC01cYvh8cNTVtBELB

azF4B2MT2tpf0MWvA0W2KEho5G3O/TnjOocBRpdRn/yDLJieqinxzBpX9rYumP0/So3FIF

VUUd0TyLEj9CikIjVQWj0KUvNKdjsDzujiWOEpfe0vUMkqosc1GUsdIDXaBAA/LIAuTQjF

dlsfYp/OcwOFyRm4+v3lwJqWT4hedPLhZmqNgfiXOehofe1T2B5W4d/+zJ1Y2BCk9Nrak+

bJGM+Za+v1HfOgxNY+XJCpBdxb0GA9Pt8VvGjSi3liQkM9LEWGk9lxUXtzD2lsaDImmwcD

WhJ/FJVV+CNirFcSKN5Z/a8BhGa0MQFRww6hKECJRet1AAAIAKosmjEkUBCzLjPrFjwitp

zOCF6OsuAACfyyLq6jex7R78mBt0FhuezZXIvkvgxCS1gNoacBU7p0fK5d1Nte3NUC5B5k

yKuvIcEtlX8zMgu8aqA1ZDMETr3+C6qfyBiQ690sxfOEsBnW4T2Yo1KAbylOisFuyMq+dh

35nV2hLzNSVLRpiTkZniMYPQHe7o/Kj0zSQtBJNtTAgyc4JkxThYKHEdJOfZ0Zah9hVrqV

wKq2Rwf4eWmZjDABD9fzAPQOVljyaVlAQf3/m4sK5eAX1QbxPsuEUNJ4p/W/SCPaNyZYZD

UlSoEOYueWkHY/yVD5labPLUgFg3p00pUPEtjt/6sAAAP9GTPVWw+wvnbn8aKIRZ6pJpn+

zurN/c8j7wi0hT+63+UShUXO59/EadI5THTlEyB1vVBF78iuoZZfB6t4s3p4dlk/Ke0CxH

Xn81ZJy6r4kd5VhZv2fEKOjq03aewONRX1K4GXbD23KqwokVgIQ9YsJ9A187hbBz8S1N2e

XqP499wAAAQAwUumdnfKNCrr0+a5kwyEvsW85SqgDnVOeP1On1garxE6O5EVYbrlRpgk+O

kDMj7x23LwT9VnZqFcmEO8cpuo3csUzMAd0cX3hkod6RKZ/H1btrEmpzkvFA2Kf7FX3D5i

uKKq5pWykEM5fZjvXe1NbFKOCXytkqnMKJ120Bm84IUAAAQA4WDUlumoGtM/vOZLge9DyW

jO1/lfr7WCoPCFne5APOVLtGMo/xlBFVmnPrnJ6deq2EMk7IRI9ru9SB1TDNV1ktXUH6af

vR6r6vhQbHytjnd1nDeiS3xODS+ePlGwH8RgWzIU+X3uOgsKNJW7Q2OqC3uGcyIGshhc+b

9lXDop7m8=

---- END SSH2ENCRYPTED PRIVATE KEY ----

Albeit a bit tedious,you’ll next need to copy the public key to every slave node as well as themaster node:

Scp ~/.ssh/d_rsa.pubhadoop-user@target:~/master_key

Manually log in tothe target node and set the master key as an authorized key (or append to thelist of authorized keys if you have others defined).

After generatingthe key, you can verify it’s correctly defined by attempting to log in to thetarget node from the master:

$ ssh retailvm1d

You are authorizedto use this System for approved business purposes only.

Use for any otherpurpose is prohibited. All transactional records, reports,

email, software andother data generated by or residing upon this System,

to the extentpermitted by local law, are the property of Citigroup Inc.

or one of itssubsidiaries or their affiliates

(individually orcollectively ' Citigroup ') and may be used by Citigroup

for any purposeauthorized and permissible in your country of work.

Activities on thisSystem are monitored to the extent permitted by local law.

PAM Authentication

Password:

Last login: Mon Jul22 2013 07:15:29 -0400 from APACCNSHZJW2594.apac.nsroot.net

Change JAVA_HOME inhadoop-env.sh.

The standalone modeis the default mode for Hadoop. When you first uncompress the Hadoop source package,it’s ignorant of your hardware setup. Hadoop chooses to be conservative andassumes a minimal configuration.