Hadoop + Spark 配置指南

---

Environment Configuration

User Creation

Add a user named hadoop which is a sudoer, and switch user to it

~/.bashrc

Add export HADOOP_HOME=/usr/local/hadoop

At least add JAVA_HOME in $HADOOP_HOME/etc/hadoop/hadoop-env.sh so that it is correctly defined on each remote node

/etc/hosts

ip_address master
ip_address slave1
ip_address slave2

ssh_key

Append master’s public key on every slave’s ~/.ssh/authorized_keys

Hadoop Configuration

$HADOOP_HOME/etc/hadoop/core-site.xml

<configuration>
      <property>
          <name>hadoop.tmp.dir</name>
          <value>file:/hadoop/tmp</value>
          <description>(Optional)A base for other temporary directories.</description>
      </property>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://master:9000</value>
      </property>
</configuration