hadoop 学习笔记002（安装和启动hadoop）

linux上的hadoop操作
ls 简单的列出目录下的文件
ll 列出目录下的详细文件

tar -zxvf hadoop-2.4.1.tar.gz -C /app

1.修改hadoop的配置信息

    /etc/hadoop/hadoop-env.sh

显示java安装目录

echo $JAVA_HOME

配置hadoop的java环境变量

# The java implementation to use.
export JAVA_HOME=/usr/local/java/jdk1.7.0_79

保存退出

    :wq

core-site.xml配置：

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://127.0.0.1:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/app/hadoop-2.4.1/tmp</value>
        </property>
</configuration>

其他参考：

    <configuration>
        <property>
            <name>fs.default.name</name>
            <value>hdfs://hadoop:9000</value>
        </property>
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/usr/local/hadoop/tmp</value>
        </property>
    </configuration>

配置参考：
hdfs://127.0.0.1:9000/
hdfs://weekend-1206-01:9000

hdfs-site.xml配置：

<configuration>
    <property>
        <name>dfs.replication</name>    
        <value>1</value>
    </property> 
    <property>
        <name>dfs.namenode.rpc-address</name>
        <value>本机ip:8080</value>
    </property> 
</configuration>

注意：
    修改/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
193.168.0.103   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

(改名)移动命令

    mv mapred-site.xml.template mapred-site.xml

mapred-site.xml配置：

<configuration>
    <property>
        <name>mapreduce.framework.name</name>   
        <value>yarn</value>
    </property>         
</configuration>    

yarn-site.xml配置：

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>  
        <value>127.0.0.1</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value>
    </property>         
</configuration>    

配置hadoop环境变量

sudo vi /etc/profile

        export HADOOP_HOME=/app/hadoop-2.4.1         
        export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$HADOOP_HOME/lib

        export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
        export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

参考例子1 start

export JAVA_HOME=/usr/local/java/jdk1.7.0_79
            export HADOOP_HOME=/app/hadoop-2.4.1
            export JRE_HOME=$JAVA_HOME/jre
            export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
            export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
            export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$HADOOP_HOME/lib
            export CLASSPATH=./:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

参考配置 (网上例子) start

            PATH=$PATH:$HOME/bin  
            export HADOOP_HOME=/home/hadoop/hadoop  
            export HADOOP_COMMON_HOME=$HADOOP_HOME  
            export HADOOP_HDFS_HOME=$HADOOP_HOME  
            export HADOOP_MAPRED_HOME=$HADOOP_HOME  
            export HADOOP_YARN_HOME=$HADOOP_HOME  
            export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop  
            export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib  

            export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native  
            export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"  

            export PATH 

启动hadoop并格化

hadoop namenode -format

如果出现，表示成功
xxx has been successfully formatted

进入：

cd /tmp/dfs/name/current
cat VERSION

启动dfs[sbin]

start-dfs.sh

====》start ssh
ssh: connect to host slave port 22: Connection timed out的问题

vim /etc/ssh/sshd_config  
MaxSessions 1000  

传送门：http://blog.csdn.net/u010503822/article/details/78255876
重启SSH服务

/etc/rc.d/init.d/sshd restart  
或者
service sshd restart 

====》end ssh

[sbin]查看

jps

[sbin]查看

netstat -nltp

处理org.apache.hadoop.security.AccessControlExcepti

 ./hadoop fs -chmod 777 /  

切换用户

su - root 

启动hadoop服务

start-all.sh

停止hadoop服务

stop-all.sh

验证是否成功,命令：jps

34203 ResourceManager
34051 SecondaryNameNode
39810 Jps
34292 NodeManager
33885 DataNode
33801 NameNode

访问地址：

http://localhost:50070 （HDFS管理界面）
http://localhost:8088 （MR管理界面）