Hadoop单机安装!
System:Centos6.5
Software:apache-maven-3.3.3-bin.tar.gz | hadoop-2.7.1.tar.gz | jdk-8u65-linux-i586.tar.gz | scala-2.11.7.tgz | spark-1.5.1-bin-without-hadoop.tgz
详细介绍:http://www.aboutyun.com/thread-12798-1-1.html
linux:
yum -y update
yum -y install glibc*
tar xzvf apache-maven-3.3.3-bin.tar.gz
tar xzvf hadoop-2.7.1.tar.gz
tar xzvf scala-2.11.7.tgz
tar xzvf spark-1.5.1-bin-without-hadoop.tgz
tar xzvf jdk-8u65-linux-i586.tar.gz
(单机免密登录)
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat .ssh/id_dsa.pub >> .ssh/authorized_keys
ssh localhost
export HADOOP\_PREFIX=/usr/local/hadoop
安装RSYNC
yum install -y rsync
复制解压缩后的文件夹到/usr/local/ 目录下 变更文件夹权限及用户 chown spark:spark -R * | chmod 775 -R *
创建用户和组(稍后更新)
写.bash_profile
export SCALA_HOME=/usr/local/scala
export JAVA_HOME=/usr/local/jdk
export MAVEN_HOME=/usr/local/maven
export HADOOP_HOME=/usr/local/hadoop
export PATH=/usr/local/maven/bin:$PATH
export PATH=$JAVA_HOME/bin:$SCALA_HOME/bin:$MAVEN/bin:$HOME/bin:$HADOOP/bin:$HADOOP/sbin:$PATH
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/jre/lib/dt.jar:$JAVA_HOME/jre/lib/tools.jar
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/lib:$SCALA/lib:$MAVEN/lib:/usr/local/lib:/usr/lin
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
修改
hadoop etc/hadoop/hadoop-env.sh
修改Java_Home export JAVA_HOME=/usr/jdk (一定是详细目录)
export HADOOP_COMMON_HOME=~/hadoop-2.7.0(没发现命令)
nano /etc/environment
/usr/local/hadoop/bin:/usr/local/hadoop/sbin
在hadoop 目录下创建input
mkdir input
cp etc/hadoop/*.xml input
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+' (注意JAR包版本号)
cat output/*
修改文件etc/hadoop/core-site.xml
添加如下内容:
含义:接收Client连接的RPC端口,用于获取文件系统metadata信息。
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
修改etc/hadoop/hdfs-site.xml:
含义:备份只有一份
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/spark/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/spark/dfs/data</value>
</property>
注意修改权限。
格式化namenode
hdfs namenode -format
或使用
bin/hdfs namenode -format
执行
start-dfs.sh
登录
localip:50070