大数据&云计算正在成为当下以及未来科技发展的趋势
步骤详解:
我的安装环境:
Mac OS X ssh配置:
macdeMac-mini-2:~ mac$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/mac/.ssh/id_rsa): 【回车】
/Users/mac/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase): 【回车】
Enter same passphrase again: 【回车】
Your identification has been saved in /Users/mac/.ssh/id_rsa.
Your public key has been saved in /Users/mac/.ssh/id_rsa.pub.
macdeMac-mini-2:.ssh mac$ cd ~/.ssh
macdeMac-mini-2:.ssh mac$ cp id_rsa.pub authorized_keys
macdeMac-mini-2:.ssh mac$ scp -p ~/.ssh/authorized_keys vb_macos:.ssh/
macdeMac-mini-2:~ mac$ ps aux | grep ssh
mac 618 0.0 0.1 2518544 5192 ?? S 9:50AM 0:00.05 /usr/bin/ssh-agent -l
mac 1773 0.0 0.0 2445080 808 s001 S+ 2:58PM 0:00.00 grep ssh
macdeMac-mini-2:~ mac$ ssh localhost
Last login: Wed Feb 3 14:58:25 2016
macdeMac-mini-2:~ mac$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
/Users/mac/.ssh/id_dsa already exists.
Overwrite (y/n)? y
Your identification has been saved in /Users/mac/.ssh/id_dsa.
Your public key has been saved in /Users/mac/.ssh/id_dsa.pub.
The key fingerprint is:
SHA256:K4A8m/0Z00CdRpmC+GJwZxoRksQ01XT8O+ec2v6xI9A mac@macdeMac-mini-2.local
The key's randomart image is:
+---[DSA 1024]----+
|==+=oo...o |
|oo= +.o+o. |
| o * ..= |
| .+... . . |
| .+.. . S o |
| = . o = E |
| o . + o * .. |
| . = .= .o |
| o .oooo. |
+----[SHA256]-----+
macdeMac-mini-2:~ mac$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
hadoop 1.2.1下载并且安装
export PAHT=$PATH:/usr/local/apache-tomcat-7.0.67/bin
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home
export JRE_HOME=$JAVA_HOME/jre
export HADOOP_HOME=/Users/mac/Documents/hadoop-1.2.1
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export HADOOP_HOME_WARN_SUPPRESS=1
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home
export JRE_HOME=$JAVA_HOME/jre
以上2各是java环境变量
export HADOOP_HOME=/Users/mac/Documents/hadoop-1.2.1
这个是hadoop环境变量
export HADOOP_HOME_WARN_SUPPRESS=1
防止出现Warning:$HADOOP_HOME is deprecated的警告错误
保存,在终端输入:source ~/.bash_profile
在Hadoop->conf目录下,找到hadoop-env.sh,打开编辑进行如下设置:
配置conf文件夹下的core-site.xml——指定了NameNode的主机名与端口
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>hdfs://localhost:9000</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8020</value>
</property>
</configuration>
配置hdfs-site.xml——指定了HDFS的默认参数副本数,因为仅运行在一个节点上,所以这里的副本数为1
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
配置mapred-site.xml——指定了JobTracker的主机名与端口
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://localhost:9001</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>2</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>2</value>
</property>
</configuration>
macdeMac-mini-2:~ mac$ hadoop
Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
namenode -format format the DFS filesystem
secondarynamenode run the DFS secondary namenode
namenode run the DFS namenode
datanode run a DFS datanode
dfsadmin run a DFS admin client
mradmin run a Map-Reduce admin client
fsck run a DFS filesystem checking utility
fs run a generic filesystem user client
balancer run a cluster balancing utility
oiv apply the offline fsimage viewer to an fsimage
fetchdt fetch a delegation token from the NameNode
jobtracker run the MapReduce job Tracker node
pipes run a Pipes job
tasktracker run a MapReduce task Tracker node
historyserver run job history servers as a standalone daemon
job manipulate MapReduce jobs
queue get information regarding JobQueues
version print the version
jar <jar> run a jar file
distcp <srcurl> <desturl> copy file or directories recursively
distcp2 <srcurl> <desturl> DistCp version 2
archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
classpath prints the class path needed to get the
Hadoop jar and the required libraries
daemonlog get/set the log level for each daemon
or
CLASSNAME run the class named CLASSNAME
Most commands print help when invoked w/o parameters.
出现以上内容说明找到了hadoop的执行程序
对namenode执行格式化操作,出现如下结果:
macdeMac-mini-2:~ mac$ hadoop namenode -format
16/02/03 15:19:43 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = macdeMac-mini-2.local/192.168.2.176
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.8.0_40
************************************************************/
16/02/03 15:19:43 INFO util.GSet: Computing capacity for map BlocksMap
16/02/03 15:19:43 INFO util.GSet: VM type = 64-bit
16/02/03 15:19:43 INFO util.GSet: 2.0% max memory = 1864368128
16/02/03 15:19:43 INFO util.GSet: capacity = 2^22 = 4194304 entries
16/02/03 15:19:43 INFO util.GSet: recommended=4194304, actual=4194304
16/02/03 15:19:44 INFO namenode.FSNamesystem: fsOwner=mac
16/02/03 15:19:44 INFO namenode.FSNamesystem: supergroup=supergroup
16/02/03 15:19:44 INFO namenode.FSNamesystem: isPermissionEnabled=true
16/02/03 15:19:44 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
16/02/03 15:19:44 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
16/02/03 15:19:44 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
16/02/03 15:19:44 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/02/03 15:19:44 INFO common.Storage: Image file hdfs:/localhost:9000/dfs/name/current/fsimage of size 109 bytes saved in 0 seconds.
16/02/03 15:19:44 INFO namenode.FSEditLog: closing edit log: position=4, editlog=hdfs:/localhost:9000/dfs/name/current/edits
16/02/03 15:19:44 INFO namenode.FSEditLog: close success: truncate to 4, editlog=hdfs:/localhost:9000/dfs/name/current/edits
16/02/03 15:19:44 INFO common.Storage: Storage directory hdfs:/localhost:9000/dfs/name has been successfully formatted.
16/02/03 15:19:44 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at macdeMac-mini-2.local/192.168.2.176
************************************************************/
说明HDFS安装成功
启动hadoop,输入jus,查看当前进程
macdeMac-mini-2:~ mac$ jps
1361 JobTracker
1953 Jps
1306 SecondaryNameNode
macdeMac-mini-2:~ mac$
OK handop已经启动