1 搭建
两个节点，一个作为master，一个作为slave02. 内存4G，硬盘75G
java7
hadoop2.6.4
2. 初始化
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
16/09/16 16:52:47 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = miaofu-Virtual-Machine/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.6.4
reduce/hadoop-mapreduce-client-jobclient-2.6.4-tests.jar:/home/miaofu/hadoop-2.6.4/contrib/capacity-scheduler/*.jar:/home/miaofu/hadoop-2.6.4/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 5082c73637530b0b7e115f9625ed7fac69f937e6; compiled by 'jenkins' on 2016-02-12T09:45Z
STARTUP_MSG: java = 1.7.0_95
************************************************************/
16/09/16 16:52:47 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/09/16 16:52:47 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-6d4ff550-c960-4752-9027-e74c7cbc25e8
16/09/16 16:52:49 INFO namenode.FSNamesystem: No KeyProvider found.
16/09/16 16:52:49 INFO namenode.FSNamesystem: fsLock is fair:true
16/09/16 16:52:49 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/09/16 16:52:49 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/09/16 16:52:49 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/09/16 16:52:49 INFO blockmanagement.BlockManager: The block deletion will start around 2016 九月 16 16:52:49
16/09/16 16:52:49 INFO util.GSet: Computing capacity for map BlocksMap
16/09/16 16:52:49 INFO util.GSet: VM type = 64-bit
16/09/16 16:52:49 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
16/09/16 16:52:49 INFO util.GSet: capacity = 2^21 = 2097152 entries
16/09/16 16:52:49 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/09/16 16:52:49 INFO blockmanagement.BlockManager: defaultReplication = 2
16/09/16 16:52:49 INFO blockmanagement.BlockManager: maxReplication = 512
16/09/16 16:52:49 INFO blockmanagement.BlockManager: minReplication = 1
16/09/16 16:52:49 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
16/09/16 16:52:49 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/09/16 16:52:49 INFO blockmanagement.BlockManager: encryptDataTransfer = false
16/09/16 16:52:49 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
16/09/16 16:52:49 INFO namenode.FSNamesystem: fsOwner = miaofu (auth:SIMPLE)
16/09/16 16:52:49 INFO namenode.FSNamesystem: supergroup = supergroup
16/09/16 16:52:49 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/09/16 16:52:49 INFO namenode.FSNamesystem: HA Enabled: false
16/09/16 16:52:49 INFO namenode.FSNamesystem: Append Enabled: true
16/09/16 16:52:49 INFO util.GSet: Computing capacity for map INodeMap
16/09/16 16:52:49 INFO util.GSet: VM type = 64-bit
16/09/16 16:52:49 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
16/09/16 16:52:49 INFO util.GSet: capacity = 2^20 = 1048576 entries
16/09/16 16:52:49 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/09/16 16:52:49 INFO util.GSet: Computing capacity for map cachedBlocks
16/09/16 16:52:49 INFO util.GSet: VM type = 64-bit
16/09/16 16:52:49 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
16/09/16 16:52:49 INFO util.GSet: capacity = 2^18 = 262144 entries
16/09/16 16:52:49 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/09/16 16:52:49 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/09/16 16:52:49 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
16/09/16 16:52:49 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/09/16 16:52:49 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/09/16 16:52:49 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/09/16 16:52:49 INFO util.GSet: VM type = 64-bit
16/09/16 16:52:49 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
16/09/16 16:52:49 INFO util.GSet: capacity = 2^15 = 32768 entries
16/09/16 16:52:49 INFO namenode.NNConf: ACLs enabled? false
16/09/16 16:52:49 INFO namenode.NNConf: XAttrs enabled? true
16/09/16 16:52:49 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/09/16 16:52:49 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1662540362-127.0.0.1-1474015969302
16/09/16 16:52:49 INFO common.Storage: Storage directory /home/miaofu/hadoop-2.6.4/tmp/dfs/name has been successfully formatted.
16/09/16 16:52:49 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/09/16 16:52:49 INFO util.ExitUtil: Exiting with status 0
16/09/16 16:52:49 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at miaofu-Virtual-Machine/127.0.0.1
************************************************************/miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /home/miaofu/hadoop-2.6.4/logs/hadoop-miaofu-namenode-miaofu-Virtual-Machine.out
master: starting datanode, logging to /home/miaofu/hadoop-2.6.4/logs/hadoop-miaofu-datanode-miaofu-Virtual-Machine.out
slave02: starting datanode, logging to /home/miaofu/hadoop-2.6.4/logs/hadoop-miaofu-datanode-miaofu-Virtual-Machine.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/miaofu/hadoop-2.6.4/logs/hadoop-miaofu-secondarynamenode-miaofu-Virtual-Machine.out
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ jps
11612 NameNode
60711 RunJar
53210 RunJar
53648 RunJar
43661 FsShell
21191 GetConf
11982 SecondaryNameNode
12132 Jps
11777 DataNode
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Configured Capacity: 160379584512 (149.37 GB)
Present Capacity: 113242701824 (105.47 GB)
DFS Remaining: 113242652672 (105.47 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Live datanodes (2):
Name: 192.168.202.105:50010 (slave02)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 80189792256 (74.68 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 22415831040 (20.88 GB)
DFS Remaining: 57773936640 (53.81 GB)
DFS Used%: 0.00%
DFS Remaining%: 72.05%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Sep 16 16:53:47 CST 2016
Name: 192.168.202.104:50010 (master)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 80189792256 (74.68 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 24721051648 (23.02 GB)
DFS Remaining: 55468716032 (51.66 GB)
DFS Used%: 0.00%
DFS Remaining%: 69.17%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Sep 16 16:53:47 CST 2016
3. 操作分布式文件系统
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ hadoop fs -ls /
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ hdfs dfs -mkdir /input
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ hadoop fs -ls /
Found 1 items
drwxr-xr-x - miaofu supergroup 0 2016-09-16 16:56 /input
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ hdfs dfs -put etc/hadoop/* /input
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ hadoop fs -ls /input
Found 30 items
-rw-r--r-- 2 miaofu supergroup 4436 2016-09-16 16:57 /input/capacity-scheduler.xml
-rw-r--r-- 2 miaofu supergroup 1335 2016-09-16 16:57 /input/configuration.xsl
-rw-r--r-- 2 miaofu supergroup 318 2016-09-16 16:57 /input/container-executor.cfg
-rw-r--r-- 2 miaofu supergroup 988 2016-09-16 16:57 /input/core-site.xml
-rw-r--r-- 2 miaofu supergroup 3670 2016-09-16 16:57 /input/hadoop-env.cmd
-rw-r--r-- 2 miaofu supergroup 4292 2016-09-16 16:57 /input/hadoop-env.sh
-rw-r--r-- 2 miaofu supergroup 2490 2016-09-16 16:57 /input/hadoop-metrics.properties
-rw-r--r-- 2 miaofu supergroup 2598 2016-09-16 16:57 /input/hadoop-metrics2.properties
-rw-r--r-- 2 miaofu supergroup 9683 2016-09-16 16:57 /input/hadoop-policy.xml
-rw-r--r-- 2 miaofu supergroup 973 2016-09-16 16:57 /input/hdfs-site.xml
-rw-r--r-- 2 miaofu supergroup 1449 2016-09-16 16:57 /input/httpfs-env.sh
-rw-r--r-- 2 miaofu supergroup 1657 2016-09-16 16:57 /input/httpfs-log4j.properties
-rw-r--r-- 2 miaofu supergroup 21 2016-09-16 16:57 /input/httpfs-signature.secret
-rw-r--r-- 2 miaofu supergroup 620 2016-09-16 16:57 /input/httpfs-site.xml
-rw-r--r-- 2 miaofu supergroup 3523 2016-09-16 16:57 /input/kms-acls.xml
-rw-r--r-- 2 miaofu supergroup 1325 2016-09-16 16:57 /input/kms-env.sh
-rw-r--r-- 2 miaofu supergroup 1631 2016-09-16 16:57 /input/kms-log4j.properties
-rw-r--r-- 2 miaofu supergroup 5511 2016-09-16 16:57 /input/kms-site.xml
-rw-r--r-- 2 miaofu supergroup 11291 2016-09-16 16:57 /input/log4j.properties
-rw-r--r-- 2 miaofu supergroup 938 2016-09-16 16:57 /input/mapred-env.cmd
-rw-r--r-- 2 miaofu supergroup 1383 2016-09-16 16:57 /input/mapred-env.sh
-rw-r--r-- 2 miaofu supergroup 4113 2016-09-16 16:57 /input/mapred-queues.xml.template
-rw-r--r-- 2 miaofu supergroup 850 2016-09-16 16:57 /input/mapred-site.xml
-rw-r--r-- 2 miaofu supergroup 758 2016-09-16 16:57 /input/mapred-site.xml.template
-rw-r--r-- 2 miaofu supergroup 15 2016-09-16 16:57 /input/slaves
-rw-r--r-- 2 miaofu supergroup 2316 2016-09-16 16:57 /input/ssl-client.xml.example
-rw-r--r-- 2 miaofu supergroup 2268 2016-09-16 16:57 /input/ssl-server.xml.example
-rw-r--r-- 2 miaofu supergroup 2237 2016-09-16 16:57 /input/yarn-env.cmd
-rw-r--r-- 2 miaofu supergroup 4618 2016-09-16 16:57 /input/yarn-env.sh
-rw-r--r-- 2 miaofu supergroup 916 2016-09-16 16:57 /input/yarn-site.xml
4. 启动yarn
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/miaofu/hadoop-2.6.4/logs/yarn-miaofu-resourcemanager-miaofu-Virtual-Machine.out
slave02: starting nodemanager, logging to /home/miaofu/hadoop-2.6.4/logs/yarn-miaofu-nodemanager-miaofu-Virtual-Machine.out
master: starting nodemanager, logging to /home/miaofu/hadoop-2.6.4/logs/yarn-miaofu-nodemanager-miaofu-Virtual-Machine.out
miaofu@miaofu-Virtual-Machine:~/hadoop-2.6.4$ jps
11612 NameNode
60711 RunJar
53210 RunJar
53648 RunJar
14392 Jps
14026 ResourceManager
43661 FsShell
21191 GetConf
14148 NodeManager
11982 SecondaryNameNode
11777 DataNode
5. 测试：wordcount
miaofu@miaofu-Virtual-Machine:~$ hadoop fs -mkdir /in
miaofu@miaofu-Virtual-Machine:~$ vi retry.sh
miaofu@miaofu-Virtual-Machine:~$ hadoop fs -put etc/hadoop/*.xml /in
put: `etc/hadoop/*.xml': No such file or directory
miaofu@miaofu-Virtual-Machine:~$ hadoop fs -put hadoop-2.6.4/etc/hadoop/*.xml /in
jpmiaofu@miaofu-Virtual-Machine:~$ jps
61591 Jps
60601 ResourceManager
60297 SecondaryNameNode
60732 NodeManager
60092 DataNode
59927 NameNode
miaofu@miaofu-Virtual-Machine:~$ hadoop jar hadoop-2.6.4/
bin/ etc/ include/ lib/ LICENSE.txt NOTICE.txt sbin/ tmp/
conf.sh home/ input/ libexec/ logs/ README.txt share/
miaofu@miaofu-Virtual-Machine:~$ hadoop jar hadoop-2.6.4/share/
doc/ hadoop/
miaofu@miaofu-Virtual-Machine:~$ hadoop jar hadoop-2.6.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount /in /out
^Z
[1]+ 已停止 hadoop jar hadoop-2.6.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount /in /out
miaofu@miaofu-Virtual-Machine:~$ hadoop jar hadoop-2.6.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount /in /out3
16/09/17 16:46:24 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.202.104:8032
16/09/17 16:46:25 INFO input.FileInputFormat: Total input paths to process : 9
16/09/17 16:46:25 INFO mapreduce.JobSubmitter: number of splits:9
16/09/17 16:46:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1474101888060_0001
16/09/17 16:46:26 INFO impl.YarnClientImpl: Submitted application application_1474101888060_0001
16/09/17 16:46:26 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1474101888060_0001/
16/09/17 16:46:26 INFO mapreduce.Job: Running job: job_1474101888060_0001
16/09/17 16:46:35 INFO mapreduce.Job: Job job_1474101888060_0001 running in uber mode : false
16/09/17 16:46:35 INFO mapreduce.Job: map 0% reduce 0%
16/09/17 16:46:44 INFO mapreduce.Job: map 22% reduce 0%
16/09/17 16:46:45 INFO mapreduce.Job: map 33% reduce 0%
16/09/17 16:46:48 INFO mapreduce.Job: map 67% reduce 0%
16/09/17 16:46:49 INFO mapreduce.Job: map 100% reduce 0%
16/09/17 16:46:51 INFO mapreduce.Job: map 100% reduce 100%
16/09/17 16:46:52 INFO mapreduce.Job: Job job_1474101888060_0001 completed successfully
16/09/17 16:46:52 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=21875
FILE: Number of bytes written=1110853
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28532
HDFS: Number of bytes written=10579
HDFS: Number of read operations=30
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Killed map tasks=1
Launched map tasks=9
Launched reduce tasks=1
Data-local map tasks=9
Total time spent by all maps in occupied slots (ms)=84614
Total time spent by all reduces in occupied slots (ms)=4042
Total time spent by all map tasks (ms)=84614
Total time spent by all reduce tasks (ms)=4042
Total vcore-milliseconds taken by all map tasks=84614
Total vcore-milliseconds taken by all reduce tasks=4042
Total megabyte-milliseconds taken by all map tasks=86644736
Total megabyte-milliseconds taken by all reduce tasks=4139008
Map-Reduce Framework
Map input records=796
Map output records=2887
Map output bytes=36776
Map output materialized bytes=21923
Input split bytes=915
Combine input records=2887
Combine output records=1265
Reduce input groups=606
Reduce shuffle bytes=21923
Reduce input records=1265
Reduce output records=606
Spilled Records=2530
Shuffled Maps =9
Failed Shuffles=0
Merged Map outputs=9
GC time elapsed (ms)=590
CPU time spent (ms)=6470
Physical memory (bytes) snapshot=2690990080
Virtual memory (bytes) snapshot=8380964864
Total committed heap usage (bytes)=1966604288
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=27617
File Output Format Counters
Bytes Written=10579
扫一扫
226
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
