# start rsync in daemon mode from init.d script?
# only allowed values are "true", "false", and "inetd"
# Use "inetd" if you want to start the rsyncd from inetd,
# all this does is prevent the init.d script from printing a message
# about not starting rsyncd (you still need to modify inetd's config yourself).
RSYNC_ENABLE=false
# which file should be used as the configuration file for rsync.
# This file is used instead of the default /etc/rsyncd.conf
# Warning: This option has no effect if the daemon is accessed
# using a remote shell. When using a different file for
# rsync you might want to symlink /etc/rsyncd.conf to
# that file.
# RSYNC_CONFIG_FILE=
sudo vim /etc/default/rsync
如下：RSYNC_ENABLE 改为true
RSYNC_ENABLE=true查找命令
root@ubuntu:/etc# which rsync
/usr/bin/rsync
查找关于rsyn*的
root@ubuntu:/etc# find -name rsyn*
./init.d/rsync
./default/rsync
./systemd/system/multi-user.target.wants/rsync.service
8.13 .18
root@ubuntu:/hadoop/hadoop2/input# hadoop jar /hadoop/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
wordmean: A map/reduce program that counts the average length of the words in the input files.
wordmedian: A map/reduce program that counts the median length of the words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
root@ubuntu:/hadoop/hadoop2/input# java -version
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)在hadoop的主路径之下mkdir input
再退出到input的上级路径执行，上次失败的命令，
root@ubuntu:/hadoop/hadoop2/input# cd ..
root@ubuntu:/hadoop/hadoop2# ls
bin etc include input lib libexec LICENSE.txt logs NOTICE.txt README.txt sbin share
root@ubuntu:/hadoop/hadoop2# hadoop jar /hadoop/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount input output
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /hadoop/hadoop2/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
24/08/13 03:13:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/08/13 03:13:55 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
24/08/13 03:13:55 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
24/08/13 03:13:55 INFO input.FileInputFormat: Total input paths to process : 3
24/08/13 03:13:55 INFO mapreduce.JobSubmitter: number of splits:3
24/08/13 03:13:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1243728045_0001
24/08/13 03:13:55 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/staging/root1243728045/.staging/job_local1243728045_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
24/08/13 03:13:55 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/staging/root1243728045/.staging/job_local1243728045_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
24/08/13 03:13:55 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local1243728045_0001/job_local1243728045_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
24/08/13 03:13:55 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local1243728045_0001/job_local1243728045_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
24/08/13 03:13:55 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
24/08/13 03:13:55 INFO mapreduce.Job: Running job: job_local1243728045_0001
24/08/13 03:13:55 INFO mapred.LocalJobRunner: OutputCommitter set in config null
24/08/13 03:13:55 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Waiting for map tasks
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Starting task: attempt_local1243728045_0001_m_000000_0
24/08/13 03:13:56 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
24/08/13 03:13:56 INFO mapred.MapTask: Processing split: file:/hadoop/hadoop2/input/LICENSE.txt:0+15458
24/08/13 03:13:56 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
24/08/13 03:13:56 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
24/08/13 03:13:56 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
24/08/13 03:13:56 INFO mapred.MapTask: soft limit at 83886080
24/08/13 03:13:56 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
24/08/13 03:13:56 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
24/08/13 03:13:56 INFO mapred.LocalJobRunner:
24/08/13 03:13:56 INFO mapred.MapTask: Starting flush of map output
24/08/13 03:13:56 INFO mapred.MapTask: Spilling map output
24/08/13 03:13:56 INFO mapred.MapTask: bufstart = 0; bufend = 22764; bufvoid = 104857600
24/08/13 03:13:56 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26205772(104823088); length = 8625/6553600
24/08/13 03:13:56 INFO mapred.MapTask: Finished spill 0
24/08/13 03:13:56 INFO mapred.Task: Task:attempt_local1243728045_0001_m_000000_0 is done. And is in the process of committing
24/08/13 03:13:56 INFO mapred.LocalJobRunner: map
24/08/13 03:13:56 INFO mapred.Task: Task 'attempt_local1243728045_0001_m_000000_0' done.
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Finishing task: attempt_local1243728045_0001_m_000000_0
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Starting task: attempt_local1243728045_0001_m_000001_0
24/08/13 03:13:56 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
24/08/13 03:13:56 INFO mapred.MapTask: Processing split: file:/hadoop/hadoop2/input/README.txt:0+1366
24/08/13 03:13:56 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
24/08/13 03:13:56 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
24/08/13 03:13:56 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
24/08/13 03:13:56 INFO mapred.MapTask: soft limit at 83886080
24/08/13 03:13:56 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
24/08/13 03:13:56 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
24/08/13 03:13:56 INFO mapred.LocalJobRunner:
24/08/13 03:13:56 INFO mapred.MapTask: Starting flush of map output
24/08/13 03:13:56 INFO mapred.MapTask: Spilling map output
24/08/13 03:13:56 INFO mapred.MapTask: bufstart = 0; bufend = 2055; bufvoid = 104857600
24/08/13 03:13:56 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213684(104854736); length = 713/6553600
24/08/13 03:13:56 INFO mapred.MapTask: Finished spill 0
24/08/13 03:13:56 INFO mapred.Task: Task:attempt_local1243728045_0001_m_000001_0 is done. And is in the process of committing
24/08/13 03:13:56 INFO mapred.LocalJobRunner: map
24/08/13 03:13:56 INFO mapred.Task: Task 'attempt_local1243728045_0001_m_000001_0' done.
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Finishing task: attempt_local1243728045_0001_m_000001_0
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Starting task: attempt_local1243728045_0001_m_000002_0
24/08/13 03:13:56 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
24/08/13 03:13:56 INFO mapred.MapTask: Processing split: file:/hadoop/hadoop2/input/NOTICE.txt:0+101
24/08/13 03:13:56 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
24/08/13 03:13:56 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
24/08/13 03:13:56 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
24/08/13 03:13:56 INFO mapred.MapTask: soft limit at 83886080
24/08/13 03:13:56 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
24/08/13 03:13:56 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
24/08/13 03:13:56 INFO mapred.LocalJobRunner:
24/08/13 03:13:56 INFO mapred.MapTask: Starting flush of map output
24/08/13 03:13:56 INFO mapred.MapTask: Spilling map output
24/08/13 03:13:56 INFO mapred.MapTask: bufstart = 0; bufend = 145; bufvoid = 104857600
24/08/13 03:13:56 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
24/08/13 03:13:56 INFO mapred.MapTask: Finished spill 0
24/08/13 03:13:56 INFO mapred.Task: Task:attempt_local1243728045_0001_m_000002_0 is done. And is in the process of committing
24/08/13 03:13:56 INFO mapred.LocalJobRunner: map
24/08/13 03:13:56 INFO mapred.Task: Task 'attempt_local1243728045_0001_m_000002_0' done.
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Finishing task: attempt_local1243728045_0001_m_000002_0
24/08/13 03:13:56 INFO mapred.LocalJobRunner: map task executor complete.
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Waiting for reduce tasks
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Starting task: attempt_local1243728045_0001_r_000000_0
24/08/13 03:13:56 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
24/08/13 03:13:56 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6c2170b0
24/08/13 03:13:56 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
24/08/13 03:13:56 INFO reduce.EventFetcher: attempt_local1243728045_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
24/08/13 03:13:56 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1243728045_0001_m_000000_0 decomp: 11017 len: 11021 to MEMORY
24/08/13 03:13:56 INFO reduce.InMemoryMapOutput: Read 11017 bytes from map-output for attempt_local1243728045_0001_m_000000_0
24/08/13 03:13:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11017, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11017
24/08/13 03:13:56 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1243728045_0001_m_000002_0 decomp: 169 len: 173 to MEMORY
24/08/13 03:13:56 INFO reduce.InMemoryMapOutput: Read 169 bytes from map-output for attempt_local1243728045_0001_m_000002_0
24/08/13 03:13:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 169, inMemoryMapOutputs.size() -> 2, commitMemory -> 11017, usedMemory ->11186
24/08/13 03:13:56 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1243728045_0001_m_000001_0 decomp: 1832 len: 1836 to MEMORY
24/08/13 03:13:56 INFO reduce.InMemoryMapOutput: Read 1832 bytes from map-output for attempt_local1243728045_0001_m_000001_0
24/08/13 03:13:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1832, inMemoryMapOutputs.size() -> 3, commitMemory -> 11186, usedMemory ->13018
24/08/13 03:13:56 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
24/08/13 03:13:56 INFO mapred.LocalJobRunner: 3 / 3 copied.
24/08/13 03:13:56 INFO reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
24/08/13 03:13:56 INFO mapred.Merger: Merging 3 sorted segments
24/08/13 03:13:56 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 12975 bytes
24/08/13 03:13:56 INFO reduce.MergeManagerImpl: Merged 3 segments, 13018 bytes to disk to satisfy reduce memory limit
24/08/13 03:13:56 INFO reduce.MergeManagerImpl: Merging 1 files, 13018 bytes from disk
24/08/13 03:13:56 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
24/08/13 03:13:56 INFO mapred.Merger: Merging 1 sorted segments
24/08/13 03:13:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 13008 bytes
24/08/13 03:13:56 INFO mapred.LocalJobRunner: 3 / 3 copied.
24/08/13 03:13:56 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
24/08/13 03:13:56 INFO mapred.Task: Task:attempt_local1243728045_0001_r_000000_0 is done. And is in the process of committing
24/08/13 03:13:56 INFO mapred.LocalJobRunner: 3 / 3 copied.
24/08/13 03:13:56 INFO mapred.Task: Task attempt_local1243728045_0001_r_000000_0 is allowed to commit now
24/08/13 03:13:56 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1243728045_0001_r_000000_0' to file:/hadoop/hadoop2/output/_temporary/0/task_local1243728045_0001_r_000000
24/08/13 03:13:56 INFO mapred.LocalJobRunner: reduce > reduce
24/08/13 03:13:56 INFO mapred.Task: Task 'attempt_local1243728045_0001_r_000000_0' done.
24/08/13 03:13:56 INFO mapred.LocalJobRunner: Finishing task: attempt_local1243728045_0001_r_000000_0
24/08/13 03:13:56 INFO mapred.LocalJobRunner: reduce task executor complete.
24/08/13 03:13:56 INFO mapreduce.Job: Job job_local1243728045_0001 running in uber mode : false
24/08/13 03:13:56 INFO mapreduce.Job: map 100% reduce 100%
24/08/13 03:13:56 INFO mapreduce.Job: Job job_local1243728045_0001 completed successfully
24/08/13 03:13:56 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=1176534
FILE: Number of bytes written=2041140
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=323
Map output records=2347
Map output bytes=24964
Map output materialized bytes=13030
Input split bytes=307
Combine input records=2347
Combine output records=897
Reduce input groups=840
Reduce shuffle bytes=13030
Reduce input records=897
Reduce output records=840
Spilled Records=1794
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1389887488
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=16925
File Output Format Counters
Bytes Written=9092
root@ubuntu:/hadoop/hadoop2#
在去到hadoop的路径，发现一个output夹
root@ubuntu:/hadoop/hadoop2# cd output/
root@ubuntu:/hadoop/hadoop2/output# ls
part-r-00000 _SUCCESS
root@ubuntu:/hadoop/hadoop2/output# cat *
"AS 4
"Contribution" 1
"Contributor" 1
"Derivative 1
"Legal 1
"License" 1
"License"); 1
"Licensor" 1
"NOTICE" 1
"Not 1
"Object" 1
"Source" 1
"Work" 1
"You" 1
"Your") 1
"[]" 1
"control" 1
"printed 1
"submitted" 1
(50%) 1
(BIS), 1
(C) 1
(Don't 1
(ECCN) 1
(INCLUDING 2
(INCLUDING, 2
(TSU) 1
(a) 1
(an 1
(and 1
(b) 1
(c) 2
(d) 1
(except 1
(http://www.apache.org/). 1
(http://www.one-lab.org) 1
(http://www.opensource.org/licenses/bsd-license.php) 1
(i) 1
(ii) 1
(iii) 1
(including 3
扫一扫
3868
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
