Hadoop 例子
MR作业的串联
有时候,用一个单独的Map/Reduce作业并不能完成一个复杂的任务,用户也许要链接多个Map/Reduce作业才行。这是容易实现的,因为作业通常输出到分布式文件系统上的,所以可以把这个作业的输出作为下一个作业的输入实现串联。
然而,这也意味着,确保每一作业完成(成功或失败)的责任就直接落在了客户身上。
org.apache.hadoop.examples.Grep
自定义Key,InputFormat,
org.apache.hadoop.examples.MultiFileWordCount
TeraSort
Trie树详解
http://www.cppblog.com/abilitytao/archive/2009/04/21/80598.html
http://www.cnblogs.com/cherish_yimi/archive/2009/10/12/1581666.html
生成TeraSort所需要的随机文件
org.apache.hadoop.examples.RandomTextWriter
3台机器,每台一个Map,产生1G的随机文件(1G对于测试机器负载太大,后来修正为200M)
bin/hadoop fs -rmr tera-sort-source
bin/hadoop jar examples-hadoop.zip org.apache.hadoop.examples.RandomTextWriter -Dtest.randomtextwrite.bytes_per_map=52428800 tera-sort-source
第一次1G:10/04/16 11:34:15--10/04/16 11:39:02
第二次200M(209715200):The job took 53 seconds
第三次50M(52428800):The job took 28 seconds.
bin/hadoop jar examples-hadoop.zip org.apache.hadoop.examples.WordCount tera-sort-source word-count-out
bin/hadoop job -kill job_201004151046_0034
bin/hadoop fs -rmr tera-sort-out
bin/hadoop jar examples-hadoop.zip org.apache.hadoop.examples.AggregateWordCount tera-sort-source tera-sort-out 3
执行TeraSort,同时利用TotalOrderPartition
org.apache.hadoop.examples.terasort.TeraSort
程序中指定输出结果的replication为1
先测试简单的英文单词文件Sort结果
bin/hadoop fs -mkdir tera-sort-simple
bin/hadoop fs -put tera-sort.txt tera-sort-simple/
bin/hadoop fs -rmr tera-sort-simple-out
bin/hadoop jar examples-hadoop.zip org.apache.hadoop.examples.terasort.TeraSort -Dmapred.reduce.tasks=36 tera-sort-simple tera-sort-simple-out
bin/hadoop fs -rmr tera-sort-out
bin/hadoop jar examples-hadoop.zip org.apache.hadoop.examples.terasort.TeraSort -Dmapred.reduce.tasks=3 tera-sort-source tera-sort-out
第一次,执行的时候,抛出异常,而且test-a机器挂掉。
第二次,中间有异常,但是Job最终执行成功,指定1个Reduce
Task Id : attempt_201004161343_0001_m_000050_0, Status : FAILED
Could not find any valid local directory for taskTracker/jobcache/job_201004161343_0001/attempt_201004161343_0001_m_000023_1/output/spill0.out
10/04/16 13:44:27
10/04/16 13:55:10
第三次,3GB,指定36个Reduce,2.79当机。
第四次600M,指定3个Reduce,OK
10/04/16 15:24:55
10/04/16 15:25:50
查看结果
bin/hadoop fs -get /user/iic/tera-sort-out/part-00000 sort1.txt
自定义Key,Partition,Comparator
org.apache.hadoop.examples.SecondarySort
WordCount
/home/bmb/hadoop-0.20.2/examples/wordcount.jar
bin/hadoop dfs -mkdir /usr/joe/wordcount/input
bin/hadoop dfs -put conf/*.xml /usr/joe/wordcount/input/
bin/hadoop dfs -ls /usr/joe/wordcount/input/
bin/hadoop dfs -rmr /usr/joe/wordcount/input/conf
bin/hadoop jar ./examples/wordcount.jar org.apache.hadoop.examples.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output
bin/hadoop dfs -cat /usr/joe/wordcount/output/part-r-00000
Hadoop数据库操作例子
测试项目D:\7g\Projects\BMB\Hadoop-Projects\Hadoop-Learning
l 加入驱动ojdbc14.jar到/home/bmb/hadoop-0.20.2/lib,(需要重启)
l 解决不支持oracle的bug: D:\7g\Projects\BMB\Hadoop-Projects\Hadoop-src
http://www.nabble.com/HADOOP-2536-supports-Oracle-too--tp21823199p22471395.html
Oracle 报错:java.io.IOException: ORA-00911: invalid character
修改DBOutputFormat的方法constructQuery(),去掉”;”
HSQL DB
bin/hadoop jar hadoop-*-examples.jar dbcount
bin/hadoop jar ./examples/bmb.jar com.bmb.examples.DBCountPageView
bin/hadoop jar ./examples/bmb.jar com.bmb.examples.DBCountPageView2
Bug:
当有2台以上的Datanode,hadoop分配数据出现重复
URL Access Log访问计算
bin/hadoop dfs -mkdir url-access-log
bin/hadoop dfs -put examples/test-data/*.txt url-access-log
bin/hadoop dfs -ls /user/root/url-access-log
bin/hadoop jar examples/bmb.jar com.bmb.examples.URLFileLogTest url-access-log url-access-log-out1
bin/hadoop dfs -cat /user/root/url-access-log-out1/part-r-00000
examples.StoreTableDataMap
计算文件里面各个用","分隔开的词语的出现次数
Hadoop提供的简化MR的Aggregate框架,做各种数学运算
org.apache.hadoop.mapred.lib.aggregate
org.apache.hadoop.examples.AggregateWordCount