Pig实例:使用Pig latin来求年最高气温(测试NCDC天气数据)

最新推荐文章于 2022-12-04 17:02:17 发布
最新推荐文章于 2022-12-04 17:02:17 发布
阅读量1.1k 收藏
点赞数
分类专栏： Pig hadoop 文章标签： hadoop pig ncdc
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wuzhongfei88/article/details/43602843
版权
hadoop 同时被 2 个专栏收录
11 篇文章 0 订阅
订阅专栏
Pig
3 篇文章 0 订阅
订阅专栏

hadoop集群模式开启:start-all.sh

    Pig的安装不在陈述,网上有许多可参考的

启动grunt shell

可以使用help命令查看帮助信息:

查看grunt shell命令:

grunt> fs 

准备测试数据(下载测试数据)

以NCDC天气数据求年最大气温为例，准备数据如下（为方便测试每列数据只包含年、气温和数据状态并以冒号分割）：

hadoop@master:~$ head ncdc_data.txt
1953:122:5
1953:83:5
1953:44:5
1953:33:5
1953:50:5
1953:44:5
1953:39:5
1953:33:5
1953:33:5
1953:33:5
hadoop@master:~$ wc -1 ncdc_data.txt
wc: invalid option -- '1'
Try 'wc --help' for more information.
hadoop@master:~$ wc -l ncdc_data.txt
321146 ncdc_data.txt
hadoop@master:~$
在grunt shell中将ncdc_data.txt存入hdfs中 

grunt> copyFromLocal ncdc_data.txt ./  
grunt> ls
hdfs://master:8020/user/hadoop/11	<dir>
hdfs://master:8020/user/hadoop/22	<dir>
hdfs://master:8020/user/hadoop/32	<dir>
hdfs://master:8020/user/hadoop/hadooplog	<dir>
<span style="color:#FF0000;">h<u>dfs://master:8020/user/hadoop/ncdc_data.txt<r 3>	3352528</u></span>
hdfs://master:8020/user/hadoop/output	<dir>
hdfs://master:8020/user/hadoop/passwd<r 3>	2019
grunt>


使用Pig latin求年最高气温加载天气数据


grunt> A = LOAD 'ncdc_data.txt' USING PigStorage(':') AS (year:int, temp:int, quality:int);
过程: 
<span style="color:#FF0000;">grunt> A = LOAD 'ncdc_data.txt' USING PigStorage(':') AS (year:int, temp:int, quality:int);</span>
2015-02-07 16:52:16,351 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-02-07 16:52:16,457 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
<span style="color:#FF0000;">grunt> describe A</span> 
A: {year: int,temp: int,quality: int}
grunt>
过滤数据
grunt> B = FILTER A BY temp != 9999 AND ((chararray)quality matches '[01459]');

或B = FILTER A BY temp != 9999 AND (
                   quality == 0 OR quality == 1 OR quality == 4 OR quality == 5 OR quality == 9);
过程: 
<span style="color:#FF0000;">grunt> B = FILTER A BY temp != 9999 AND ((chararray)quality matches '[01459]');
grunt> describe B</span>
B: {year: int,temp: int,quality: int}
grunt>

按年分组天气数据


grunt> C = GROUP B BY year;
过程: 
<span style="color:#FF0000;">grunt> C = GROUP B BY year;
grunt> describe C </span>         
C: {group: int,B: {(year: int,temp: int,quality: int)}}
grunt>
逐行扫描数据并求最大值和对应的年份(group)


grunt> D = FOREACH C GENERATE group, MAX(B.temp) AS max_temp;
过程: 
<span style="color:#FF0000;">grunt> D = FOREACH C GENERATE group, MAX(B.temp) AS max_temp;
grunt> describe D  </span>                                          
D: {group: int,max_temp: int}
grunt>
输出结果


grunt> DUMP D;
过程: 
grunt> dump D;
2015-02-07 16:56:15,488 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,FILTER
2015-02-07 16:56:15,538 [main] INFO  org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2015-02-07 16:56:15,650 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2015-02-07 16:56:15,652 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2015-02-07 16:56:15,655 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2015-02-07 16:56:15,655 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2015-02-07 16:56:15,691 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-02-07 16:56:15,728 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at master/10.15.43.214:8032
2015-02-07 16:56:15,755 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2015-02-07 16:56:15,756 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2015-02-07 16:56:15,757 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2015-02-07 16:56:15,757 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2015-02-07 16:56:15,758 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=3352528
2015-02-07 16:56:15,759 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2015-02-07 16:56:15,759 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - This job cannot be converted run in-process
2015-02-07 16:56:15,808 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job4573583805590959622.jar
2015-02-07 16:56:21,212 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job4573583805590959622.jar created
2015-02-07 16:56:21,358 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2015-02-07 16:56:21,374 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2015-02-07 16:56:21,374 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2015-02-07 16:56:21,374 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2015-02-07 16:56:21,534 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2015-02-07 16:56:21,536 [JobControl] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at master/10.15.43.214:8032
2015-02-07 16:56:22,728 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2015-02-07 16:56:22,728 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2015-02-07 16:56:22,758 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2015-02-07 16:56:23,029 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2015-02-07 16:56:23,263 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1423285333973_0005
2015-02-07 16:56:24,011 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1423285333973_0005
2015-02-07 16:56:25,174 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: http://master:8088/proxy/application_1423285333973_0005/
2015-02-07 16:56:25,174 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1423285333973_0005
2015-02-07 16:56:25,174 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases A,B,C,D
2015-02-07 16:56:25,174 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: A[1,4],A[-1,-1],B[2,4],D[4,4],C[3,4] C: D[4,4],C[3,4] R: D[4,4]
2015-02-07 16:56:25,589 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2015-02-07 16:56:25,589 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1423285333973_0005]
2015-02-07 16:56:47,391 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 24% complete
2015-02-07 16:56:47,392 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1423285333973_0005]
2015-02-07 16:56:49,394 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2015-02-07 16:56:49,395 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1423285333973_0005]
2015-02-07 16:56:59,908 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1423285333973_0005]
2015-02-07 16:57:05,580 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2015-02-07 16:57:05,581 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 

HadoopVersion	PigVersion	UserId	StartedAt	FinishedAt	Features
2.4.0	0.13.0	hadoop	2015-02-07 16:56:15	2015-02-07 16:57:05	GROUP_BY,FILTER

Success!

Job Stats (time in seconds):
JobId	Maps	Reduces	MaxMapTime	MinMapTIme	AvgMapTime	MedianMapTime	MaxReduceTime	MinReduceTime	AvgReduceTime	MedianReducetime	Alias	Feature	Outputs
job_1423285333973_0005	1	1	11	11	11	11	9	99	9	A,B,C,D	GROUP_BY,COMBINER	hdfs://master:8020/tmp/temp624427084/tmp-368727104,

Input(s):
Successfully read 321146 records (3352891 bytes) from: "hdfs://master:8020/user/hadoop/ncdc_data.txt"

Output(s):
<span style="color:#FF0000;">Successfully stored 43 records (430 bytes) in: "hdfs://master:8020/tmp/temp624427084/tmp-368727104"</span>

Counters:
Total records written : 43
Total bytes written : 430
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1423285333973_0005


2015-02-07 16:57:05,604 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2015-02-07 16:57:05,623 [main] INFO  org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2015-02-07 16:57:05,645 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2015-02-07 16:57:05,645 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1901,317)
(1902,261)
(1903,278)
(1904,194)
(1905,278)
(1906,283)
(1907,300)
(1908,322)
(1909,350)
(1910,322)
(1911,322)
(1912,411)
(1913,361)
(1914,378)
(1915,411)
(1916,289)
(1917,478)
(1918,450)
(1919,428)
(1920,344)
(1921,417)
(1922,400)
(1923,394)
(1924,456)
(1925,322)
(1926,411)
(1928,161)
(1929,178)
(1930,311)
(1931,450)
(1932,322)
(1933,411)
(1934,300)
(1935,311)
(1936,389)
(1937,339)
(1938,411)
(1939,433)
(1940,433)
(1941,462)
(1942,278)
(1949,367)
(1953,400)
grunt>
存储结果到文件


grunt> STORE D INTO 'max_temp' USING PigStorage(':');
过程: 
grunt> STORE D INTO 'max_temp' USING PigStorage(':');
2015-02-07 16:59:53,558 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-02-07 16:59:53,600 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,FILTER
2015-02-07 16:59:53,603 [main] INFO  org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2015-02-07 16:59:53,610 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2015-02-07 16:59:53,611 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2015-02-07 16:59:53,613 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2015-02-07 16:59:53,613 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2015-02-07 16:59:53,628 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-02-07 16:59:53,631 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at master/10.15.43.214:8032
2015-02-07 16:59:53,633 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2015-02-07 16:59:53,633 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2015-02-07 16:59:53,634 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2015-02-07 16:59:53,634 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2015-02-07 16:59:53,637 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=3352528
2015-02-07 16:59:53,637 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2015-02-07 16:59:53,637 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - This job cannot be converted run in-process
2015-02-07 16:59:53,655 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job6820753826732922606.jar
2015-02-07 16:59:57,504 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job6820753826732922606.jar created
2015-02-07 16:59:57,539 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2015-02-07 16:59:57,545 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2015-02-07 16:59:57,545 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2015-02-07 16:59:57,545 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2015-02-07 16:59:57,581 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2015-02-07 16:59:57,583 [JobControl] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at master/10.15.43.214:8032
2015-02-07 16:59:58,411 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2015-02-07 16:59:58,411 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2015-02-07 16:59:58,413 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2015-02-07 16:59:58,557 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2015-02-07 16:59:58,731 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1423285333973_0006
2015-02-07 16:59:58,797 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1423285333973_0006
2015-02-07 16:59:58,801 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: http://master:8088/proxy/application_1423285333973_0006/
2015-02-07 16:59:58,802 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1423285333973_0006
2015-02-07 16:59:58,802 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases A,B,C,D
2015-02-07 16:59:58,802 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: A[1,4],A[-1,-1],B[2,4],D[4,4],C[3,4] C: D[4,4],C[3,4] R: D[4,4]
2015-02-07 16:59:58,807 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2015-02-07 16:59:58,807 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1423285333973_0006]
2015-02-07 17:00:18,496 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2015-02-07 17:00:18,496 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1423285333973_0006]
2015-02-07 17:00:26,004 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1423285333973_0006]
2015-02-07 17:00:29,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2015-02-07 17:00:29,139 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 

HadoopVersion	PigVersion	UserId	StartedAt	FinishedAt	Features
2.4.0	0.13.0	hadoop	2015-02-07 16:59:53	2015-02-07 17:00:29	GROUP_BY,FILTER

Success!

Job Stats (time in seconds):
JobId	Maps	Reduces	MaxMapTime	MinMapTIme	AvgMapTime	MedianMapTime	MaxReduceTime	MinReduceTime	AvgReduceTime	MedianReducetime	Alias	Feature	Outputs
job_1423285333973_0006	1	1	10	10	10	10	4	44	4	A,B,C,D	GROUP_BY,COMBINER	hdfs://master:8020/user/hadoop/max_temp,

Input(s):
Successfully read 321146 records (3352891 bytes) from: "hdfs://master:8020/user/hadoop/ncdc_data.txt"

Output(s):
<span style="color:#FF0000;">Successfully stored 43 records (387 bytes) in: "hdfs://master:8020/user/hadoop/max_temp"</span>

Counters:
Total records written : 43
Total bytes written : 387
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1423285333973_0006


2015-02-07 17:00:29,162 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
grunt>
查看结果


grunt> cat max_temp
过程: 

grunt> cat max_temp
1901:317
1902:261
1903:278
1904:194
1905:278
1906:283
1907:300
1908:322
1909:350
1910:322
1911:322
1912:411
1913:361
1914:378
1915:411
1916:289
1917:478
1918:450
1919:428
1920:344
1921:417
1922:400
1923:394
1924:456
1925:322
1926:411
1928:161
1929:178
1930:311
1931:450
1932:322
1933:411
1934:300
1935:311
1936:389
1937:339
1938:411
1939:433
1940:433
1941:462
1942:278
1949:367
1953:400
grunt>

江南浙里
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
NCDC气象数据的提取与处理（二）范例数据
01-03
NCDC作为全球最重要的气候和天气数据存储库之一，提供了丰富的历史气象信息，供研究者和开发者进行各种分析和应用。这个“NCDC气象数据的提取与处理（二）范例数据”旨在为学习者提供一个实际操作的平台，涵盖了从...
MapReduce案例——求每年最高气温
zzf135的博客
04-23 3017
MapReduce的逻辑数据流 第一步、需要一个 map函数： import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; //hadoop 本身提供了一套可优化网络序列化传输的基本类型，而不直接使用java内嵌的类型 import org.apache.hadoop.mapre
mapreduce编程（二）－ 大象书中求每一年的最高温度
和大黄的博客
09-17 5574
书上的例子是为了取出一年当中气温最高的值，那么将年份和气温做了一个复合的key.  1 通过设置了partitioner来进行分区。因为分区是按照年份来进行，所以同年的数据就可以分区到一个reducer中。 2 自定义key比较器，按照年份升序，温度值降序。这样map输出的所
大数据平台运维之Pig
KamRoseLee的博客
05-11 2093
Pig 42.在master节点安装Pig Clients，打开Linux Shell以MapReduce模式启动它的Grunt，将启动命令和启动结果显示如下。简写：[root@master ~]# pigWARNING: Use "yarn jar" to launch YARNapplications.17/05/07 07:58:29 INFO pig.ExecTypeProvider: T...
Hadoop学习笔记（一）之示例程序：计算每年的最高温度MaxTemperature
diaoqi5743的博客
12-07 606
　　本《hadoop学习笔记》系列是在《hadoop: the definitive guide 3th》的基础上通过网上额外搜集资料和查看hadoop的API再加上自己的实践方面的理解编写而成的，主要针对hadoop的特性和功能学习以及Hadoop生态圈中的其他工具（如Pig，Hive，Hbase，Avro等等）。另外设计到hadoop编程方面的请查阅另一个笔记系列：《Hadoop编程...
统计全球每年的最高/最低气温
BIPT919的博客
05-16 4513
文章目录数据准备1.下载数据2.处理数据一、统计全球每年的最高气温和最低气温1.YearMaxTAndMinT2.Mapper:MaxTAndMinTMapper3.Combiner:MaxTAndMinTCombiner4.Reducer:MaxTAndMinTReducer5.运行代码:MaxTAndMinT.java6.输出结果二、筛选气温在15~25°C之间的数据1.MTAMTMapper类2.MTAMTReducer类3.MTAMT类4.输出结果总结 数据准备 先启动hadoop，然后打开ecl
weathertinturf:使用 turf.js 和 NOAA NCDC 天气可视化天气模式
06-09
可视化 2015 年 2 月的天气数据在美国大部分地区，二月是一个特别寒冷的月份。 对于俄勒冈州的我们来说，我们一直享受着温暖的天气……但更远的东部大部分地区都出现了创纪录的低点和大雪。 我一直对寻找可视化这些...
中国2020年均气温数据点加栅格.zip
12-09
ncdc处理结果，2020年均气温数据
Hadoop权威指南 NCDC气象数据
09-23
本指南将深入探讨如何使用Hadoop处理来自美国国家气候数据中心（NCDC）的气象数据，这些数据涵盖了长达52年的气象历史记录，为科学研究和业务决策提供了宝贵的信息。 首先，NCDC是全球最大的气象和海洋学资料库之一...
美国ncdc官网气象数据集1901至1942年数据
07-03
《美国ncdc官网气象数据集1901至1942年数据》 这个数据集是大数据领域中一个极具价值的学习资源，源自美国国家气候数据中心（National Centers for Environmental Information, NCEI，原名为NCDC）。NCEI是美国国家...
Pig安装与Pig Latin语言，应用案例.
12-03
来试试Pig安装,与学学Pig Latin语言，玩玩应用案例
MR案例 - 求年度最高气温
howard2005的专栏
12-04 775
求年度最高温度
MapReduce分析年气象数据平均温度与最高温度（云计算实验作业20180523）
qq_40276310的博客
05-23 7331
课程原地址：http://hbust.shiyanbar.com/course/91079上课老师：李歆实验时间：20180523地点：云桌面实验人：郭畅 【实验目的】1) 理解mapreduce执行原理2) 理解map,reduce阶段3) 熟悉map和reduce代码的编写【实验原理】把采集的气象数据信息以日志的方式保存到指定的位置，该位置可以是本地，也可以是hdfs分布式系统上，利用hado...
部分大数据平台运维题及答案
zhangbaoan_abc的博客
12-09 1万+
大数据平台运维 HDFS题： 1.在HDFS文件系统的根目录下创建递归目录“1daoyun/file”，将附件中的BigDataSkills.txt文件，上传到1daoyun/file目录中，使用相关命令查看文件系统中1daoyun/file目录的文件列表信息，将操作命令和返回结果以文本形式提交到答题框。 [root@master ~]# hadoop fs -mkdir -p /1daoyun/...
Pig Latin示例
蔡先生的专栏
03-07 854
1.创建并上传数据到HDFS 2.加载 grunt> records = LOAD 'hdfs://127.0.0.1:8020/pig/pigtestdata' AS (year:chararray,temperature:int,quality:int); grunt> DUMP records; grunt> DESCRIBE records
Pig安装及简单实例
热门推荐
zythy的专栏
01-10 1万+
前面讲到了如何用MapReduce进行数据分析。当业务比较复杂的时候，使用MapReduce将会是一个很复杂的事情，比如你需要对数据进行很多预处理或转换，以便能够适应MapReduce的处理模式。另一方面，编写MapReduce程序，发布及运行作业都将是一个比较耗时的事情。 Pig的出现很好的弥补了这一不足。Pig能够让你专心于数据及业务本身，而不是纠结于数据的格式转换以及MapReduce程序编写。
第8周 Pig安装与Pig Latin语言，应用案例
事后诸葛亮的程序人生（微信：zq9017197）
01-15 2458
pig 简介 Pig可以看做hadoop的客户端软件，可以连接到hadoop集群进行数据分析工作 Pig方便不熟悉java的用户，使用一种较为简便的类似于SQL的面向数据流的语言pig latin进行数据处理 Pig latin可以进行排序、过滤、求和、分组、关联等常用操作，还可以自定义函数，这是一种面向数据分析处理的轻量级脚本语言 Pig可以看做是pig latin到map-
hadoop统计全球每年的最高气温和最低气温
weixin_46034600的博客
12-11 5096
1、课程设计题目及要求 （1）从ftp://ftp.ncdc.noaa.gov/pub/data/gsod下载2014到2018年的天气数据，然后对数据进行清洗，仅保留日期和当天的气温，数据文件保存为temperature.txt； （2）根据temperature.txt，统计全球每年的最高气温和最低气温；要求：按每年每月统计最高和最低气温；按每年每月统计平均气温，并按年月由近到远排序；按年月筛选15-25度之间的气温数据，并按2014到2016年分别存储到3个文件中。 （3）需要提交纸质作业报告，以及程
pig将查询翻译为MapReduce作业
侯上校
07-17 248
wget http://mirrors.cnnic.cn/apache/pig/pig-0.13.0/pig-0.13.0-src.tar.gz tar -xvf pig-0.13.0-src.tar.gz hadoop@hadoop:~$ vim .bashrc export ANT_OPTS="-Dhttp.proxyHost=proxy.com -Dhttp.proxyPort...
使用MapReduce处理NCDC气象数据
最新发布
06-12
NCDC气象数据是指由美国国家气象数据中心（NCDC）收集的大规模气象数据，包括气温、降雨量等。下面是使用MapReduce处理NCDC气象数据的步骤： 1. Map阶段：将NCDC气象数据分成多个小文件，并将每个小文件分配给不同...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值