Pig的安装及基本使用

最新推荐文章于 2024-09-19 10:26:13 发布
最新推荐文章于 2024-09-19 10:26:13 发布
阅读量2.6k 收藏
点赞数
文章标签： hadoop pig
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wuxintdrh/article/details/53242124
版权

pig的安装

下载并解压安装包

  在Apache下载最新的Pig软件包，点击下载会推荐最快的镜像站点，以下为下载地址：
   pig下载地址

配置环境

  解压缩到安装路径， 用如下命令编辑/etc/profile文件：
Pig工作模式

本地模式：只需要配置PATH环境变量${PIG_HOME}/bin即可，适用于测试

Mapreduce模式：需要添加环境变量PIG_CLASSPATH=${HADOOP_HOME}/conf/，指向hadoop的conf目录,我的是hadoop2.6 ，hadoop home: /usr/local/hadoop/etc/hadoop

 sudo vi /etc/profile
 添加：
    export PIG_HOME=/app/pig-0.13.0
    export PIG_CLASSPATH=/usr/local/hadoop/etc/hadoop
    export PATH=$PATH:$PIG_HOME/bin

基本使用

将测试数据复制到hdfs上：  测试数据下载

hadoop fs -put ncdc_data.txt  /input/in1/

 使用Pig latin求年最高气温

  1、加载天气数据

 第一次将地址写错了， 导致一直没有找到文件

grunt> A = LOAD '/input/in1/ncdc_data.txt' USING PigStorage(':') AS (year:int, temp:int, quality:int);

 1、过滤数据

grunt> B = FILTER A BY temp != 9999 AND ((chararray)quality matches '[01459]');
或  B = FILTER A BY temp != 9999 AND (quality == 0 OR quality == 1 OR quality == 4 OR quality == 5 OR quality == 9);

  按年分组天气数据

grunt> C = GROUP B BY year;

  逐行扫描数据并求最大值和对应的年份(group)

grunt> D = FOREACH C GENERATE group, MAX(B.temp) AS max_temp;

 输出结果

grunt> DUMP D;
2016-11-20 06:02:41,902 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,FILTER
2016-11-20 06:02:42,053 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-11-20 06:02:42,054 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-11-20 06:02:42,067 [main] INFO  org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2016-11-20 06:02:42,069 [main] INFO  org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2016-11-20 06:02:42,107 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2016-11-20 06:02:42,114 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil - Choosing to move algebraic foreach to combiner
2016-11-20 06:02:42,140 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2016-11-20 06:02:42,140 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2016-11-20 06:02:42,241 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-11-20 06:02:42,250 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:02:42,263 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2016-11-20 06:02:42,278 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2016-11-20 06:02:42,280 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2016-11-20 06:02:42,280 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2016-11-20 06:02:42,308 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=3673672
2016-11-20 06:02:42,308 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2016-11-20 06:02:42,308 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - This job cannot be converted run in-process
2016-11-20 06:02:43,095 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/pig-0.16.0-core-h2.jar to DistributedCache through /tmp/temp-60624248/tmp72750994/pig-0.16.0-core-h2.jar
2016-11-20 06:02:43,367 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/automaton-1.11-8.jar to DistributedCache through /tmp/temp-60624248/tmp-2105835473/automaton-1.11-8.jar
2016-11-20 06:02:43,518 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/antlr-runtime-3.4.jar to DistributedCache through /tmp/temp-60624248/tmp1218719075/antlr-runtime-3.4.jar
2016-11-20 06:02:43,701 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/joda-time-2.9.3.jar to DistributedCache through /tmp/temp-60624248/tmp-2048402576/joda-time-2.9.3.jar
2016-11-20 06:02:43,707 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2016-11-20 06:02:43,710 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2016-11-20 06:02:43,710 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2016-11-20 06:02:43,710 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2016-11-20 06:02:43,840 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2016-11-20 06:02:43,847 [JobControl] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:02:44,029 [JobControl] WARN  org.apache.hadoop.mapreduce.JobSubmitter - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2016-11-20 06:02:44,159 [JobControl] INFO  org.apache.pig.builtin.PigStorage - Using PigTextInputFormat
2016-11-20 06:02:44,172 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2016-11-20 06:02:44,172 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2016-11-20 06:02:44,350 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2016-11-20 06:02:44,709 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2016-11-20 06:02:47,105 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1479576092520_0006
2016-11-20 06:02:47,816 [JobControl] INFO  org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources.
2016-11-20 06:02:53,694 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1479576092520_0006
2016-11-20 06:02:54,016 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: http://TEST:8088/proxy/application_1479576092520_0006/
2016-11-20 06:02:54,017 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1479576092520_0006
2016-11-20 06:02:54,017 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases A,B,C,D
2016-11-20 06:02:54,017 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: A[5,4],A[-1,-1],B[6,4],D[8,4],C[7,4] C: D[8,4],C[7,4] R: D[8,4]
2016-11-20 06:02:54,252 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2016-11-20 06:02:54,252 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0006]
2016-11-20 06:04:53,944 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 5% complete
2016-11-20 06:04:53,945 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0006]
2016-11-20 06:04:56,974 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 21% complete
2016-11-20 06:04:56,978 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0006]
2016-11-20 06:05:04,031 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 33% complete
2016-11-20 06:05:04,031 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0006]
2016-11-20 06:05:24,319 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2016-11-20 06:05:24,320 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0006]
2016-11-20 06:10:06,870 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 66% complete
2016-11-20 06:10:06,870 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0006]
2016-11-20 06:10:14,258 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 83% complete
2016-11-20 06:10:14,258 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0006]
2016-11-20 06:10:22,325 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0006]
2016-11-20 06:11:03,514 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:11:03,646 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:11:49,363 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:11:49,434 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:11:49,883 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:11:49,910 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:11:50,354 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-11-20 06:11:50,367 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
2.6.0   0.16.0  chb     2016-11-20 06:02:42     2016-11-20 06:11:50     GROUP_BY,FILTER

Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime      MinMapTime      AvgMapTime      MedianMapTime   MaxReduceTime   MinReduceTime   AvgReduceTime   MedianReducetime       Alias    Feature Outputs
job_1479576092520_0006  1       1       88      88      88      88      310     310     310     310     A,B,C,D GROUP_BY,COMBINER       hdfs://192.168.1.124:9000/tmp/temp-60624248/tmp-1087782019,

Input(s):
Successfully read 321146 records (3674048 bytes) from: "/input/in1/ncdc_data.txt"

Output(s):
Successfully stored 43 records (430 bytes) in: "hdfs://192.168.1.124:9000/tmp/temp-60624248/tmp-1087782019"

Counters:
Total records written : 43
Total bytes written : 430
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1479576092520_0006


2016-11-20 06:11:50,377 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:11:50,397 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:11:50,554 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:11:50,573 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:11:51,275 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:11:51,349 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:11:52,066 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2016-11-20 06:11:52,068 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-11-20 06:11:52,069 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-11-20 06:11:52,070 [main] INFO  org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2016-11-20 06:11:52,528 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2016-11-20 06:11:52,528 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1901,317)
(1902,261)
(1903,278)
(1904,194)
(1905,278)
(1906,283)
(1907,300)
(1908,322)
(1909,350)
(1910,322)
(1911,322)
(1912,411)
(1913,361)
(1914,378)
(1915,411)
(1916,289)
(1917,478)
(1918,450)
(1919,428)
(1920,344)
(1921,417)
(1922,400)
(1923,394)
(1924,456)
(1925,322)
(1926,411)
(1928,161)
(1929,178)
(1930,311)
(1931,450)
(1932,322)
(1933,411)
(1934,300)
(1935,311)
(1936,389)
(1937,339)
(1938,411)
(1939,433)
(1940,433)
(1941,462)
(1942,278)
(1949,367)
(1953,400)
grunt>

 存储结果到文件

grunt>  STORE D INTO 'max_temp' USING PigStorage(':');
2016-11-20 06:28:32,644 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-11-20 06:28:32,645 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-11-20 06:28:32,925 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
2016-11-20 06:28:33,159 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,FILTER
2016-11-20 06:28:33,444 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-11-20 06:28:33,444 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-11-20 06:28:33,447 [main] INFO  org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2016-11-20 06:28:33,448 [main] INFO  org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2016-11-20 06:28:33,496 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2016-11-20 06:28:33,520 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil - Choosing to move algebraic foreach to combiner
2016-11-20 06:28:33,546 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2016-11-20 06:28:33,546 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2016-11-20 06:28:33,751 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-11-20 06:28:33,773 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:28:33,781 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2016-11-20 06:28:33,804 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2016-11-20 06:28:33,806 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2016-11-20 06:28:33,806 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2016-11-20 06:28:33,826 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=3673672
2016-11-20 06:28:33,826 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2016-11-20 06:28:33,826 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - This job cannot be converted run in-process
2016-11-20 06:28:36,502 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/pig-0.16.0-core-h2.jar to DistributedCache through /tmp/temp-60624248/tmp-1199985731/pig-0.16.0-core-h2.jar
2016-11-20 06:28:36,765 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/automaton-1.11-8.jar to DistributedCache through /tmp/temp-60624248/tmp721246289/automaton-1.11-8.jar
2016-11-20 06:28:37,076 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/antlr-runtime-3.4.jar to DistributedCache through /tmp/temp-60624248/tmp341502194/antlr-runtime-3.4.jar
2016-11-20 06:28:37,560 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/joda-time-2.9.3.jar to DistributedCache through /tmp/temp-60624248/tmp-587981636/joda-time-2.9.3.jar
2016-11-20 06:28:37,567 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2016-11-20 06:28:37,574 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2016-11-20 06:28:37,574 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2016-11-20 06:28:37,574 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2016-11-20 06:28:37,907 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2016-11-20 06:28:37,943 [JobControl] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:28:38,104 [JobControl] WARN  org.apache.hadoop.mapreduce.JobSubmitter - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2016-11-20 06:28:38,208 [JobControl] INFO  org.apache.pig.builtin.PigStorage - Using PigTextInputFormat
2016-11-20 06:28:38,233 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2016-11-20 06:28:38,234 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2016-11-20 06:28:38,249 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2016-11-20 06:28:38,887 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2016-11-20 06:28:39,586 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1479576092520_0007
2016-11-20 06:28:39,610 [JobControl] INFO  org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources.
2016-11-20 06:28:39,843 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1479576092520_0007
2016-11-20 06:28:39,945 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: http://TEST:8088/proxy/application_1479576092520_0007/
2016-11-20 06:28:39,945 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1479576092520_0007
2016-11-20 06:28:39,947 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases A,B,C,D
2016-11-20 06:28:39,947 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: A[5,4],A[-1,-1],B[6,4],D[8,4],C[7,4] C: D[8,4],C[7,4] R: D[8,4]
2016-11-20 06:28:40,011 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2016-11-20 06:28:40,011 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0007]
2016-11-20 06:30:39,691 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 9% complete
2016-11-20 06:30:39,704 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0007]
2016-11-20 06:30:44,340 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 33% complete
2016-11-20 06:30:44,340 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0007]
2016-11-20 06:30:54,464 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2016-11-20 06:30:54,465 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0007]
2016-11-20 06:32:23,937 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 83% complete
2016-11-20 06:32:23,937 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0007]
2016-11-20 06:32:29,164 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1479576092520_0007]
2016-11-20 06:32:51,921 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:32:52,670 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:33:02,007 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:33:02,123 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:33:02,537 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:33:02,561 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:33:02,822 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-11-20 06:33:02,824 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
2.6.0   0.16.0  chb     2016-11-20 06:28:33     2016-11-20 06:33:02     GROUP_BY,FILTER

Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime      MinMapTime      AvgMapTime      MedianMapTime   MaxReduceTime   MinReduceTime   AvgReduceTime   MedianReducetime       Alias    Feature Outputs
job_1479576092520_0007  1       1       35      35      35      35      97      97      97      97      A,B,C,D GROUP_BY,COMBINER       hdfs://192.168.1.124:9000/user/chb/max_temp,

Input(s):
Successfully read 321146 records (3674048 bytes) from: "/input/in1/ncdc_data.txt"

Output(s):
Successfully stored 43 records (387 bytes) in: "hdfs://192.168.1.124:9000/user/chb/max_temp"

Counters:
Total records written : 43
Total bytes written : 387
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1479576092520_0007


2016-11-20 06:33:02,847 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:33:02,884 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:33:03,175 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:33:03,209 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:33:03,469 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at TEST/192.168.1.124:8032
2016-11-20 06:33:03,491 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2016-11-20 06:33:03,725 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
grunt>

 查看结果:

grunt> cat max_temp
1901:317
1902:261
1903:278
1904:194
1905:278
1906:283
1907:300
1908:322
1909:350
1910:322
1911:322
1912:411
1913:361
1914:378
1915:411
1916:289
1917:478
1918:450
1919:428
1920:344
1921:417
1922:400
1923:394
1924:456
1925:322
1926:411
1928:161
1929:178
1930:311
1931:450
1932:322
1933:411
1934:300
1935:311
1936:389
1937:339
1938:411
1939:433
1940:433
1941:462
1942:278
1949:367
1953:400
grunt>
宝哥大数据
关注
专栏目录
Pig安装配置及基本使用
杨鑫newlife的专栏
07-13 1146
Pig安装配置及基本使用
pig安装和配置 详细教程
xq_iris的专栏
02-02 2074
首先，通过winscp工具或者共享文件夹 把pig-0.11.1.tar.gz 复制到 /usr/local目录下 （winscp下载地址 ：  http://download.csdn.net/detail/xq_iris/9413944  ）  ( pig 下载地址：     http://download.csdn.net/detail/xq_iris/9425711）
[pig] pig 基础使用
a84480766的博客
06-08 444
1.pig运行模式 本地模式： pig -x local 直接访问本地磁盘 集群模式： pig 或者 pig -x mapreduce 2.pig latin 交互 帮助信息 help 上传本地文件到hdfs中 查看内容 cat 加载数据 grunt> A = load '/pig/input/a.data' using PigStorag...
Pig简单使用
u014135240的专栏
03-15 728
6.Pig是一个利用mapreduce进行计算框架。有自己的一套语法PIG latin，可以非常方便的书写计算代码。 7.如何使用pig？   解压缩之后，修改文件conf/pig.properties，增加两行内容，如下   fs.default.name=hdfs://hadoop0:9000   mapred.job.tracker=hadoop0:9001   保存退出后，执行b
Pig安装讲解
weixin_34008805的博客
11-07 137
Pig 简介： Pig 是 Apache 项目的一个子项目，Pig 提供了一个支持大规模数据分析的平台，Pig 突出的特点就是它的结构经得起大量并行任务的检验，使得它能够处理大规模数据集 Pig 特点： Pig可简化MapReduce任务的开发 Pig可以看做Hadoop的客户端软件，可以连接到Hadoop集...
大数据学习——数据处理工具Pig入门使用
编程圈子-谢厂节的博客
03-29 6413
简介Pig是一个基于Hadoop的大规模数据分析平台，它提供的SQL-LIKE语言叫Pig Latin，该语言的编译器会把类SQL的数据分析请求转为一系列经过优化处理的MapReduce运算。特点 专注于大量数据集分析 运行在集群的计算架构上，Yahoo Pig提供了多层抽象，简化并行计算让普通用户使用，这些抽象完成自动把用户请求queries翻译成有效的并行评估计划，然后在物理集群上执行这些计划；
Pig的安装
栎梓天冲的专栏
04-14 329
Pig安装 上传文件，解压到指定目录tar –zxvf - pig-0.11.1.tar.gz 对文件进行重命名 mv pig-0.11.1 pig 修改环境变量 vi /etc/profile 添加内容： export PIG_HOME=/opt/pig/bin 修改path内容： $PIG_HOME/bin 重启环境变量：source /etc/profile
大数据技术基础实验报告-pig的安装配置与应用.doc
08-21
在这个实验报告中，我们将深入理解Pig的安装、配置以及基本使用方法。 **一、Pig的安装与配置** 在Linux系统中安装Pig通常涉及以下步骤： 1. **安装Java开发工具（JDK）**：Pig依赖于Java环境，确保系统已安装JDK。...
d2l-zh-1.0.zip_D2L 文件_d2lzh安装_deeplearning_mxnet_pig17v
07-14
"d2l-zh-1.0.zip_D2L 文件_d2lzh安装_deeplearning_mxnet_pig17v" 是该教程的一个版本，它包含了一系列用于学习和实践的代码，主要基于MXNet框架。MXNet是一个高效的深度学习库，支持多种编程语言，如Python，便于...
完整版大数据云计算课程 Hadoop数据分析平台系列课程 Hadoop 08 Pig安装与Pig Latin语言，应用案例 共2
09-29
本课程聚焦于“Hadoop 08 Pig安装与Pig Latin语言”，旨在深入理解Pig的工作原理及其在实际应用中的价值。 Pig拉丁语（Pig Latin）是Pig的编程语言，它允许用户以一种声明性的方式编写数据处理逻辑，而非传统的低级...
Pig安装配置
大头皮鞋的博客
11-21 1991
前提条件： 1. 安装好hadoop2.7.3 安装步骤： 1. 下载pig安装文件，下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/pig/或http://archive.apache.org/dist/pig/ 下载版本：pig-0.17.0.tar.gz 2. 解压 tar -zxvf pig-0.17.0...
Pig相关操作
sinat_41642318的博客
08-17 532
命令行 Pig的常用命令：操作HDFS ls、cd、cat、mkdir、pwd、copyFromLocal（上传）、copyToLocal（下载） sh: 调用操作系统的命令 register、define -----> 部署pig的自定义函数的jar包 使用PigLatin语句分析数据 1、需要启动Yarn的HistoryServer：记录所有执行过的任务 mr-jo...
Pig的详细使用文档和使用案例
qq_37480069的博客
05-19 1597
Pig允许用户编写自定义函数，以便在脚本中使用。自定义函数可以是任何Java函数，并且可以使用Pig提供的API来与Pig交互。try {} } }这个自定义函数使用了Pig提供的EvalFunc类，并实现了exec方法来计算输入字符串的长度。这个脚本首先使用REGISTER命令将自定义函数所在的JAR文件注册到Pig中。然后，它使用DEFINE命令定义了一个名为StringLength的自定义函数。接着，它从文本文件中加载数据，并使用自定义函数计算每个单词的长度。
Pig的安装和使用方法
蝈蝈的博客
07-21 9743
本文使用的pig版本是pig-0.12.0.tar.gz，在安装以前已经安装好了hadoop，hadoop的安装方法参考 hadoop-1.2.1安装方法详解 pig的安装方法很简单，配置一下环境即可，pig有两种工作模式：本地模式和MapReduce模式(默认)。 1、上传并解压pig-0.12.0.tar.gz [hadoop@mdw temp]$ tar zxf pig-0.1
九、pig安装与使用
qq_53488289的博客
08-05 467
解压文件export PIG_HOME=/usr/local/pig。
【WPF】02 按钮控件圆角配置及状态切换
最新发布
kewaqi618的博客
09-19 370
实际开发中的经验总结
Hadoop与HBase伪分布式安装及Pig配置指南
总结：本文详细介绍了在Linux环境下搭建Hadoop、HBase和Pig的基本步骤，涵盖了从JDK安装、Hadoop配置到Hbase伪分布式安装以及Pig的安装和使用。每一步都至关重要，确保了大数据处理框架的正确运行。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值