hello pig 入门

appadmin@hadoop-namenode1:/data/projects/clusters/pig-0.9.2/bin$ ./pig 
Warning: $HADOOP_HOME is deprecated.

2014-12-08 16:49:10,442 [main] INFO  org.apache.pig.Main - Logging error messages to: /data/projects/clusters/pig-0.9.2/bin/pig_1418028550440.log
2014-12-08 16:49:10,665 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://hadoop-namenode1:9000
2014-12-08 16:49:10,959 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: hadoop-namenode1:9001
grunt> records = load '/user/appadmin1/input03/temp02.txt' as (year: chararray,temperature: int);
grunt> dump records;
2014-12-08 16:49:27,656 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2014-12-08 16:49:27,838 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-12-08 16:49:27,868 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-12-08 16:49:27,869 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-12-08 16:49:27,946 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-12-08 16:49:27,962 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-12-08 16:49:27,964 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2683774938576550508.jar
2014-12-08 16:49:30,281 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2683774938576550508.jar created
2014-12-08 16:49:30,296 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-12-08 16:49:30,331 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-12-08 16:49:30,627 [Thread-6] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-12-08 16:49:30,627 [Thread-6] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2014-12-08 16:49:30,640 [Thread-6] WARN  org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is available
2014-12-08 16:49:30,641 [Thread-6] INFO  org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
2014-12-08 16:49:30,641 [Thread-6] INFO  org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library loaded
2014-12-08 16:49:30,643 [Thread-6] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-12-08 16:49:30,832 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-12-08 16:49:31,370 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201405201827_0949
2014-12-08 16:49:31,370 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://hadoop-namenode1:50030/jobdetails.jsp?jobid=job_201405201827_0949
2014-12-08 16:49:37,397 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2014-12-08 16:49:40,941 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-12-08 16:49:40,944 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion	PigVersion	UserId	StartedAt	FinishedAt	Features
1.1.1	0.9.2	appadmin	2014-12-08 16:49:27	2014-12-08 16:49:40	UNKNOWN

Success!

Job Stats (time in seconds):
JobId	Maps	Reduces	MaxMapTime	MinMapTIme	AvgMapTime	MaxReduceTime	MinReduceTime	AvgReduceTime	Alias	Feature	Outputs
job_201405201827_0949	1	0	3	3	3	0	0	0	records	MAP_ONLY	hdfs://hadoop-namenode1:9000/tmp/temp-355337597/tmp213760747,

Input(s):
Successfully read 6 records (430 bytes) from: "/user/appadmin1/input03/temp02.txt"

Output(s):
Successfully stored 6 records (85 bytes) in: "hdfs://hadoop-namenode1:9000/tmp/temp-355337597/tmp213760747"

Counters:
Total records written : 6
Total bytes written : 85
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201405201827_0949


2014-12-08 16:49:40,953 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2014-12-08 16:49:40,962 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-12-08 16:49:40,963 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1990,21)
(1990,18)
(1991,21)
(1992,30)
(1992,999)
(1990,23)
grunt> describe records;
records: {year: chararray,temperature: int}
grunt> valid_records = filter records by temperature!=999;
grunt> grouped_records = group valid_records by year;
grunt> dump grouped_records;
2014-12-08 16:49:41,065 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,FILTER
2014-12-08 16:49:41,077 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-12-08 16:49:41,084 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-12-08 16:49:41,084 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-12-08 16:49:41,086 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-12-08 16:49:41,087 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-12-08 16:49:41,088 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job8508248355278349679.jar
2014-12-08 16:49:43,199 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job8508248355278349679.jar created
2014-12-08 16:49:43,206 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-12-08 16:49:43,212 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=49
2014-12-08 16:49:43,212 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-12-08 16:49:43,247 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-12-08 16:49:43,391 [Thread-17] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-12-08 16:49:43,391 [Thread-17] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2014-12-08 16:49:43,393 [Thread-17] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-12-08 16:49:43,748 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201405201827_0950
2014-12-08 16:49:43,748 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://hadoop-namenode1:50030/jobdetails.jsp?jobid=job_201405201827_0950
2014-12-08 16:49:43,750 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-12-08 16:49:48,770 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2014-12-08 16:49:56,799 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 66% complete
2014-12-08 16:50:03,839 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-12-08 16:50:03,839 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion	PigVersion	UserId	StartedAt	FinishedAt	Features
1.1.1	0.9.2	appadmin	2014-12-08 16:49:41	2014-12-08 16:50:03	GROUP_BY,FILTER

Success!

Job Stats (time in seconds):
JobId	Maps	Reduces	MaxMapTime	MinMapTIme	AvgMapTime	MaxReduceTime	MinReduceTime	AvgReduceTime	Alias	Feature	Outputs
job_201405201827_0950	1	1	2	2	2	9	9	9	grouped_records,records,valid_records	GROUP_BY    hdfs://hadoop-namenode1:9000/tmp/temp-355337597/tmp-1370881103,

Input(s):
Successfully read 6 records (430 bytes) from: "/user/appadmin1/input03/temp02.txt"

Output(s):
Successfully stored 3 records (97 bytes) in: "hdfs://hadoop-namenode1:9000/tmp/temp-355337597/tmp-1370881103"

Counters:
Total records written : 3
Total bytes written : 97
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201405201827_0950


2014-12-08 16:50:03,849 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2014-12-08 16:50:03,854 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-12-08 16:50:03,854 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1990,{(1990,21),(1990,18),(1990,23)})
(1991,{(1991,21)})
(1992,{(1992,30)})
grunt> describe grouped_records;
grouped_records: {group: chararray,valid_records: {(year: chararray,temperature: int)}}
grunt> max_temperature = foreach grouped_records generate group,MAX(valid_records.temperature);
grunt> 
grunt> dump max_temperature 
2014-12-08 16:50:03,945 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,FILTER
2014-12-08 16:50:03,953 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-12-08 16:50:03,956 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2014-12-08 16:50:03,963 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-12-08 16:50:03,964 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-12-08 16:50:03,966 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-12-08 16:50:03,967 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-12-08 16:50:03,968 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job4768516553316242620.jar
2014-12-08 16:50:06,093 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job4768516553316242620.jar created
2014-12-08 16:50:06,099 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-12-08 16:50:06,108 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=49
2014-12-08 16:50:06,108 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2014-12-08 16:50:06,125 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-12-08 16:50:06,256 [Thread-27] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-12-08 16:50:06,256 [Thread-27] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2014-12-08 16:50:06,257 [Thread-27] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-12-08 16:50:06,625 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201405201827_0951
2014-12-08 16:50:06,626 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://hadoop-namenode1:50030/jobdetails.jsp?jobid=job_201405201827_0951
2014-12-08 16:50:06,627 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-12-08 16:50:11,143 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2014-12-08 16:54:37,042 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2014-12-08 17:00:24,193 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 66% complete
2014-12-08 17:00:31,725 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-12-08 17:00:31,726 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion	PigVersion	UserId	StartedAt	FinishedAt	Features
1.1.1	0.9.2	appadmin	2014-12-08 16:50:03	2014-12-08 17:00:31	GROUP_BY,FILTER

Success!

Job Stats (time in seconds):
JobId	Maps	Reduces	MaxMapTime	MinMapTIme	AvgMapTime	MaxReduceTime	MinReduceTime	AvgReduceTime	Alias	Feature	Outputs
job_201405201827_0951	1	1	2	2	2	9	9	9	grouped_records,max_temperature,records,valid_recordGROUP_BY,COMBINER	hdfs://hadoop-namenode1:9000/tmp/temp-355337597/tmp-675408653,

Input(s):
Successfully read 6 records (430 bytes) from: "/user/appadmin1/input03/temp02.txt"

Output(s):
Successfully stored 3 records (42 bytes) in: "hdfs://hadoop-namenode1:9000/tmp/temp-355337597/tmp-675408653"

Counters:
Total records written : 3
Total bytes written : 42
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201405201827_0951


2014-12-08 17:00:31,734 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2014-12-08 17:00:31,738 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-12-08 17:00:31,738 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1990,23)
(1991,21)
(1992,30)

<span style="font-family:Arial, Helvetica, sans-serif;"><span style="white-space: normal;">
</span></span>