使用pig的注意事项
1.pig中dump和store的区别:
dump和store的作用都是将pig处理的结果进行输出,前者是打印结果,后者是将结果输出至hdfs文件系统中。
(1).Dump表示将数据打印出来,输出所有的元祖,会带上”( )”。
Eg:(zhangsan,22,男)
(2).Store表示将数据输出至hdfs,输出时不会带上”( )”。
Eg:zhangsan22,男
2.pig中组函数的使用:
在关系型数据库的标准sql中,可以不分组而直接使用组函数(比如SUM),这是因为sql引擎会把该表的所有记录当作一个大组对待。但pig中组函数在使用前必须先显示地进行分组才行。
eg:有一个test表,里面有字段name,count,现在要统计test表中的count字段的和,那么可以通过下面简单的sql来做:
select SUM(count) as sum;
如果在pig中那就得先手动的分组了,如何手动分组?只要给每个元祖加上一条内容相同的字段,然后按照该字段进行分组,然后统计分组内的count之和。过程如下:
data = load 'test.csv' as(name:chararray,count:int);
tmp01 = foreach data generate 'pigIsVeryGood' as id,name,count;
tmp02 = group tmp01 by id;
result = foreach tmp02 generate SUM($1.count) as sum;
dump result;
如果要实现输出如下形式的结果:name,count,sum,那么如何做?用sql比较简单:select name,count,sum(count)。
如果用pig,如何实现?思路是在上例的基础上在result上面再加一个字段id,内容为"pigIsVeryGood",过程如下:
data = load 'test.csv' as(name:chararray,count:int);
tmp01 = foreach data generate 'pigIsVeryGood' as id,name,count;
tmp02 = group tmp01 by id;
result = foreach tmp02 generate SUM($1.count) as sum;
dump result;
tmp03 = foreach resultgenerate 'pigIsVeryGood' as id,sum;
tmp04 = join tmp01 by id,tmp03 by id;
finalResult= foreach tmp04 generate name,count,sum;
dump finalResult;
3.Filter使用
是把符合条件的保留下,非符合条件的过滤掉
eg:info = Filter studentsInfo BY (age > 18)
把大于18岁的保留下,小于或等于18岁的过滤掉
4.引用运算符(Dereference Operators.)与解歧义运算符(Disambiguate Operator个人感觉还是叫“域运算符”吧 ::)的区别及使用说明:
(1).引用运算符(Dereference Operators.)
Dereference Operators
Description
Operator | Symbol | Notes |
tuple dereference | tuple.id or tuple.(id,…) | Tuple dereferencing can be done by name (tuple.field_name) or position (mytuple.$0). If a set of fields are dereferenced (tuple.(name1, name2) or tuple.($0, $1)), the expression represents a tuple composed of the specified fields. Note that if the dot operator is applied to a bytearray, the bytearray will be assumed to be a tuple. |
bag dereference | bag.id or bag.(id,…) | Bag dereferencing can be done by name (bag.field_name) or position (bag.$0). If a set of fields are dereferenced (bag.(name1, name2) or bag.($0, $1)), the expression represents a bag composed of the specified fields. |
map dereference | map#'key' | Map dereferencing must be done by key (field_name#key or $0#key). If the pound operator is applied to a bytearray, the bytearray is assumed to be a map. If the key does not exist, the empty string is returned. |
Examples
Tuple Example
Suppose we have relation A.
LOAD 'data' as (f1:int, f2:tuple(t1:int,t2:int,t3:int)); DUMP A; (1,(1,2,3)) (2,(4,5,6)) (3,(7,8,9)) (4,(1,4,7)) (5,(2,5,8))
In this example dereferencing is used to retrieve two fields from tuple f2.
X = FOREACH A GENERATE f2.t1,f2.t3; DUMP X; (1,3) (4,6) (7,9) (1,7) (2,8)
Bag Example
Suppose we have relation B, formed by grouping relation A (see the GROUP operator for information about the field names in relation B).
A = LOAD 'data' AS (f1:int, f2:int,f3:int); DUMP A; (1,2,3) (4,2,1) (8,3,4) (4,3,3) (7,2,5) (8,4,3) B = GROUP A BY f1; DUMP B; (1,{(1,2,3)}) (4,{(4,2,1),(4,3,3)}) (7,{(7,2,5)}) (8,{(8,3,4),(8,4,3)}) ILLUSTRATE B; etc … ---------------------------------------------------------- | b | group: int | a: bag({f1: int,f2: int,f3: int}) | ----------------------------------------------------------
In this example dereferencing is used with relation X to project the first field (f1) of each tuple in the bag (a).
X = FOREACH B GENERATE a.f1; DUMP X; ({(1)}) ({(4),(4)}) ({(7)}) ({(8),(8)})
Tuple/Bag Example
Suppose we have relation B, formed by grouping relation A (see the GROUP operator for information about the field names in relation B).
A = LOAD 'data' AS (f1:int, f2:int, f3:int); DUMP A; (1,2,3) (4,2,1) (8,3,4) (4,3,3) (7,2,5) (8,4,3) B = GROUP A BY (f1,f2); DUMP B; ((1,2),{(1,2,3)}) ((4,2),{(4,2,1)}) ((4,3),{(4,3,3)}) ((7,2),{(7,2,5)}) ((8,3),{(8,3,4)}) ((8,4),{(8,4,3)}) ILLUSTRATE B; etc … ------------------------------------------------------------------------------- | b | group: tuple({f1: int,f2: int}) | a: bag({f1: int,f2: int,f3: int}) | ------------------------------------------------------------------------------- | | (8, 3) | {(8, 3, 4), (8, 3, 4)} | -------------------------------------------------------------------------------
In this example dereferencing is used to project a field (f1) from a tuple (group) and a field (f1) from a bag (a).
X = FOREACH B GENERATE group.f1, a.f1; DUMP X; (1,{(1)}) (4,{(4)}) (4,{(4)}) (7,{(7)}) (8,{(8)}) (8,{(8)})
Map Example
Suppose we have relation A.
A = LOAD 'data' AS (f1:int, f2:map[]); DUMP A; (1,[open#apache]) (2,[apache#hadoop]) (3,[hadoop#pig]) (4,[pig#grunt])
In this example dereferencing is used to look up the value of key 'open'.
X = FOREACH A GENERATE f2#'open'; DUMP X; (apache) () () ()(2).解歧义运算符(Disambiguate Operator个人感觉还是叫“域运算符”吧 ::
Disambiguate Operator
Use the disambiguate operator ( :: ) to identify field names after JOIN, COGROUP, CROSS, or FLATTEN operators.
In this example, to disambiguate y, use A::y or B::y. In cases where there is no ambiguity, such as z, the :: is not necessary but is still supported.
A = load 'data1' as (x, y); B = load 'data2' as (x, y, z); C = join A by x, B by x; D = foreach C generate A::x,A::y,B::x,B::y,B::z;
未完待续.......