Pig初探

Pig环境安装

pig的安装非常简单，将pig-0.14.0.tar.gz解压到合适的目录下

tar -zxvf pig-0.14.0.tar.gz

修改环境变量：

#pig  
export PIG_HOME=/usr/local/cloud/pig-0.11.1/pig-0.11.1
export PATH=.:$PIG_HOME/bin:$PATH
export PIG_CLASSPATH=$HADOOP_HOME/etc/hadoop   #为了让pig识别你的hadoop版本，如果只想用pig的本地模式，不用配置这项。

使环境变量修改生效：
source /etc/profile

本地模式启动方式pig -x local

[root@leaf pig-0.11.1]# pig -x local
2014-11-24 07:50:19,622 [main] INFO  org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
2014-11-24 07:50:19,622 [main] INFO  org.apache.pig.Main - Logging error messages to: /usr/local/cloud/pig-0.11.1/pig-0.11.1/logs/pig_1416844219621.log
2014-11-24 07:50:19,663 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2014-11-24 07:50:19,901 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-11-24 07:50:19,903 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2014-11-24 07:50:19,907 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-11-24 07:50:19,907 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-11-24 07:50:20,188 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-11-24 07:50:20,190 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

修改pig默认的日志存放目录
pig解压后的目录下pig-0.14.0/conf/pig.properties中，修改为合适的目录：

pig.logfile=/usr/local/cloud/pig-0.11.1/pig-0.11.1/logs

pig的mapreduce启动方式(必须启动hadoop)：

[root@leaf pig-0.11.1]# pig
2014-11-24 07:57:16,370 [main] INFO  org.apache.pig.Main - Apache Pig version <span style="font-family: Arial, Helvetica, sans-serif;">0.11.1</span><span style="font-family: Arial, Helvetica, sans-serif;">(r1459641) compiled Mar 22 2013, 02:13:53</span>
2014-11-24 07:57:16,370 [main] INFO  org.apache.pig.Main - Logging error messages to: /usr/local/cloud/pig-0.11.1/pig-0.11.1/logs/pig_1416844636369.log
2014-11-24 07:57:16,410 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2014-11-24 07:57:16,681 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://192.168.1.240:9000
2014-11-24 07:57:16,684 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2014-11-24 07:57:16,685 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-11-24 07:57:17,634 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

测试是否安装成功：

grunt> ls hdfs:///
hdfs://192.168.1.240:9000/source	<dir>
hdfs://192.168.1.240:9000/testdata	<dir>
hdfs://192.168.1.240:9000/tmp	<dir>
hdfs://192.168.1.240:9000/user	<dir>
hdfs://192.168.1.240:9000/usr	<dir>

pig常见命令：

grunt> cd /user/root/output

grunt> ls
hdfs://192.168.1.240:9000/user/root/output/_policy<r 1>	194
hdfs://192.168.1.240:9000/user/root/output/clusteredPoints	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-0	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-1	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-10-final	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-2	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-3	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-4	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-5	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-6	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-7	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-8	<dir>
hdfs://192.168.1.240:9000/user/root/output/clusters-9	<dir>
hdfs://192.168.1.240:9000/user/root/output/data	<dir>
hdfs://192.168.1.240:9000/user/root/output/random-seeds	<dir>

grunt> pig -help
2014-11-24 08:04:11,969 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <IDENTIFIER> "pig "" at line 1, column 1.
Was expecting one of:
    <EOF> 
    "cat" ...
    "clear" ...
    "fs" ...
    "sh" ...
    "cd" ...
    "cp" ...
    "copyFromLocal" ...
    "copyToLocal" ...
    "dump" ...
    "describe" ...
    "aliases" ...
    "explain" ...
    "help" ...
    "history" ...
    "kill" ...
    "ls" ...
    "mv" ...
    "mkdir" ...
    "pwd" ...
    "quit" ...
    "register" ...
    "rm" ...
    "rmf" ...
    "set" ...
    "illustrate" ...
    "run" ...
    "exec" ...
    "scriptDone" ...
    "" ...
    "" ...
    <EOL> ...
    ";" ...