Hadoop 之Pig的安装的与配置之遇到的问题---待解决

Hadoop 之Pig的安装的与配置之遇到的问题---待解决

1. 前提是hadoop集群已经配置完成并且可以正常启动;以下是我的配置方案:

首先配置vim /etc/hosts

192.168.1.64 xuegod64

192.168.1.65 xuegod65

192.168.1.63 xuegod63

(将配置好的文件拷贝到其他两台机器,我是在xuegod64上配置的,使用scp /etc/hosts xuegod63:/etc/进行拷贝,进行该步骤前提是已经配置好SSH免密码登录;关于SSH免密码登录在此就不再详说了)

2.准备安装包如下图

[hadoop@xuegod64 ~]$ ls

hadoop-2.4.1.tar.gz

pig-0.15.0.tar.gz

jdk-8u66-linux-x64.rpm

zookeeper-3.4.7.tar.gz(可以不用)

image

3.配置/etc/profile

[hadoop@xuegod64 ~]$ vim /etc/profile #前提是使用root用户将编辑此文件的权限赋予hadoop用户

export JAVA_HOME=/usr/java/jdk1.8.0_66/

export HADOOP_HOME=/home/hadoop/hadoop-2.4.1/

export HBASE_HOME=/home/hadoop/hbase-1.1.2/

export ZOOKEEPER_HOME=/home/hadoop/zookeeper-3.4.7/

export PIG_HOME=/home/hadoop/pig-0.15.0/

export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOO

KEEPER_HOME/bin:$HBASE_HOME/bin:$PIG_HOME/bin:$PATH

clip_image002

4.检验pig是否配置成功

[hadoop@xuegod64 ~]$ pig -help

Apache Pig version 0.15.0 (r1682971)

compiled Jun 01 2015, 11:44:35

USAGE: Pig [options] [-] : Run interactively in grunt shell.

Pig [options] -e[xecute] cmd [cmd ...] : Run cmd(s).

Pig [options] [-f[ile]] file : Run cmds found in file.

5.Pig执行模式

Pig有两种执行模式,分别为:

1)本地模式(Local)本地模式下,Pig运行在单一的JVM中,可访问本地文件。该模式适用于处理小规模数据或学习之用。

运行以下命名设置为本地模式:

pig –x local

2)MapReduce模式在MapReduce模式下,Pig将查询转换为MapReduce作业提交给Hadoop(可以说群集 ,也可以说伪分布式)。应该检查当前Pig版本是否支持你当前所用的Hadoop版本。某一版本的Pig仅支持特定版本的Hadoop,你可以通过访问Pig官网获取版本支持信息。

Pig会用到HADOOP_HOME环境变量。如果该变量没有设置,Pig也可以利用自带的Hadoop库,但是这样就无法保证其自带肯定库和你实际使用的HADOOP版本是否兼容,所以建议显式设置HADOOP_HOME变量。且还需要设置如下变量:

export PIG_CLASSPATH=$HADOOP_HOME/etc/hadoop

Pig默认模式是mapreduce,你也可以用以下命令进行设置:

[hadoop@xuegod64 ~]$ pig -x mapreduce

(中间略)

grunt>

下一步,需要告诉Pig它所用Hadoop集群的Namenode和Jobtracker。一般情况下,正确安装配置Hadoop后,这些配置信息就已经可用了,不需要做额外的配置

6.运行Pig程序
Pig程序执行方式有三种

1) 脚本方式

直接运行包含Pig脚本的文件,比如以下命令将运行本地scripts.pig文件中的所有命令:

pig scripts.pig

2)Grunt方式

a) Grunt提供了交互式运行环境,可以在命令行编辑执行命令

b) Grund同时支持命令的历史记录,通过上下方向键访问。

c) Grund支持命令的自动补全功能。比如当你输入a = foreach b g时,按下Tab键,则命令行自动变成a = foreach b generate。你甚至可以自定义命令自动补全功能的详细方式。具体请参阅相关文档。

3) 嵌入式方式

可以在java中运行Pig程序,类似于使用JDBC运行SQL程序

(不熟悉)

6.启动集群

[hadoop@xuegod64 ~]$ start-all.sh

[hadoop@xuegod64 ~]$ jps

4722 DataNode

5062 DFSZKFailoverController

5159 ResourceManager

4905 JournalNode

5321 Jps

4618 NameNode

2428 QuorumPeerMain

5279 NodeManager

[hadoop@xuegod64 ~]$ ssh xuegod63

Last login: Sat Jan 2 23:10:21 2016 from xuegod64

[hadoop@xuegod63 ~]$ jps

2130 QuorumPeerMain

3125 Jps

2982 NodeManager

2886 JournalNode

2795 DataNode

[hadoop@xuegod64 ~]$ ssh xuegod65

Last login: Sat Jan 2 15:11:33 2016 from xuegod64

[hadoop@xuegod65 ~]$ jps

3729 Jps

2401 QuorumPeerMain

3415 JournalNode

3484 DFSZKFailoverController

3325 DataNode

3583 NodeManager

3590 SecondNameNode

7.简单示例

我们以查找最高气温为例,演示如何利用Pig统计每年的最高气温。假设数据文件内容如下(每行一个记录,tab分割)

以local模式进入pig,依次输入以下命令(注意以分号结束语句):

[hadoop@xuegod64 ~]$ pig -x local

grunt> records = load'/home/hadoop/zuigaoqiwen.txt'as(year:chararray,temperature:int);

2016-01-02 16:12:05,700 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

2016-01-02 16:12:05,701 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

grunt> dump records;

(1930:28:1,)

(1930:0:1,)

(1930:22:1,)

(1930:22:1,)

(1930:22:1,)

(1930:22:1,)

(1930:28:1,)

(1930:0:1,)

(1930:0:1,)

(1930:0:1,)

(1930:11:1,)

(1930:0:1,)

(过程略)

grunt> describe records;

records: {year: chararray,temperature: int}

grunt> valid_records = filter records by temperature!=999;

grunt> grouped_records = group valid_records by year;

grunt> dump grouped_records;

grunt> describe grouped_records;

grouped_records: {group: chararray,valid_records: {(year: chararray,temperature: int)}}

grunt> grouped_records = group valid_records by year;

grunt> dump grouped_records;

.

.

2016-01-02 16:16:02,974 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Processing split: Number of splits :1

Total Length = 7347344

Input split[0]:

Length = 7347344

ClassName: org.apache.hadoop.mapreduce.lib.input.FileSplit

Locations:

2016-01-02 16:16:08,011 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete

2016-01-02 16:16:08,012 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAFeatures

2.4.1 0.15.0 hadoop 2016-01-02 16:16:02 2016-01-02 16:16:08 GROUP_BY,FILTER

Success!

Job Stats (time in seconds):

JobId Maps Reduces MaxMapTime MinMapTime AvgMapTimMedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs

job_local798558500_0002 1 1 n/a n/a n/a n/a n/a n/a n/a n/a grouped_records,records,valid_records GROUP_BY file:/tmp/temp-206603117/tmp-1002834084,

Input(s):

Successfully read 642291 records from: "/home/hadoop/zuigaoqiwen.txt"

Output(s):

Successfully stored 0 records in: "file:/tmp/temp-206603117/tmp-1002834084"

Counters:

Total records written : 0

Total bytes written : 0

Spillable Memory Manager spill count : 0

Total bags proactively spilled: 0

Total records proactively spilled: 0

Job DAG:

job_local798558500_0002

grunt> describe grouped_records;

grouped_records: {group: chararray,valid_records: {(year: chararray,temperature: int)}}

grunt> max_temperature = foreach grouped_records generate group,MAX(valid_records.temperature);

grunt> dump max_temperature;

(1990,23)

(1991,21)

(1992,30)

grunt> quit

2016-01-02 16:24:25,303 [main] INFO org.apache.pig.Main - Pig script completed in 14 minutes, 27 seconds and 123 milliseconds (867123 ms)

中间有些问题,搞不定:

错误提示:

2016-01-02 16:18:28,049 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2016-01-02 16:18:28,050 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2016-01-02 16:18:28,050 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2016-01-02 16:18:28,055 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 642291 time(s).

2016-01-02 16:18:28,055 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

2016-01-02 16:18:28,055 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

2016-01-02 16:18:28,056 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

2016-01-02 16:18:28,056 [main] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized

2016-01-02 16:18:28,246 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1

2016-01-02 16:18:28,246 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1

错误日志:

at java.lang.reflect.Method.invoke(Method.java:497)

Pig Stack Trace

at org.apache.pig.tools.grunt.GruntParser.processPig(Grunt

Parser.java:1082)

at org.apache.pig.tools.pigscript.parser.PigScriptParser.p

arse(PigScriptParser.java:505)

at org.apache.pig.tools.grunt.GruntParser.parseStopOnError

(GruntParser.java:230)

at org.apache.pig.tools.grunt.GruntParser.parseStopOnError

(GruntParser.java:205)

at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)

at org.apache.pig.Main.run(Main.java:565)

at org.apache.pig.Main.main(Main.java:177)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Met

hod)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMetho

dAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegat

ingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

==============

posted on 2017-03-11 09:40 Lucky_7 阅读(...) 评论(...) 编辑 收藏

hadoop pig 分类循环问题

07-22

我在写pig程序时遇到了一个问题,不清楚pig里面能不能用循环,望大神指点。rn原始的数据叫test.txt是这样的: rn5.1,win,win97,1,0,dosrn5.1,win,win97,1,0,dosrn5.1,win,winxp,1,0,dosrn5.1,win,win97,2,0,dosrn5.1,win,win97,1,0,dosrn5.1,win,winxp,1,0,dosrn5.1,win,winxp,2,0,dosrn5.1,redhat,centos6,3,0,linuxrn5.1,redhat,centos7,1,0,linuxrn5.1,apple,ios7,2,0,bsdrn5.1,redhat,centos6,3,0,linuxrn5.1,apple,ios7,2,0,bsdrn5.1,apple,ios7,3,0,bsdrn5.1,win,winxp,2,0,dosrn5.1,apple,ios6,3,0,bsdrn5.1,apple,ios6,3,0,bsdrn5.1,redhat,centos7,3,0,linuxrn下面是我对这段数据写的pig的初步处理:rndata = LOAD 'test.txt' USING PigStorage(',') AS (Time:chararray, Company:chararray , Id:chararray, Warning:int, Wstatus:int, Area:chararray);rndata_id = GROUP data BY (Id,Company,Area,Warning);rndata_id_warning = FOREACH data_id GENERATE FLATTEN(group), COUNT(data) AS TOTAL;rndata_id_company = RANK data_id_warning BY group::Company ASC DENSE;rn此时dump data_id_company的结果如下:rn(1,ios7,apple,bsd,3,1)rn(1,ios7,apple,bsd,2,2)rn(1,ios6,apple,bsd,3,2)rn(2,centos7,redhat,linux,3,1)rn(2,centos7,redhat,linux,1,1)rn(2,centos6,redhat,linux,3,2)rn(3,winxp,win,dos,2,2)rn(3,winxp,win,dos,1,2)rn(3,win97,win,dos,2,1)rn(3,win97,win,dos,1,3)rn而我要的最后结果是把(1,ios7,apple,bsd,3,1) rn (1,ios7,apple,bsd,2,2)rn (1,ios6,apple,bsd,3,2)输出成一个文件,并把其命名为company的名字,即apple;rn把(2,centos7,redhat,linux,3,1)rn (2,centos7,redhat,linux,1,1)rn (2,centos6,redhat,linux,3,2)输出成一个文件,并把其命名为company的名字,即redhat;rn把(3,winxp,win,dos,2,2)rn (3,winxp,win,dos,1,2)rn (3,win97,win,dos,2,1)rn (3,win97,win,dos,1,3)输出成一个文件,并把其命名为company的名字,即win;rn[color=#FF0000]最重要的是数据量是不确定的,就是不止这一个数据要处理,company的类型也不会只是3种,而是变量。[/color]rn希望大神能用pig和shell脚本来执行这个功能。rnrnrn

没有更多推荐了,返回首页

私密
私密原因:
请选择设置私密原因
  • 广告
  • 抄袭
  • 版权
  • 政治
  • 色情
  • 无意义
  • 其他
其他原因:
120
出错啦
系统繁忙,请稍后再试