MapReduce数据类型
数据类型都要实现Writable接口,以便用这些类型定义的数据可以被序列化进行网络传输和文件存储。自定义key数据类型的时候,因为需要对key进行排序,需要继承java中的比较器,所以可以直接继承WritableComparable(WritableComparable继承了Writable和Comparable)。
基本数据类型:
- BooleanWritable
- ByteWritable
- DoubleWritable
- FloatWritable
- IntWritable
- LongWritable
- Text
- NullWritable:当
自定义MapReduce数据类型
1、实现Writable接口
重写 write()和readFields()方法
2、实现WritableComparable接口
重写 write(),readFields()和compareTo()方法,compareTo()方法用户shuffle过程中根据key排序时使用。
注意:以上两种方式在使用的使用都需要进行重写重写toString(), hashCode(), equals()
具体的自定义数据类型的代码如下:
public class UserWritable implements WritableComparable <UserWritable>{
private int id;
private String name;
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public UserWritable() {
}
public UserWritable(int id, String name) {
set(id, name);
}
public void set(int id, String name) {
this.setId(id);
this.setName(name);
}
//注意!!!:此两个方法读写字段的顺序必须保持一致,不一致将会出错
public void readFields(DataInput in) throws IOException {
this.id = in.readInt();
this.name = in.readUTF();
}
public void write(DataOutput out) throws IOException {
out.writeInt(id);
out.writeUTF(name);
}
public int compareTo(UserWritable o) {
int comp = Integer.valueOf(this.getId()).compareTo(o.getId());
if(0 != comp)
return comp;
return this.getName().compareTo(o.getName());
}
@Override
public String toString() {
return id +"\t"+ name;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
UserWritable that = (UserWritable) o;
if (id != that.id) return false;
return name != null ? name.equals(that.name) : that.name == null;
}
@Override
public int hashCode() {
int result = id;
result = 31 * result + (name != null ? name.hashCode() : 0);
return result;
}
}