先来看下org.apache.hadoop.io.serializer的类图(hadoop2.7.1):
由类图看:
接口三个:
1、Deserializer:定义反序列化接口;
2、Serializer:定义序列化接口;
3、Serialization:定义了一系列和序列化相关并相互依赖对象的接口。
依据这三个接口,分别实现了2个类,分别是支持Writable机制的WritableSerialization和支持Java序列化的JavaSerialization,这样一共是6个实现类。
SerilizationFactory:维护一个Serilization的ArrayList。它具有参数为Configuration的构造函数,把parameter io.serializations中逗号隔开的serialization都添加进来。
Deserializer:将字节流转为一个对象。这个接口的方法有:打开流,反序列化,关闭流
源码:
package org.apache.hadoop.io.serializer;
import java.io.IOException;
import java.io.InputStream;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
/**
* <p>
* Provides a facility for deserializing objects of type <T> from an
* {@link InputStream}.
* </p>
*
* <p>
* Deserializers are stateful, but must not buffer the input since
* other producers may read from the input between calls to
* {@link #deserialize(Object)}.
* </p>
* @param <T>
*/
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Evolving
public interface Deserializer<T> {
/**
* <p>Prepare the deserializer for reading.</p>
*/
void open(InputStream in) throws IOException;
/**
* <p>
* Deserialize the next object from the underlying input stream.
* If the object <code>t</code> is non-null then this deserializer
* <i>may</i> set its internal state to the next object read from the input
* stream. Otherwise, if the object <code>t</code> is null a new
* deserialized object will be created.
* </p>
* @return the deserialized object
*/
T deserialize(T t) throws IOException;
/**
* <p>Close the underlying input stream and clear up any resources.</p>
*/
void close() throws IOException;
}
Serializer:将一个对象转换为一个字节流的实现实例,该接口的方法有:打开流,序列化,关闭流
源码:
package org.apache.hadoop.io.serializer;
import java.io.IOException;
import java.io.OutputStream;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
/**
* <p>
* Provides a facility for serializing objects of type <T> to an
* {@link OutputStream}.
* </p>
*
* <p>
* Serializers are stateful, but must not buffer the output since
* other producers may write to the output between calls to
* {@link #serialize(Object)}.
* </p>
* @param <T>
*/
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Evolving
public interface Serializer<T> {
/**
* <p>Prepare the serializer for writing.</p>
*/
void open(OutputStream out) throws IOException;
/**
* <p>Serialize <code>t</code> to the underlying output stream.</p>
*/
void serialize(T t) throws IOException;
/**
* <p>Close the underlying output stream and clear up any resources.</p>
*/
void close() throws IOException;
}
Serialization:使用抽象工厂的设计模式,封装了一对Serializer/Deserializer,判断是否支持输入的类,根据输入的类给出序列化接口和反序列化接口。
源码:
package org.apache.hadoop.io.serializer;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
/**
* <p>
* Encapsulates a {@link Serializer}/{@link Deserializer} pair.
* </p>
* @param <T>
*/
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Evolving
public interface Serialization<T> {
/**
* Allows clients to test whether this {@link Serialization}
* supports the given class.
*/
boolean accept(Class<?> c);
/**
* @return a {@link Serializer} for the given class.
*/
Serializer<T> getSerializer(Class<T> c);
/**
* @return a {@link Deserializer} for the given class.
*/
Deserializer<T> getDeserializer(Class<T> c);
}
SerializationFactory :序列化工厂,初始化时从配置项io.serializations中获取序列化工具,默认使用org.apache.hadoop.io.serializer.WritableSerialization作为序列化工具。通过调用getSerializer和getDeserializer来获取序列化与反序列化工具。
源码:
package org.apache.hadoop.io.serializer;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.CommonConfigurationKeys;
import org.apache.hadoop.io.serializer.avro.AvroReflectSerialization;
import org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization;
import org.apache.hadoop.util.ReflectionUtils;
/**
* <p>
* A factory for {@link Serialization}s.
* </p>
*/
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Evolving
public class SerializationFactory extends Configured {
private static final Log LOG =
LogFactory.getLog(SerializationFactory.class.getName());
private List<Serialization<?>> serializations = new ArrayList<Serialization<?>>();
/**
* <p>
* Serializations are found by reading the <code>io.serializations</code>
* property from <code>conf</code>, which is a comma-delimited list of
* classnames.
* </p>
*/
public SerializationFactory(Configuration conf) {
super(conf);
for (String serializerName : conf.getTrimmedStrings(
CommonConfigurationKeys.IO_SERIALIZATIONS_KEY,
new String[]{WritableSerialization.class.getName(),
AvroSpecificSerialization.class.getName(),
AvroReflectSerialization.class.getName()})) {
add(conf, serializerName);
}
}
@SuppressWarnings("unchecked")
private void add(Configuration conf, String serializationName) {
try {
Class<? extends Serialization> serializionClass =
(Class<? extends Serialization>) conf.getClassByName(serializationName);
serializations.add((Serialization)
ReflectionUtils.newInstance(serializionClass, getConf()));
} catch (ClassNotFoundException e) {
LOG.warn("Serialization class not found: ", e);
}
}
public <T> Serializer<T> getSerializer(Class<T> c) {
Serialization<T> serializer = getSerialization(c);
if (serializer != null) {
return serializer.getSerializer(c);
}
return null;
}
public <T> Deserializer<T> getDeserializer(Class<T> c) {
Serialization<T> serializer = getSerialization(c);
if (serializer != null) {
return serializer.getDeserializer(c);
}
return null;
}
@SuppressWarnings("unchecked")
public <T> Serialization<T> getSerialization(Class<T> c) {
for (Serialization serialization : serializations) {
if (serialization.accept(c)) {
return (Serialization<T>) serialization;
}
}
return null;
}
}
下面对SerializationFactory生产Serializations做个简单的解析说明:
首先来看其构造函数里的一个全局参数:CommonConfigurationKeys.IO_SERIALIZATIONS_KEY,它的值定义如下:
/** See <a href="{@docRoot}/../core-default.html">core-default.xml</a> */
public static final String IO_SERIALIZATIONS_KEY = "io.serializations";
而使用SerializationFactory的构造函数:public SerializationFactory(Configuration conf) 时,使用配置文件:Configuration: core-default.xml, core-site.xml。如:
SerializationFactory factory = new SerializationFactory(conf);
而在hadoop2.7.1中默认配置文件core-default.xml的io.serializations的属性如下:
由此,通过SerializationFactory生产的Serializations有三个:
如果serialization的值为:org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,则将调用:
而在hadoop2.7.1中默认配置文件core-default.xml的io.serializations的属性如下:
<property>
<name>io.serializations</name>
<value>org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization<alue>
<description>A list of serialization classes that can be used for
obtaining serializers and deserializers.</description>
</property>
由此,通过SerializationFactory生产的Serializations有三个:
org.apache.hadoop.io.serializer.WritableSerialization,
org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,
org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
通过其方法public <T> Serializer<T> getSerializer(Class<T> c), public <T> Serialization<T> getSerialization(Class<T> c)便能得到相应的Serialization:
注1: if (serialization.accept(c))将会调用相应类的accept函数,例如:如果serialization的值为:org.apache.hadoop.io.serializer.WritableSerialization,则将调用:
org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,
org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
通过其方法public <T> Serializer<T> getSerializer(Class<T> c), public <T> Serialization<T> getSerialization(Class<T> c)便能得到相应的Serialization:
public <T> Serializer<T> getSerializer(Class<T> c) {
Serialization<T> serializer = getSerialization(c);
if (serializer != null) {
return serializer.getSerializer(c);
}
return null;
}
<div> @SuppressWarnings("unchecked")
public <T> Serialization<T> getSerialization(Class<T> c) {
for (Serialization serialization : serializations) {
if (serialization.accept(c))<strong> </strong>{ //注1
return (Serialization<T>) serialization;
}
}
return null;
}</div>
注1: if (serialization.accept(c))将会调用相应类的accept函数,例如:如果serialization的值为:org.apache.hadoop.io.serializer.WritableSerialization,则将调用:
@InterfaceAudience.Private
@Override
public boolean accept(Class<?> c) {
return Writable.class.isAssignableFrom(c);
}
如果serialization的值为:org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,则将调用:
@InterfaceAudience.Private
@Override
public boolean accept(Class<?> c) {
return SpecificRecord.class.isAssignableFrom(c); //注2
}
注2:
public boolean isAssignableFrom(Class<?> cls)
-
判定此
Class
对象所表示的类或接口与指定的Class
参数所表示的类或接口是否相同,或是否是其超类或超接口。如果是则返回true
;否则返回false
。如果该Class
表示一个基本类型,且指定的Class
参数正是该Class
对象,则该方法返回true
;否则返回false
。特别地,通过身份转换或扩展引用转换,此方法能测试指定
Class
参数所表示的类型能否转换为此Class
对象所表示的类型。有关详细信息,请参阅 Java Language Specification 的第 5.1.1 和 5.1.4 节。 -
-
参数:
-
cls
- 要检查的Class
对象
返回:
-
表明
cls
类型的对象能否赋予此类对象的boolean
值
抛出:
- NullPointerException - 如果指定的 Class 参数为 null。
-