spark-parquet列存储之：文件存储细节之：RowWriteSupport和RecordWriter

最新推荐文章于 2021-12-18 22:11:15 发布

天然呆的技术博客

最新推荐文章于 2021-12-18 22:11:15 发布

阅读量1.9k

点赞数 1

分类专栏： Spark

本文链接：https://blog.csdn.net/u013494310/article/details/44832451

版权

Spark 专栏收录该内容

18 篇文章 0 订阅

订阅专栏

RowWriteSupport

RowWriteSupport继承自WriteSupport(该文件位于hadoop.parquet jar)，为数据写入提供WriteContext支持

field：

writer:RecordConsumer

数据的消费者，负责写入数据，其有三个子类：

ValidatingRecordConsumer，负责校验工作，然后传递给下一个消费者执行同样的操作

RecordConsumerLoggingWrapper，负责记录日志，然后传递给下一个消费者执行同样的操作

MessageColumnIORecordConsumer，负责写入数据

attributes: Seq[Attribute]

内部含有Schema信息，FieldName,DataType,isNullable

function

初始化过程：

init(configuration: Configuration): WriteSupport.WriteContext

val origAttributesStr: String = configuration.get(RowWriteSupport.SPARK_ROW_SCHEMA)
    val metadata = new JHashMap[String, String]()
    metadata.put(RowReadSupport.SPARK_METADATA_KEY, origAttributesStr)

    if (attributes == null) {
      attributes = ParquetTypesConverter.convertFromString(origAttributesStr)
    }

    log.debug(s"write support initialized for requested schema $attributes")
    ParquetRelation.enableLogForwarding()
    new WriteSupport.WriteContext(ParquetTypesConverter.convertFromAttributes(attributes), metadata)

获取数据的Schema信息，并传递给WriteContext

获取ParquetTypeInfo的详细过程如下：

  def fromPrimitiveDataType(ctype: DataType): Option[ParquetTypeInfo] = ctype match {
    case StringType => Some(ParquetTypeInfo(
      ParquetPrimitiveTypeName.BINARY, Some(ParquetOriginalType.UTF8)))
    case BinaryType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.BINARY))
    case BooleanType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.BOOLEAN))
    case DoubleType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.DOUBLE))
    case FloatType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.FLOAT))
    case IntegerType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.INT32))
    // There is no type for Byte or Short so we promote them to INT32.
    case ShortType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.INT32))
    case ByteType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.INT32))
    case LongType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.INT64))
    case DecimalType.Fixed(precision, scale) if precision <= 18 =>
      // TODO: for now, our writer only supports decimals that fit in a Long
      Some(ParquetTypeInfo(ParquetPrimitiveTypeName.FIXED_LEN_BYTE_ARRAY,
        Some(ParquetOriginalType.DECIMAL),
        Some(new DecimalMetadata(precision, scale)),
        Some(BYTES_FOR_PRECISION(precision))))
    case _ => None
  }

然后将ParquetTypeInfo包装成MessageType

  def convertFromAttributes(attributes: Seq[Attribute]): MessageType = {
    val fields = attributes.map(
      attribute =>
        fromDataType(attribute.dataType, attribute.name, attribute.nullable))
    new MessageType("root", fields)
  }