hive streming sink 投入生产,发现写入性能底下,排查过后,发现是hive在每写入一条数据,都会判断文件大小来决定文件是否需要滚动,判断文件大小使用的hdfs的api,需要访问namenode,这就是写入性能底下的根源。截止到flink 1.11.2这个问题任然没有解决。
这个问题解决,需要自己实现HiveBulkWriterFactory,代码如下:
public class HiveBulkWriterFactory implements HadoopPathBasedBulkWriter.Factory<RowData> {
private static final long serialVersionUID = 1L;
private final HiveWriterFactory factory;
public HiveBulkWriterFactory(HiveWriterFactory factory) {
this.factory = factory;
}
@Override
public HadoopPathBasedBulkWriter<RowData> create(Path targetPath, Path inProgressPath) throws IOException {
FileSinkOperator.RecordWriter recordWriter = factory.createRecordWriter(inProgressPath);
Function<RowData, Writable> rowConverter = factory.createRowDataConverter();
FileSystem fs = FileSystem.get(inProgressPath.toUri(), factory.getJobConf());
return new HadoopPathBasedBu