使用datax往starrocks里灌数据时一直报错
Caused by: java.lang.NullPointerException
at com.starrocks.connector.datax.plugin.writer.starrockswriter.manager.StarRocksStreamLoadVisitor.joinRows(StarRocksStreamLoadVisitor.java:111)
at com.starrocks.connector.datax.plugin.writer.starrockswriter.manager.StarRocksStreamLoadVisitor.doStreamLoad(StarRocksStreamLoadVisitor.java:65)
at com.starrocks.connector.datax.plugin.writer.starrockswriter.manager.StarRocksWriterManager.asyncFlush(StarRocksWriterManager.java:161)
定位Datax源码com.starrocks.connector.datax.plugin.writer.starrockswriter.manager.StarRocksStreamLoadVisitor
需要配置loadProps参数。
这里的坑是,我查看官方文档时,他说的是非必填
现在源码里看到这里需要该参数,我又看文档,下面居然还有这个注意事项
注意事项
导入参数配置
默认传入的数据均会被转为字符串,并以\t作为列分隔符,\n作为行分隔符,组成csv文件进行StreamLoad导入操作。 如需更改列分隔符,则正确配置 loadProps 即可:
"loadProps": {
"column_separator": "\\x01",
"row_delimiter": "\\x02"
}
如需更改导入格式为json,则正确配置 loadProps 即可:
"loadProps": {
"format": "json",
"strip_outer_array": true
}
因此这里的解决方法就是增加loadProps的参数;完整writer参数为:
"writer": {
"name": "starrockswriter",
"parameter": {
"username": "root",
"password": "****",
"database": "database",
"table": "table_name",
"preSql":[
],
"loadProps": {"column_separator": "\\x01","row_delimiter": "\\x02"},
"column": ["x","x","x"],
"jdbcUrl": "jdbc:mysql://10.x.x.x:9030",
"loadUrl": ["ip:8030","ip:8030","ip:8030"]
}
}