spark数据传输报错
报错信息如下代码段:
关键句:"Message": "Too many versions.
问题解释:太多版本了,即:太多批次了。写太快了,控制写入速率,添加参数:target.batchSize=50000,意思是5万条数据为一批次传输。
java.lang.RuntimeException: stream load failed, loadResult: {
"TxnId": 144161435,
"Label": "fa40f79df0ff48899429f3671ff04b9d",
"Status": "Fail",
"Message": "Too many versions. tablet_id: 148704782, version_count: 1001, limit: 1000: be:10.21.121.183",
"NumberTotalRows": 0,
"NumberLoadedRows": 0,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 765533,
"LoadTimeMs": 22,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 1,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 21,
"CommitAndPublishTimeMs": 0
}