GDAL读取AWS上S3数据异常

最新推荐文章于 2023-12-29 18:10:04 发布

松山仪龙

最新推荐文章于 2023-12-29 18:10:04 发布

阅读量660

点赞数 1

分类专栏：开源技术-GDAL 文章标签： c++

本文链接：https://blog.csdn.net/longba1112/article/details/124453638

版权

开源技术-GDAL 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

今天发现S3下影像数据集打开后，读取数据信息，第一次读取很快，第二次读取特别慢，还报错。代码如下所示：

#include <iostream>
#include <string>
#include <time.h>

#include "gdal.h"
#include "gdal_priv.h"

int main()
{
	// 设置编码
	CPLSetConfigOption("GDAL_FILENAME_IS_UTF8", "NO");
	// 注册驱动
	GDALAllRegister();

	// 设置参数信息
	const char* regionLabel = "AWS_REGION";
	const char* endpointLabel = "AWS_S3_ENDPOINT";
	const char* access_keyLabel = "AWS_SECRET_ACCESS_KEY";
	const char* access_key_idLabel = "AWS_ACCESS_KEY_ID";
	const char* ssl_cert_fileLabel = "SSL_CERT_FILE";

	const char* region = "cn-northwest-1";
	const char* endpoint = "s3.cn-northwest-1.amazonaws.com.cn";
	const char* access_key = "your access_key";
	const char* access_key_id = "your access_key_id ";
	const char* ssl_cert_file = "cacert.pem";

	CPLSetConfigOption(regionLabel, region);
	CPLSetConfigOption(access_keyLabel, access_key);
	CPLSetConfigOption(access_key_idLabel, access_key_id);
	CPLSetConfigOption(endpointLabel, endpoint);
	CPLSetConfigOption(ssl_cert_fileLabel, ssl_cert_file);
	CPLSetConfigOption("CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE", "YES");

	// 打开S3上的文件
	std::string pFilePath1 = "your s3 file path";
	GDALDataset* pDatasetSource = (GDALDataset*)GDALOpen(pFilePath1.c_str(), GA_ReadOnly);

	if (pDatasetSource == nullptr)
	{
		std::cout << "fail in open files!!!" << std::endl << CPLGetLastErrorMsg() << std::endl;
		system("pause");
		return 1;
	}
	int width = pDatasetSource->GetRasterXSize();
	int height = pDatasetSource->GetRasterYSize();

	int64_t now1 = time(nullptr);

	unsigned char* pafScanblock1 = (unsigned char*)CPLMalloc(sizeof(unsigned char) * 256 * 256); //建议能小则小否则会造成内存不足的情况
	pDatasetSource->GetRasterBand(1)->RasterIO(GF_Read, 0, 0, 512, 512, pafScanblock1, 256, 256, GDT_Byte, 0, 0);
	delete pafScanblock1;
	pafScanblock1 = nullptr;

	int64_t now2 = time(nullptr);
	std::cout << "Sub1-Read Time:" << now2 - now1 << std::endl;

	unsigned char* pafScanblock2 = (unsigned char*)CPLMalloc(sizeof(unsigned char) * 256 * 256); 
	pDatasetSource->GetRasterBand(1)->RasterIO(GF_Read, 0, 0, 512, 512, pafScanblock2, 256, 256, GDT_Byte, 0, 0);
	delete pafScanblock2;
	pafScanblock2 = nullptr;

	int64_t now3 = time(nullptr);
	std::cout << "Sub2-Read Time:" << now3 - now2 << std::endl;

	// 关闭数据集
	GDALClose(pDatasetSource);
	system("pause");

	return EXIT_SUCCESS;

}

控制台输出的日志信息如下所示：

Sub1-Read Time:6
ERROR 1: Request for 24572-30712 failed
ERROR 1: Request for 36854-42994 failed
ERROR 1: Request for 49136-55276 failed
ERROR 1: Request for 61418-67558 failed
ERROR 1: Request for 73700-79840 failed
ERROR 1: Request for 85982-92122 failed
<-此处省略很多->
ERROR 1: Request for 3095072-3101212 failed
ERROR 1: Request for 3107354-3113494 failed
ERROR 1: Request for 3119636-3125776 failed
ERROR 1: Request for 3131918-3138058 failed
Sub2-Read Time:65
请按任意键继续. . .

经过无数次测试，发现，是读取的时候的参数设置引起的这个问题。在RasterIO中，如果读取的数据的范围和用于存放数据的“指针”的大小相等就没有任何问题，如果读取数据的范围小于存放数据的“指针”的大小也没有问题，但是大于就会出现问题，不知道什么原因，求大神帮忙解答……

// 有问题
pDatasetSource->GetRasterBand(1)->RasterIO(GF_Read, 0, 0, 512, 512, pafScanblock1, 256, 256, GDT_Byte, 0, 0);

// 没有问题
pDatasetSource->GetRasterBand(1)->RasterIO(GF_Read, 0, 0, 256, 256, pafScanblock1, 256, 256, GDT_Byte, 0, 0);

// 没有问题
pDatasetSource->GetRasterBand(1)->RasterIO(GF_Read, 0, 0, 128, 128, pafScanblock1, 256, 256, GDT_Byte, 0, 0);