背景:
众所知周,AWS S3 client功能强大,能与各类符合OSS标准的存储进行工作。本项目中刚好用到ceph,就是直接使用S3 java client上传文件。在根据ceph官网的的一些文档操作时遇到一些问题,这里记录下来,仅供遇到类似问题的网友参考。
问题与解决方法
按照ceph https://docs.ceph.com/en/quincy/radosgw/s3/java/ 官网的示例,在本人环境中无法遍历bucket或者upload文件, 在多方搜索对比测试后发现,需要添加几个参数才行,目前不确定是否是因为本地ceph的设置导致的问题, 这里记录下来仅供参考。
// disable SSL cert check
String accessKey = "insert your access key here!";
String secretKey = "insert your secret key here!";
//具体的域名, 每个项目或者公司都不一样
String cephHost = "http://cepha.b.com";
System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY,"true");
if (SDKGlobalConfiguration.isCertCheckingDisabled())
{
System.out.println("Cert checking is disabled");
}
// S3 Client configuration
ClientConfiguration config = new ClientConfiguration();
// Not the standard "AWS3SignerType", maar expliciet signerTypeV2
config.setSignerOverride("S3SignerType");
// or else Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch will happen. 这个也很重要,不然给putRequest 添加一些mata时会报SignatureDoesNotMatch的错误
config.setSignerOverride("AWSS3V4SignerType");
config.setProtocol(Protocol.HTTP);
BasicAWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.
EndpointConfiguration(cephHost, "local-ceph");
// 重点就是这个 "local-ceph", 不设置它,在本人项目中,可以list buckt但是不能upload file
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withClientConfiguration(config)
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(endpointConfiguration)
.enablePathStyleAccess()
.disableChunkedEncoding()
.build();
对比一下ceph官网的文档
String accessKey = "insert your access key here!";
String secretKey = "insert your secret key here!";
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 s3Client = new AmazonS3Client(credentials);
s3Client.setEndpoint("objects.dreamhost.com");
其他
在创建了AmazonS3 对象后记可以操作bucket中内容
List<Bucket> buckets = s3Client.listBuckets();
上传文件,并带有meta信息,
try {
ObjectMetadata metaData = new ObjectMetadata();
Map<String, String> userMetaData = new HashMap<>();
userMetaData.put("key1", "val1");
userMetaData.put("key2", "val2");
userMetaData.put("key3", "val3");
metaData.setUserMetadata(userMetaData);
PutObjectRequest putObjReq = new PutObjectRequest(bucketName, fileKey, file);
putObjReq.setMetadata(metaData);
PutObjectResult result = s3Client.putObject(putObjReq);
log.info("upload result {}", result);
} catch (Exception ex) {
log.error("fail to upload to fileKey {}, exMsg:{}",
fileKey, ex.getMessage(), ex);
}
如果上传的文件较大,使用TransferManager, 参看S3官方文档
https://docs.aws.amazon.com/zh_cn/sdk-for-java/v1/developer-guide/examples-s3-transfermanager.html
PutObjectRequest putObjReq = new PutObjectRequest(bucketName, fileKey, file);
putObjReq.setMetadata(metaData);
TransferManager tm = TransferManagerBuilder.standard()
.withS3Client(s3Client)
.build();
Upload upload = tm.upload(putObjReq, progressListener);
log.info("start to upload file '{}' to fileKey:{},",
file.getAbsolutePath(), fileKey);
// wait for the upload to finish before continuing.
upload.waitForCompletion();