rgw分片上传/下载分析

rgw分片上传/下载分析

上传

1.客户端发送httppost请求，请求uploadid.
2.客户端收到uploadid后，按照上传不同的分片，url需要带上uploadid和分片数（partnumber）.
3.全部分片上传完成后发送http-post请求，标记上传完成.

1.请求上传

[root@sns-nvceph01 ~]# s3cmd put 111.txt s3://mpbucket01/123.dat -d
.
.
.
DEBUG: String 'root' encoded to 'root'
DEBUG: String 'root' encoded to 'root'
DEBUG: attr_header: {'x-amz-meta-s3cmd-attrs': u'atime:1553238345/ctime:1553241376/gid:0/gname:root/md5:43643d513924af57c46c65f60af8b817/mode:33188/mtime:1553241376/uid:0/uname:root'}
.
.
.
DEBUG: format_uri(): /mpbucket01/123.dat?uploads
DEBUG: Sending request method_string='POST', uri=u'/mpbucket01/123.dat?uploads', headers={'x-amz-meta-s3cmd-attrs': u'atime:1553238345/ctime:1553241376/gid:0/gname:root/md5:43643d513924af57c46c65f60af8b817/mode:33188/mtime:1553241376/uid:0/uname:root', 'content-type': 'text/plain', 'Authorization': u'AWS P13CYRER52386RFC7JYI:OAo9lLJIcbwmAOa2nFtHFLtUqTM=', 'x-amz-date': 'Fri, 26 Apr 2019 09:39:41 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://10.4.44.88:7480#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?><InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>mpbucket01</Bucket><Key>123.dat</Key><UploadId>2~TMUmSED2HdmQQMZhtkd_YwxiXGjedxI</UploadId></InitiateMultipartUploadResult>',
 'headers': {'content-length': '248',
             'content-type': 'application/xml',
             'date': 'Fri, 26 Apr 2019 09:39:41 GMT',
             'x-amz-request-id': 'tx00000000000000544ca39-005cc2d1dd-11c8-default'},
 'reason': 'OK',
 'status': 200}
 .
 .
 .

上图是贴了一段s3cmd 分片上传的debuginfo，客户端首先发起post请求，请求uploadid（2~TMUmSED2HdmQQMZhtkd_YwxiXGjedxI）.
s3cmd客户端上传的时候添加了自定义md5字段，用于下载完整性的校验，因为下载大文件时etag已经不是文件的md5值了.

2.分片上传文件

客户端可以并发上传，利用http-put请求，并且可以上传不同的rgw，需要客户端自己去协调各个分片的序号，并最后发起complete操作.
每个分片的md5值需要记录下载，发起完成请求时需要每个分片的md5.

.
.
.
DEBUG: DeUnicodising u'111.txt' using UTF-8
DEBUG: Using signature v2
DEBUG: SignHeaders: u'PUT\n\n\n\nx-amz-date:Fri, 26 Apr 2019 09:39:43 +0000\n/mpbucket01/123.dat?partNumber=11&uploadId=2~TMUmSED2HdmQQMZhtkd_YwxiXGjedxI'
DEBUG: get_hostname(mpbucket01): 10.4.44.88:7480
DEBUG: ConnMan.get(): re-using connection: http://10.4.44.88:7480#11
DEBUG: format_uri(): /mpbucket01/123.dat?partNumber=11&uploadId=2~TMUmSED2HdmQQMZhtkd_YwxiXGjedxI
    65536 of 15728640     0% in    0s  1240.19 kB/sDEBUG: ConnMan.put(): connection put back to pool (http://10.4.44.88:7480#12)
DEBUG: Response:
{'data': '',
 'headers': {'accept-ranges': 'bytes',
             'content-length': '0',
             'date': 'Fri, 26 Apr 2019 09:39:43 GMT',
             'etag': '"e88436e684d037a8704f5bafd14a0a0f"',
             'x-amz-request-id': 'tx00000000000000544ca59-005cc2d1df-11c8-default'},
 'reason': 'OK',
 'size': 15728640,
 'status': 200}
 15728640 of 15728640   100% in    0s    65.87 MB/s  done
DEBUG: MD5 sums: computed=e88436e684d037a8704f5bafd14a0a0f, received=e88436e684d037a8704f5bafd14a0a0f
.
.
.

此例为第11号分片的上传.

3.完成分片上传

待所有分片上传完成后，客户端发起http-post请求，完成分片上传.

.
.
.
DEBUG: MultiPart: Upload finished: 14 parts
DEBUG: MultiPart: Completing upload: 2~TMUmSED2HdmQQMZhtkd_YwxiXGjedxI
DEBUG: CreateRequest: resource[uri]=/123.dat
DEBUG: Using signature v2
DEBUG: SignHeaders: u'POST\n\n\n\nx-amz-date:Fri, 26 Apr 2019 09:39:44 +0000\n/mpbucket01/123.dat?uploadId=2~TMUmSED2HdmQQMZhtkd_YwxiXGjedxI'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(mpbucket01): 10.4.44.88:7480
DEBUG: ConnMan.get(): re-using connection: http://10.4.44.88:7480#15
DEBUG: format_uri(): /mpbucket01/123.dat?uploadId=2~TMUmSED2HdmQQMZhtkd_YwxiXGjedxI
DEBUG: Sending request method_string='POST', uri=u'/mpbucket01/123.dat?uploadId=2~TMUmSED2HdmQQMZhtkd_YwxiXGjedxI', headers={'content-length': '1232', 'Authorization': u'AWS P13CYRER52386RFC7JYI:n1az/AZ5AjoxbQPUm2GrxMDq4sY=', 'x-amz-date': 'Fri, 26 Apr 2019 09:39:44 +0000'}, body=(1232 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://10.4.44.88:7480#16)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?><CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://10.4.44.88:7480/mpbucket01/123.dat</Location><Bucket>mpbucket01</Bucket><Key>123.dat</Key><ETag>b29c2c878e4e07876f1fb3b8c6ff8556-14</ETag></CompleteMultipartUploadResult>',
 'headers': {'content-length': '304',
             'content-type': 'application/xml',
             'date': 'Fri, 26 Apr 2019 09:39:44 GMT',
             'x-amz-request-id': 'tx00000000000000544ca5f-005cc2d1e0-11c8-default'},
 'reason': 'OK',
 'status': 200}
 .
 .
 .

request的body为每一个分片的md5值，服务端会比较之前上传的分片是否完整.

下载

同样利用s3cmd下载刚才上传的文件

[root@sns-nvceph01 ~]# s3cmd get  s3://mpbucket01/123.dat 123.dat -d
DEBUG: s3cmd version 2.0.2
DEBUG: ConfigParser: Reading file '/root/.s3cfg'
DEBUG: ConfigParser: access_key->P1...17_chars...I
DEBUG: ConfigParser: cloudfront_host->http://10.4.44.88:7480
DEBUG: ConfigParser: host_base->http://10.4.44.88:7480
.
.
.
DEBUG: SignHeaders: u'GET\n\n\n\nx-amz-date:Fri, 26 Apr 2019 10:59:43 +0000\n/mpbucket01/123.dat'
download: 's3://mpbucket01/123.dat' -> '123.dat'  [1 of 1]
DEBUG: get_hostname(mpbucket01): 10.4.44.88:7480
DEBUG: ConnMan.get(): creating new connection: http://10.4.44.88:7480
DEBUG: non-proxied HTTPConnection(10.4.44.88, 7480)
DEBUG: format_uri(): /mpbucket01/123.dat
DEBUG: Response:
{'headers': {'accept-ranges': 'bytes',
             'content-length': '220194451',
             'content-type': 'text/plain',
             'date': 'Fri, 26 Apr 2019 10:59:43 GMT',
             'etag': '"b29c2c878e4e07876f1fb3b8c6ff8556-14"',
             'last-modified': 'Fri, 26 Apr 2019 09:39:44 GMT',
             'x-amz-meta-s3cmd-attrs': 'atime:1553238345/ctime:1553241376/gid:0/gname:root/md5:43643d513924af57c46c65f60af8b817/mode:33188/mtime:1553241376/uid:0/uname:root',
             'x-amz-request-id': 'tx000000000000005453a33-005cc2e49f-11c8-default'},
 'reason': 'OK',
 's3cmd-attrs': {'atime': '1553238345',
                 'ctime': '1553241376',
                 'gid': '0',
                 'gname': 'root',
                 'md5': '43643d513924af57c46c65f60af8b817',
                 'mode': '33188',
                 'mtime': '1553241376',
                 'uid': '0',
                 'uname': 'root'},
 'status': 200}
     65536 of 220194451     0% in    0s  1570.98 kB/sDEBUG: ConnMan.put(): connection put back to pool (http://10.4.44.88:7480#1)
 220194451 of 220194451   100% in    0s   250.68 MB/s  done
DEBUG: ReceiveFile: Computed MD5 = 43643d513924af57c46c65f60af8b817
DEBUG: DeUnicodising u'123.dat' using UTF-8
DEBUG: set mtime to 1556242784.0

可以看到下载的respone中的etag并不是文件的md5值，而是一个通过各个分片计算出来的md5值，后面的-14表示有14个md5，etag的计算看这里.
可以通过比较上传时用户自定义字段设置的md5来比较是否完整

上传失败

如果上传失败或者中断了，会有一些临时对象留在pool中，需要手动进行删除参考这里