openstack swift对象存储大文件分段操作实践

openstack swift对象存储大文件分段操作实践
上传大文件耗时，如果期间网络断了，上传没有完成，又需要重传比较麻烦。因此大文件要分段上传，可以实现网络中断时断点上传，openstack swift对象存储实现了这个功能
#静态大对象SLO中间件实现的
创建大对象，先上传各分段，最后上传manifest分段目录
curl -i -X PUT 127.0.0.1:8080/v1/swift/NDK000000/slargeobj2-part1 --data-binary ‘111’
curl -i -X PUT 127.0.0.1:8080/v1/swift/NDK000000/slargeobj2-part2 --data-binary ‘222’
curl -i -X PUT 127.0.0.1:8080/v1/swift/NDK000000/slargeobj2-part3 --data-binary ‘333’

curl -i -X PUT 127.0.0.1:8080/v1/swift/NDK000000/slargeobj2?multipart-manifest=put --data-binary ‘[{“path”: “/NDK000000/slargeobj2-part1”}, {“path”: “/NDK000000/slargeobj2-part2”}, {“path”: “/NDK000000/slargeobj2-part3”}]’

PUT /v1/swift/NDK000000/slargeobj2?multipart-manifest=put HTTP/1.1
User-Agent: curl/7.29.0
Host: 127.0.0.1:8080
Accept: /
Content-Length: 123
Content-Type: application/x-www-form-urlencoded

• upload completely sent off: 123 out of 123 bytes
  < HTTP/1.1 201 Created
  HTTP/1.1 201 Created
  < Last-Modified: Fri, 29 Dec 2023 07:43:30 GMT
  Last-Modified: Fri, 29 Dec 2023 07:43:30 GMT
  < Content-Length: 0
  Content-Length: 0
  < Etag: “43187d8bc2eb7acc04f9f1a699333bba”
  Etag: “43187d8bc2eb7acc04f9f1a699333bba”
  < Content-Type: text/html; charset=UTF-8
  Content-Type: text/html; charset=UTF-8
  < X-Trans-Id: tx158ef5a7a7a54631a1a27-00658e78a1
  X-Trans-Id: tx158ef5a7a7a54631a1a27-00658e78a1
  < X-Openstack-Request-Id: tx158ef5a7a7a54631a1a27-00658e78a1
  X-Openstack-Request-Id: tx158ef5a7a7a54631a1a27-00658e78a1
  < Date: Fri, 29 Dec 2023 07:43:29 GMT
  Date: Fri, 29 Dec 2023 07:43:29 GMT
  下载大对象
  curl -i 127.0.0.1:8080/v1/swift/NDK000000/slargeobj2
  HTTP/1.1 200 OK
  Content-Length: 9
  Accept-Ranges: bytes
  Last-Modified: Fri, 29 Dec 2023 07:43:30 GMT
  Etag: “43187d8bc2eb7acc04f9f1a699333bba”
  X-Timestamp: 1703835809.58266
  X-Static-Large-Object: True
  Content-Type: application/x-www-form-urlencoded
  X-Trans-Id: tx38d71e387267451d9d336-00658e7a7e
  X-Openstack-Request-Id: tx38d71e387267451d9d336-00658e7a7e
  Date: Fri, 29 Dec 2023 07:51:26 GMT

111222333
删除大对象
curl -i -X DELETE 127.0.0.1:8080/v1/swift/NDK000000/slargeobj2?multipart-manifest=delete

查询有哪些分段构成的
curl -i “127.0.0.1:8080/v1/swift/NDK000000/slargeobj2?multipart-manifest=get&format=raw”

HTTP/1.1 200 OK
Content-Length: 306
Accept-Ranges: bytes
Last-Modified: Fri, 29 Dec 2023 07:43:30 GMT
Etag: 1019b9486234c4a40e8c8f9163403602
X-Timestamp: 1703835809.58266
X-Static-Large-Object: True
Content-Type: application/x-www-form-urlencoded
X-Trans-Id: tx674edf3788364e4092661-00658e7a33
X-Openstack-Request-Id: tx674edf3788364e4092661-00658e7a33
Date: Fri, 29 Dec 2023 07:50:11 GMT

[{“etag”: “698d51a19d8a121ce581499d7b701668”, “size_bytes”: 3, “path”: “/NDK000000/slargeobj2-part1”}, {“etag”: “bcbe3365e6ac95ea2c0343a2395834dd”, “size_bytes”: 3, “path”: “/NDK000000/slargeobj2-part2”}, {“etag”: “310dcbbf4cce62f762a2aaa148d556bd”, “size_bytes”: 3, “path”: “/NDK000000/slargeobj2-part3”}]

作者 伍增田 wu Tommy