IBM云对象存储 - Linux主机通过rclone和COS API上传大文件

云对象存储作为主流公有云数据存储服务已大规模应用，但其基于HTTP/HTTPs协议（RESTful API）、扁平数据结构和网络依赖等特性，在某些文件归档和备份场景中,通过类似s3fs转成文件系统挂载使用时，或多或少有些限制，比如大文件传送或高频繁I/O ，这里以IBM Cloud Object Storage为例，我们尝试通过rclone和ICOS API方式来实现较为稳定的文件传送和带宽控制。

1. Rclone

IBM Cloud官方配置连接文档:
https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-rclone

[root@centos-s3fs ~]# yum install -y unzip
[root@centos-s3fs ~]# curl https://rclone.org/install.sh | sudo bash
...
rclone v1.53.3 has successfully installed.
Now run "rclone config" for setup. Check https://rclone.org/docs/ for more details.

[root@centos-s3fs ~]# rclone config
2020/12/19 09:55:26 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> icos-test
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
   \ "s3"
... ...
Storage> 4
** See help for s3 backend at: https://rclone.org/s3/ **

Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Amazon Web Services (AWS) S3
   \ "AWS"
 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
   \ "Alibaba"
 3 / Ceph Object Storage
   \ "Ceph"
 4 / Digital Ocean Spaces
   \ "DigitalOcean"
 5 / Dreamhost DreamObjects
   \ "Dreamhost"
 6 / IBM COS S3
   \ "IBMCOS"
... ...
provider> 6
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth