aws s3 cli_使用AWS CLI将大文件上传到Amazon S3

aws s3 cli

Amazon S3 is a widely used public cloud storage system. S3 allows an object/file to be up to 5TB which is enough for most applications. The AWS Management Console provides a Web-based interface for users to upload and manage files in S3 buckets. However, uploading a large files that is 100s of GB is not easy using the Web interface. From my experience, it fails frequently. There are various third party commercial tools that claims to help people upload large files to Amazon S3 and Amazon also provides a Multipart Upload API which is most of these tools based on.

Amazon S3是一种广泛使用的公共云存储系统。 S3允许对象/文件的最大容量为5TB ，足以满足大多数应用程序的需求。 AWS管理控制台提供了基于Web的界面，供用户上载和管理S3存储桶中的文件。 但是，使用Web界面上传大小为100s GB的大文件并不容易。 根据我的经验，它经常失败。 有各种各样的第三方商业工具声称可以帮助人们将大文件上传到Amazon S3，并且Amazon还提供了Multipart Upload API（基于这些工具中的大多数）。

While these tools are helpful, they are not free and AWS already provides users a pretty good tool for uploading large files to S3—the open source aws s3 CLI tool from Amazon. From my test, the aws s3 command line tool can achieve more than 7MB/s uploading speed in a shared 100Mbps network, which should be good enough for many situations and network environments. In this post, I will give a tutorial on uploading large files to Amazon S3 with the aws command line tool.

尽管这些工具很有用，但它们并不是免费的，而且AWS已经为用户提供了一个很好的工具，可以将大文件上传到S3 – Amazon的开源 aws s3 CLI工具 。 根据我的测试， aws s3命令行工具在共享的100Mbps 网络中可以实现超过7MB / s的上传速度，这对于许多情况和网络环境来说应该足够了。 在本文中，我将提供有关使用aws命令行工具将大文件上传到Amazon S3的教程 。

安装AWS CLI工具∞ (Install aws CLI tool ∞)

Assume that you already have Python environment set up on your computer. You can install aws tools using pip or using the bundled installer

假设您已经在计算机上设置了Python环境。 您可以安装aws工具 使用pip或 使用捆绑的安装程序

$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

Try to run aws after installation. If you see output as follows, you should have installed it successfully.

安装后尝试运行aws 。 如果看到以下输出，则应已成功安装。

$ aws
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: too few arguments

配置aws工具访问权限∞ (Configure aws tool access ∞)

The quickest way to configure the AWS CLI is to run the aws configure command:

配置AWS CLI的最快方法是运行aws configure命令：

$ aws configure
AWS Access Key ID: foo
AWS Secret Access Key: bar
Default region name [us-west-2]: us-west-2
Default output format [None]: json

Here, your AWS Access Key ID and AWS Secret Access Key can be found in Your Security Credentials on the AWS Console.

在这里，可以在AWS控制台上的“ 安全凭证”中找到您的AWS Access Key ID和AWS Secret Access Key 。

上传大文件∞ (Uploading large files ∞)

Lastly, the fun comes. Here, assume we are uploading the large ./150GB.data to s3://systut-data-test/store_dir/ (that is, directory store-dir under bucket systut-data-test) and the bucket and directory are already created on S3. The command is:

最后，乐趣来了。 在这里，假设我们将大型./150GB.data上传到s3://systut-data-test/store_dir/ （即bucket systut-data-test下的目录store-dir ），并且存储桶和目录已经创建在S3上。 该命令是：

$ aws s3 cp ./150GB.data s3://systut-data-test/store_dir/

After it starts to upload the file, it will print the progress message like

开始上传文件后，它将打印进度消息，例如

Completed 1 part(s) with ... file(s) remaining

at the beginning, and the progress message as follows when it is reaching the end.

开头，进度消息到达末尾时如下所示。

Completed 9896 of 9896 part(s) with 1 file(s) remaining

After it successfully uploads the file, it will print a message like

成功上传文件后，它将显示一条消息，例如

upload: ./150GB.data to s3://systut-data-test/store_dir/150GB.data

aws has more commands to operate files on S3. I hope this tutorial helps you start with it. Check the manual for more details.

aws具有更多命令来操作S3上的文件。 我希望本教程可以帮助您开始。 有关更多详细信息，请查阅手册 。

翻译自: https://www.systutorials.com/uploading-large-files-amazon-s3-aws-cli/

aws s3 cli