SeaweedFS 分布式存储使用s3cmd管理对象存储

_Johnny_

已于 2024-08-29 17:56:00 修改

阅读量275

点赞数 4

分类专栏： Data 文章标签：分布式

于 2024-08-26 17:53:53 首次发布

本文链接：https://blog.csdn.net/z_johnny/article/details/141569687

版权

Data 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

使用s3 browser无法连接SeaweedFS ，采用官方推荐的Program S3cmd，S3cmd S3 Sync How-To

S3cmd（S3cmd）是一个免费的命令行工具和客户端，用于在Amazon S3和其他使用S3协议的云存储服务提供商（如Google cloud storage
或DreamHost DreamObjects）中上传、检索和管理数据。它最适合熟悉命令行程序的高级用户。它也是批处理脚本和从cron触发的S3自动备份的理想选择。

S3cmd是用Python编写的。这是一个遵循GNU公共许可证v2（GPLv2）的开源项目，可供商业和私人使用。您只需向亚马逊支付使用其存储空间的费用。

自2008年首次发布以来，S3cmd增加了许多功能和选项……我们最近统计了60多个命令行选项，包括多部分上传、加密、增量备份、s3同步、ACL和元数据管理、s3存储桶大小、存储桶策略等等！

开源地址：https://github.com/s3tools/s3cmd
官网：https://s3tools.org/s3cmd

功能特性

易于安装和配置：只需要简单的几个步骤即可完成 s3cmd 的安装和配置。
多平台支持：s3cmd 支持 Windows, Linux 和macOS 平台。
丰富的命令选项：提供了大量的命令选项以满足不同需求，如指定上传/下载速率限制、设置文件权限等。
SSL加密：所有的数据传输都通过 SSL 进行加密，保证了数据的安全性。
可扩展性：可以通过编写自定义脚本或插件来实现更复杂的功能。

使用场景

备份和恢复：使用 s3cmd 将本地文件系统备份到 S3 存储上，或者从 S3 存储中恢复数据。
CDN 配置：将静态资源（如图片、CSS 文件）上传至 S3 存储，然后利用 CDN 提供快速的全球访问。
日志存储：将服务器日志直接发送到 S3 存储，便于集中管理和分析。
云存储迁移：将其他云存储提供商的数据迁移到 Amazon S3。

下载

1、对于 Linux 用户，可以使用 apt-get 或 yum 软件仓库进行安装：

sudo apt-get install s3cmd # Debian/Ubuntu
sudo yum install s3cmd   # CentOS/RHEL

2、对于 macOS 用户，请使用 Homebrew 进行安装：

brew install s3cmd

3、离线下载

参考：https://github.com/s3tools/s3cmd/blob/master/INSTALL.md

4、对于 Windows 用户，请访问 GitHub Releases https://github.com/s3tools/s3cmd/releases 页面下载最新版本的 .zip 文件并解压。

配置

首先需要生成一个名为 .s3cfg 的配置文件，其中包含了您的 AWS 访问密钥和安全密钥。运行如下命令以创建配置文件：

s3cmd --configure

根据提示输入相关信息，包括访问密钥、安全密钥等。

配置的域名按需配置host后在进行Test access. /etc/hosts

对于我的环境：

进行配置：Access Key,Secret Key
S3 Endpoint是必须填写带端口的，如seaweedfstest.com:10003 ,同时在/etc/hosts中配置host映射后在进行测试
Use HTTPS protocol输入no，其他默认即可

root@installerdev04:/home/jupiter# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key [ZwRZMJTiTcKAlxvK]: mMiShWTub4WfdLyq
Secret Key [ZwRZMJTiTcKAlxvK]: mMiShWTub4WfdLyq
Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [seaweedfs.com:10003]: seaweedfstest.com:10003                

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [seaweedfs.com]: seaweedfstest.com:10003

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [No]: 

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: mMiShWTub4WfdLyq
  Secret Key: mMiShWTub4WfdLyq
  Default Region: US
  S3 Endpoint: seaweedfstest.com:10003
  DNS-style bucket+hostname:port template for accessing a bucket: seaweedfstest.com:10003
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

如果想从新配置，删除配置rm /root/.s3cfg

# 删除配置
root@installerdev04:/home/jupiter# rm /root/.s3cfg 
root@installerdev04:/home/jupiter# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
# 输入Access Key、Secret Key
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: LKMt8BSGsgqoOoBl
Secret Key: LKMt8BSGsgqoOoBl
# 保持默认，直接回车
Default Region [US]: 
# 输入带端口的域名
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: seaweedfs.com:10003
# 域名一致就行
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: seaweedfs.com:10003
# 保持默认，直接回车
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 
# 按需，这里就输入no
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no
# 保持默认，直接回车
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: LKMt8BSGsgqoOoBl
  Secret Key: LKMt8BSGsgqoOoBl
  Default Region: US
  S3 Endpoint: seaweedfs.com:10003
  DNS-style bucket+hostname:port template for accessing a bucket: seaweedfs.com:10003
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0
# 先去/etc/hosts中增加记录，在输入Y
Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.
# 输入y进行保存
Save settings? [y/N] Y
Configuration saved to '/root/.s3cfg'

# 验证连接成功
root@installerdev04:/home/jupiter# s3cmd ls
root@installerdev04:/home/jupiter# s3cmd mb s3://test
Bucket 's3://test/' created

测试命令

# 创建bucket
root@installerdev04:/home/jupiter# s3cmd mb s3://mybucket
Bucket 's3://mybucket/' created

# 列出bucket
root@installerdev04:/home/jupiter# s3cmd ls
2024-08-26 09:50  s3://mybucket

# 导入文件到bucket
root@installerdev04:/home/jupiter# echo "hello123" > hello.txt
root@installerdev04:/home/jupiter# s3cmd put hello.txt s3://mybucket
upload: 'hello.txt' -> 's3://mybucket/hello.txt'  [1 of 1]
 9 of 9   100% in    0s    40.18 B/s  done
## 如果bucket没有创建，在上传文件时会自动创建，如
root@installerdev04:/home/jupiter# s3cmd ls
2024-08-29 09:48  s3://bucket
2024-08-29 09:48  s3://bucket11
2024-08-29 09:49  s3://bucket111
2024-08-29 09:49  s3://bucket11144444
2024-08-29 09:48  s3://bucket22
root@installerdev04:/home/jupiter# s3cmd put hello.txt s3://new
upload: 'hello.txt' -> 's3://new/hello.txt'  [1 of 1]
 1310 of 1310   100% in    0s     5.35 KB/s  done
root@installerdev04:/home/jupiter# s3cmd ls
2024-08-29 09:48  s3://bucket
2024-08-29 09:48  s3://bucket11
2024-08-29 09:49  s3://bucket111
2024-08-29 09:49  s3://bucket11144444
2024-08-29 09:48  s3://bucket22
2024-08-29 09:54  s3://new
root@installerdev04:/home/jupiter# s3cmd ls s3://new
2024-08-29 09:54         1310  s3://new/hello.txt


 test/
└── buckets
    ├── bucket
    │   └── 222.txt
    ├── bucket11
    │   └── hello.txt
    ├── bucket111
    │   └── hello.txt
    ├── bucket11144444
    │   └── hello.txt
    ├── bucket22
    └── new
        └── hello.txt

7 directories, 5 file
 
# 列出bucket中的文件
root@installerdev04:/home/jupiter# s3cmd ls s3://mybucket
2024-08-26 09:51            9  s3://mybucket/hello.txt

root@installerdev04:/home/jupiter# s3cmd la
2024-08-27 06:17            9  s3://bucket11/hello.txt

# 导入文件夹到bucket并递归查看
root@installerdev04:/home/jupiter# tree s3file/
s3file/
├── 111.txt
├── 222.txt
├── 333.txt
├── hello.txt
└── test
    └── test.txt

1 directory, 5 files
## 递归文件夹导入
root@installerdev04:/home/jupiter# s3cmd put s3file/ --recursive s3://test
upload: 's3file/111.txt' -> 's3://test/111.txt'  [1 of 5]
 0 of 0     0% in    0s     0.00 B/s  done
upload: 's3file/222.txt' -> 's3://test/222.txt'  [2 of 5]
 0 of 0     0% in    0s     0.00 B/s  done
upload: 's3file/333.txt' -> 's3://test/333.txt'  [3 of 5]
 0 of 0     0% in    0s     0.00 B/s  done
upload: 's3file/hello.txt' -> 's3://test/hello.txt'  [4 of 5]
 9 of 9   100% in    0s   545.45 B/s  done
upload: 's3file/test/test.txt' -> 's3://test/test/test.txt'  [5 of 5]
 0 of 0     0% in    0s     0.00 B/s  done
root@installerdev04:/home/jupiter# s3cmd ls s3://test
                          DIR  s3://test/test/
2024-08-27 01:46            0  s3://test/111.txt
2024-08-27 01:46            0  s3://test/222.txt
2024-08-27 01:46            0  s3://test/333.txt
2024-08-27 01:46            9  s3://test/hello.txt
## 递归查看
root@installerdev04:/home/jupiter# s3cmd ls --recursive s3://test
2024-08-27 01:46            0  s3://test/111.txt
2024-08-27 01:46            0  s3://test/222.txt
2024-08-27 01:46            0  s3://test/333.txt
2024-08-27 01:46            9  s3://test/hello.txt
2024-08-27 01:46            0  s3://test/test/test.txt
root@installerdev04:/home/jupiter# 


# 下载文件
root@installerdev04:/home/jupiter# ls s3file/
root@installerdev04:/home/jupiter# s3cmd get s3://test/hello.txt s3file/
download: 's3://test/hello.txt' -> 's3file/hello.txt'  [1 of 1]
 9 of 9   100% in    0s  1053.12 B/s  done
root@installerdev04:/home/jupiter# ls s3file/
hello.txt

root@installerdev04:/home/jupiter# s3cmd get s3://bucket11/hello.txt hello000.txt
download: 's3://bucket11/hello.txt' -> 'hello000.txt'  [1 of 1]
 9 of 9   100% in    0s   988.36 B/s  done
 
root@installerdev04:/home/jupiter# s3cmd get s3://bucket22/222.txt --force 333.txt 
download: 's3://bucket22/222.txt' -> '333.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done 

# 删除bucket文件
root@installerdev04:/home/jupiter# s3cmd ls s3://test
2024-08-27 01:30            9  s3://test/hello.txt
root@installerdev04:/home/jupiter# 
root@installerdev04:/home/jupiter# s3cmd del s3://test/hello.txt
delete: 's3://test/hello.txt'
root@installerdev04:/home/jupiter# s3cmd ls s3://test
root@installerdev04:/home/jupiter# 

# 删除bucket
root@installerdev04:/home/jupiter# s3cmd rb s3://mybucket
Bucket 's3://mybucket/' removed
root@installerdev04:/home/jupiter# s3cmd ls
root@installerdev04:/home/jupiter#

ref
https://github.com/s3tools/s3cmd
https://s3tools.org/s3cmd-sync
https://s3tools.org/usage
https://github.com/s3tools/s3cmd/blob/master/INSTALL.md
https://www.bookstack.cn/read/seaweedfs-wiki/23ddfe9ce403ead6.md

_Johnny_

关注

4
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
SeaweedFS 分布式存储使用s3cmd管理对象存储

S3cmd（S3cmd）是一个免费的命令行工具和客户端，用于在Amazon S3和其他使用S3协议的云存储服务提供商（如Google cloud storage或DreamHost DreamObjects）中上传、检索和管理数据。它最适合熟悉命令行程序的高级用户。它也是批处理脚本和从cron触发的S3自动备份的理想选择。S3cmd是用Python编写的。这是一个遵循GNU公共许可证v2（GPLv2）的开源项目，可供商业和私人使用。您只需向亚马逊支付使用其存储空间的费用。
复制链接

扫一扫

专栏目录