seaweedfs 调研

简介
SeaweedFS是一个简单并且高度可扩展的分布式文件系统，可以存储数十亿的文件并且快速获得文件，特别适合于有效处理小文件概念：

 

• Node 系统抽象的节点，抽象为DataCenter、Rack、DataNode
• DataCenter 数据中心，对应现实中的不同机房
• Rack 机架，对应现实中的机柜
• Datanode 存储节点，用于管理、存储逻辑卷
• Volume 逻辑卷，存储的逻辑结构，逻辑卷下存储Needle
• Needle 逻辑卷中的Object，对应存储的文件
• Collection 文件集，可以分布在多个逻辑卷上

备份策略

weed-fs提供了若干种replication策略(rack – 机架，一个逻辑上的概念)：
000 no replication, just one copy
001 replicate once on the same rack
010 replicate once on a different rack in the same data center
100 replicate once on a different data center
200 replicate twice on two other different data center
110 replicate once on a different rack, and once on a different data center

集群模式：注意选择的备份策略，不同的策略需要不同的数据中心机器。否则会报“Cannot grow volume group! No enough data node found” 错误
master: earth139, earth140, earth141(三台机器)

./weed master -ip 10.141.160.93 -port 9333 -mdir ./weedfs_master -peers 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -defaultReplication 100 >master.log 2>&1 &

./weed master -ip 10.141.160.94 -port 9333 -mdir ./weedfs_master -peers 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -defaultReplication 100 >master.log 2>&1 &

./weed master -ip 10.141.160.97 -port 9333 -mdir ./weedfs_master -peers 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -defaultReplication 100 >master.log 2>&1 &

volume:earth139, earth140, earth141

./weed volume -ip 10.141.160.93 -port 9222 -dir ./weedfs_volume -mserver 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -dataCenter dc1 -rack rack1 >volume.log 2>&1 &

./weed volume -ip 10.141.160.94 -port 9222 -dir ./weedfs_volume -mserver 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -dataCenter dc1 -rack rack1 >volume.log 2>&1 &

./weed volume -ip 10.141.160.97 -port 9222 -dir ./weedfs_volume -mserver 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -dataCenter dc2 -rack rack1 >volume.log 2>&1 &

搭建好后，浏览器打开master节点 ip:port看到

filer: 文件服务
 

./weed scaffold -config filer -output="." // 生成 filer.toml 文件

./weed filer -master 10.141.160.93:9333,10.141.160.94:9333,10.141.160.95:9333 -port 8888

浏览器访问 http://10.141.160.93:8888/

 

也可以mount filer到某个目录进行使用

./weed mount -filer="10.141.160.93:8888" -dir="./mountp_seaweed/"

filer server都是工作在standalone模式下的。可使用redis或Cassandra作为后端，同步，需要在

存文件

curl -F file=@master.log http://10.141.160.93:9333/submit

{"eTag":"8ffdca8b","fid":"1,02fd956ac2","fileName":"master.log","fileUrl":"10.141.160.94:9222/1,02fd956ac2","size":1165}

也可以请求一个fid，再传

curl -X POST http://10.141.160.93:9333/dir/assign

{"fid":"2,0194c2cb41","url":"10.141.160.94:9222","publicUrl":"10.141.160.94:9222","count":1}

curl -X PUT -F file=@2.out http://10.141.160.94:9222/2,0194c2cb41

获取

curl http://10.141.160.94:9222/1,02fd956ac2

或者浏览器打开 http://10.141.160.94:9222/1,02fd956ac2

删除

curl -X DELETE http://10.141.160.94:9222/1,02fd956ac2

垃圾回收

一般删除后，卷的大小并不会马上更新，当然可手动发起更新大小操作

curl "http://10.141.160.93:9333/vol/vacuum"

参考

https://tonybai.com/2015/08/22/intro-of-using-weedfs/

https://blog.csdn.net/DPnice/article/details/84990050

https://www.jianshu.com/p/8adda9b8856e

https://cloud.tencent.com/developer/article/1363021

https://blog.yasking.org/a/weed-fs.html

https://github.com/bingoohuang/blog/issues/57