FastDFS入门

概述

分布式文件系统

分布式文件系统(Distributed File System)是一个软件服务器,这个软件可以用来管理文件。但这个软件所管理的文件通常不是在一个服务器节点上,而是在多个服务器节点上,这些服务器节点通过网络相连构成一个庞大的文件存储器集群,这些服务器都用于存储文件资源,通过分布式文件系统来管理这些服务器上的文件。
分布式文件系统与传统文件系统对比。
在这里插入图片描述
传统方式弊端:

  1. 如果用户数量多,IO操作比较多,对磁盘访问压力很大。
  2. 如果磁盘发生故障,会造成数据丢失。
  3. 存储容量有限。

FastDFS简介

FastDFS代码托管在github上:

https://github.com/happyfish100/fastdfs

FastDFS整体架构

FastDFS文件系统由两大部分构成,一个是客户端,一个是服务端。

客户端

客户端通常指我们的程序,比如我们的Java程序去连接FastDFS、操作FastDFS,那我们的Java程序就是一个客户端,FastDFS提供专有API访问,目前提供了C、Java和PHP几种编程语言的API,用来访问FastDFS文件系统。

服务端

服务端由两个部分构成:一个跟踪器(tracker),一个存储节点(storage)。
跟踪器(tracker)主要做调度工作,在内存中记录集群中存储节点storage的状态信息,是前端client和后代存储节点storage的枢纽。因为相关信息全部在内存中,Tracker server的性能非常高,一个较大的机器(比如上百个group)中有3台就足够了。
存储节点(storage)用于存储文件,包括文件和文件属性(meta data)都保存到存储服务器磁盘上,完成文件管理的所有功能:文件存储、文件同步和提供文件访问等。

FastDFS环境搭建

FastDFS安装

安装前的准备

  1. 检查Linux上是否安装了gcc、libevent、libevent-devel
    • yum list installed | grep gcc
    • yum list installed | grep libevent
    • yum list installed | grep libevent-devel
  2. 如果没有安装,则需要进行安装
    yum install gcc libevent libevent-devel -y

安装libfastcommon库

libfastcommon库是FastDFS文件系统运行需要的公共C语言函数库。
下载地址:https://github.com/happyfish100

[root@localhost fdfs]# tar -zxvf libfastcommon-1.0.49.tar.gz
[root@localhost fdfs]# ls
fastdfs-6.07.tar.gz  libfastcommon-1.0.49  libfastcommon-1.0.49.tar.gz
[root@localhost fdfs]# cd libfastcommon-1.0.49/
[root@localhost libfastcommon-1.0.49]# ls
doc      INSTALL             LICENSE  php-fastcommon  src
HISTORY  libfastcommon.spec  make.sh  README
  • 执行make 脚本进行编译
[root@localhost libfastcommon-1.0.49]# ./make.sh

注意:make编译的时候如果报错,需解决错误后再次进行make,通常发生错误是由于Linux缺少某些依赖库导致,根据错误提示解决错误

  • 执行make install 进行安装
[root@localhost libfastcommon-1.0.49]# ./make.sh install

安装FastDFS

FastDFS没有Windows版本,不能在Windows下使用。
FastDFS需要安装部署在Linux环境下,博主使用的是6.07版本。
下载地址:https://github.com/happyfish100/fastdfs/tree/V6.07

[root@localhost fdfs]# tar -zxvf fastdfs-6.07.tar.gz
[root@localhost fdfs]# cd fastdfs-6.07/
[root@localhost fastdfs-6.07]# ls
client  COPYING-3_0.txt  HISTORY  INSTALL     README.md     setup.sh  test
common  docker           images   make.sh     README_zh.md  stop.sh   tracker
conf    fastdfs.spec     init.d   php_client  restart.sh    storage
[root@localhost fastdfs-6.07]#
  • 执行make脚本进行编译
[root@localhost fastdfs-6.07]# ./make.sh
  • 执行make install 进行安装
[root@localhost fastdfs-6.07]# ./make.sh install

FastDFS安装已经完成了。
所有编译出来的文件存放在/usr/bin目录下
所有配置文件存放在/etc/fdfs目录下

# 后期服务器的监控、管理等都需要使用这里面的程序
[root@localhost bin]# pwd
/usr/bin
[root@localhost bin]# ll | grep fdfs
-rwxr-xr-x. 1 root root      462128 May 15 11:17 fdfs_appender_test
-rwxr-xr-x. 1 root root      461960 May 15 11:17 fdfs_appender_test1
-rwxr-xr-x. 1 root root      448736 May 15 11:17 fdfs_append_file
-rwxr-xr-x. 1 root root      445520 May 15 11:17 fdfs_crc32
-rwxr-xr-x. 1 root root      448768 May 15 11:17 fdfs_delete_file
-rwxr-xr-x. 1 root root      449752 May 15 11:17 fdfs_download_file
-rwxr-xr-x. 1 root root      449352 May 15 11:17 fdfs_file_info
-rwxr-xr-x. 1 root root      469904 May 15 11:17 fdfs_monitor
-rwxr-xr-x. 1 root root      448936 May 15 11:17 fdfs_regenerate_filename
-rwxr-xr-x. 1 root root     1621248 May 15 11:17 fdfs_storaged
-rwxr-xr-x. 1 root root      471680 May 15 11:17 fdfs_test
-rwxr-xr-x. 1 root root      471000 May 15 11:17 fdfs_test1
-rwxr-xr-x. 1 root root      655368 May 15 11:17 fdfs_trackerd
-rwxr-xr-x. 1 root root      449496 May 15 11:17 fdfs_upload_appender
-rwxr-xr-x. 1 root root      450872 May 15 11:17 fdfs_upload_file
# /usr/bin是Linux的环境变量,可通过echo $PATH查看
[root@localhost fdfs]# echo $PATH
/home/gxm/.local/bin:/home/gxm/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
[root@localhost fdfs]#

# 所有编译出的配置文件
[root@localhost fdfs]# pwd
/etc/fdfs
[root@localhost fdfs]# ls
client.conf.sample   storage_ids.conf.sample
storage.conf.sample  tracker.conf.sample
  • 注意需要把解压后的http.conf,mime.types拷贝到/etc/fdfs/目录下,否则后续会有很多奇怪问题不好解决。
[root@localhost fdfs]# cd fastdfs-6.07/
[root@localhost fastdfs-6.07]# ls
client  COPYING-3_0.txt  HISTORY  INSTALL     README.md     setup.sh  test
common  docker           images   make.sh     README_zh.md  stop.sh   tracker
conf    fastdfs.spec     init.d   php_client  restart.sh    storage
[root@localhost fastdfs-6.07]# cd conf/
[root@localhost conf]# ls
anti-steal.jpg  http.conf   storage.conf      tracker.conf
client.conf     mime.types  storage_ids.conf
[root@localhost conf]# cp http.conf /etc/fdfs
[root@localhost conf]# cp mime.types /etc/fdfs
[root@localhost conf]#
[root@localhost conf]# cd /etc/fdfs
[root@localhost fdfs]# ls
client.conf.sample  mime.types           storage_ids.conf.sample
http.conf           storage.conf.sample  tracker.conf.sample

FastDFS配置

去掉/etc/fdfs/ 目录下FastDFS配置文件的后缀名

[root@localhost fdfs]# ll
total 68
-rw-r--r--. 1 root root  1909 May 15 11:17 client.conf.sample
-rw-r--r--. 1 root root   965 May 15 11:29 http.conf
-rw-r--r--. 1 root root 31172 May 15 11:29 mime.types
-rw-r--r--. 1 root root 10246 May 15 11:17 storage.conf.sample
-rw-r--r--. 1 root root   620 May 15 11:17 storage_ids.conf.sample
-rw-r--r--. 1 root root  9138 May 15 11:17 tracker.conf.sample
[root@localhost fdfs]# mv client.conf.sample client.conf
[root@localhost fdfs]# mv storage.conf.sample storage.conf
[root@localhost fdfs]# mv storage_ids.conf.sample storage_ids.conf
[root@localhost fdfs]# mv tracker.conf.sample tracker.conf
[root@localhost fdfs]# ll
total 68
-rw-r--r--. 1 root root  1909 May 15 11:17 client.conf
-rw-r--r--. 1 root root   965 May 15 11:29 http.conf
-rw-r--r--. 1 root root 31172 May 15 11:29 mime.types
-rw-r--r--. 1 root root 10246 May 15 11:17 storage.conf
-rw-r--r--. 1 root root   620 May 15 11:17 storage_ids.conf
-rw-r--r--. 1 root root  9138 May 15 11:17 tracker.conf
[root@localhost fdfs]#

修改tracker.conf文件

默认指向的FastDFS作者余庆的目录,我们手动改一下
base_path=/opt/fastdfs/tracker # 配置tracker日志、数据存储等目录

# the base path to store data and log files
base_path = /opt/fastdfs/tracker

修改storage.conf文件

bash_bash=/opt/fastdfs/storage # storage存储数据、日志目录
store_path0=/opt/fastdfs/storage/files # 真正存放文件的目录
tracker_server=192.168.174.128:22122 # 注册当前存储节点的跟踪器地址(最后绑定安装tracker服务器的内外地址)

# the base path to store data and log files
# NOTE: the binlog files maybe are large, make sure
#       the base path has enough disk space,
#       eg. the disk free space should > 50GB
base_path = /opt/fastdfs/storage

# store path (disk or mount point) count, default value is 1
store_path_count = 1

# store_path#, based on 0, to configure the store paths to store files
# if store_path0 not exists, it's value is base_path (NOT recommended)
# the paths must be exist.
#
# IMPORTANT NOTE:
#       the store paths' order is very important, don't mess up!!!
#       the base_path should be independent (different) of the store paths

store_path0 = /opt/fastdfs/storage/files
#store_path1 = /home/yuqing/fastdfs2

# subdir_count  * subdir_count directories will be auto created under each
# store_path (disk), value can be 1 to 256, default value is 256
subdir_count_per_path = 256

# tracker_server can ocur more than once for multi tracker servers.
# the value format of tracker_server is "HOST:PORT",
#   the HOST can be hostname or ip address,
#   and the HOST can be dual IPs or hostnames seperated by comma,
#   the dual IPS must be an inner (intranet) IP and an outer (extranet) IP,
#   or two different types of inner (intranet) IPs.
#   for example: 192.168.2.100,122.244.141.46:22122
#   another eg.: 192.168.1.10,172.17.4.21:22122

tracker_server = 192.168.174.128:22122

在linux服务器上创建上面指定的目录(fdfs不会自动创建,需要手动创建)

[root@localhost opt]# tree fastdfs/
fastdfs/
├── storage
│   └── files
└── tracker

3 directories, 0 files

启动FastDFS

启动FastDFS的tracker服务

在任意目录下执行:fdfs_trackerd /etc/fdfs/tracker.conf

[root@localhost opt]# fdfs_trackerd /etc/fdfs/tracker.conf
[root@localhost opt]# ps -ef|grep fdfs
root       6198      1  0 14:56 ?        00:00:00 fdfs_trackerd /etc/fdfs/tracker.conf
root       6214   2666  0 14:56 pts/0    00:00:00 grep --color=auto fdfs
[root@localhost opt]#

启动FastDFS的storage服务

在任意目录下执行:fdfs_storaged /etc/fdfs/storage.conf

[root@localhost opt]# ps -ef|grep fdfs
root       6198      1  0 14:56 ?        00:00:00 fdfs_trackerd /etc/fdfs/tracker.conf
root       6229      1 99 14:58 ?        00:00:02 fdfs_storaged /etc/fdfs/storage.conf

查看storage是否已经注册到tracker下

[root@localhost opt]# fdfs_monitor /etc/fdfs/storage.conf
[2021-05-15 15:00:15] DEBUG - base_path=/opt/fastdfs/storage, connect_timeout=5, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=1, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 192.168.174.128:22122

group count: 1

Group 1:
group name = group1
disk total space = 39,164 MB
disk free space = 31,016 MB
trunk free space = 0 MB
storage server count = 1
active server count = 1
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.174.128
                ip_addr = 192.168.174.128  ACTIVE

首次启动storage后,会在配置的路径下创建存储文件的目录

[root@localhost files]# ls data/
00  0D  1A  27  34  41  4E  5B  68  75  82  8F  9C  A9  B6  C3  D0  DD  EA  F7
01  0E  1B  28  35  42  4F  5C  69  76  83  90  9D  AA  B7  C4  D1  DE  EB  F8
02  0F  1C  29  36  43  50  5D  6A  77  84  91  9E  AB  B8  C5  D2  DF  EC  F9
03  10  1D  2A  37  44  51  5E  6B  78  85  92  9F  AC  B9  C6  D3  E0  ED  FA
04  11  1E  2B  38  45  52  5F  6C  79  86  93  A0  AD  BA  C7  D4  E1  EE  FB
05  12  1F  2C  39  46  53  60  6D  7A  87  94  A1  AE  BB  C8  D5  E2  EF  FC
06  13  20  2D  3A  47  54  61  6E  7B  88  95  A2  AF  BC  C9  D6  E3  F0  FD
07  14  21  2E  3B  48  55  62  6F  7C  89  96  A3  B0  BD  CA  D7  E4  F1  FE
08  15  22  2F  3C  49  56  63  70  7D  8A  97  A4  B1  BE  CB  D8  E5  F2  FF
09  16  23  30  3D  4A  57  64  71  7E  8B  98  A5  B2  BF  CC  D9  E6  F3
0A  17  24  31  3E  4B  58  65  72  7F  8C  99  A6  B3  C0  CD  DA  E7  F4
0B  18  25  32  3F  4C  59  66  73  80  8D  9A  A7  B4  C1  CE  DB  E8  F5
0C  19  26  33  40  4D  5A  67  74  81  8E  9B  A8  B5  C2  CF  DC  E9  F6
[root@localhost files]#

FastDFS重启

重启tracker

[root@localhost files]# fdfs_trackerd /etc/fdfs/tracker.conf restart
waiting for pid [6198] exit ...
starting ...
[root@localhost files]#

重启storage

[root@localhost files]# fdfs_storaged /etc/fdfs/storage.conf restart
waiting for pid [6229] exit ...
starting ...
[root@localhost files]# ps -ef|grep fdfs
root       6450      1  0 15:07 ?        00:00:00 fdfs_trackerd /etc/fdfs/tracker.conf restart
root       6465      1  0 15:08 ?        00:00:00 fdfs_storaged /etc/fdfs/storage.conf restart
root       6475   2666  0 15:08 pts/0    00:00:00 grep --color=auto fdfs
[root@localhost files]#

FastDFS关闭

关闭tracker服务器

[root@localhost files]# fdfs_trackerd /etc/fdfs/tracker.conf stop
waiting for pid [6450] exit ...
pid [6450] exit.
[root@localhost files]# ps -ef|grep fdfs
root       6465      1  0 15:08 ?        00:00:00 fdfs_storaged /etc/fdfs/storage.conf restart
root       6500   2666  0 15:09 pts/0    00:00:00 grep --color=auto fdfs
[root@localhost files]#

关闭storage服务器

[root@localhost files]# fdfs_storaged /etc/fdfs/storage.conf stop
waiting for pid [6465] exit ...
pid [6465] exit.
[root@localhost files]# ps -ef|grep fdfs
root       6518   2666  0 15:10 pts/0    00:00:00 grep --color=auto fdfs
[root@localhost files]#

或者kill关闭fastdfs,但不建议在线上使用kill -9 强制关闭,因为可能会导致文件信息不同步的问题,导致storage重新注册异常。

FastDFS测试

FastDFS安装完成后,可使用fdfs_test脚本测试文件上传

测试之前,需要修改client.conf配置文件

  • bash_path=/opt/fastdfs/client
  • tracker_server=192.168.174.128:22122
[root@localhost fastdfs]# mkdir client
[root@localhost fastdfs]# ls
client  storage  tracker
[root@localhost fastdfs]#

测试文件上传

  • 准备需要上传的文件
[root@localhost test]# pwd
/opt/test
[root@localhost test]# ls
aa.txt
[root@localhost test]#
  • 执行上传命令 fdfs_test /etc/fdfs/client.conf upload /opt/test/aa.txt
[root@localhost test]# fdfs_test /etc/fdfs/client.conf upload /opt/test/aa.txt
This is FastDFS client test program v6.07

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.fastken.com/
for more detail.

[2021-05-15 15:36:13] DEBUG - base_path=/opt/fastdfs/client, connect_timeout=5, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

tracker_query_storage_store_list_without_group:
        server 1. group_name=, ip_addr=192.168.174.128, port=23000

group_name=group1, ip_addr=192.168.174.128, port=23000
storage_upload_by_filename
group_name=group1, remote_filename=M00/00/00/wKiugGCfee2AS5jEAAAAKX7RhFU983.txt
source ip address: 192.168.174.128
file timestamp=2021-05-15 15:36:13
file size=41
file crc32=2127660117
example file url: http://192.168.174.128/group1/M00/00/00/wKiugGCfee2AS5jEAAAAKX7RhFU983.txt
storage_upload_slave_by_filename
group_name=group1, remote_filename=M00/00/00/wKiugGCfee2AS5jEAAAAKX7RhFU983_big.txt
source ip address: 192.168.174.128
file timestamp=2021-05-15 15:36:13
file size=41
file crc32=2127660117
example file url: http://192.168.174.128/group1/M00/00/00/wKiugGCfee2AS5jEAAAAKX7RhFU983_big.txt
[root@localhost test]#
  • 切换到存储目录查看文件上传情况
[root@localhost test]# cd /opt/fastdfs/storage/files/data/00/00/
[root@localhost 00]# ls
wKiugGCfee2AS5jEAAAAKX7RhFU983_big.txt    wKiugGCfee2AS5jEAAAAKX7RhFU983.txt
wKiugGCfee2AS5jEAAAAKX7RhFU983_big.txt-m  wKiugGCfee2AS5jEAAAAKX7RhFU983.txt-m
[root@localhost 00]#

可以看看文件已经存储到fdfs storage服务器上了。

FastDFS生成的文件目录结构及名称实例

在这里插入图片描述

测试文件删除

fdfs_delete_file /etc/fdfs/client.conf group名称/要删除的文件路径

[root@localhost test]# cd /opt/fastdfs/storage/files/data/00/00/
[root@localhost 00]# ls
wKiugGCfee2AS5jEAAAAKX7RhFU983_big.txt    wKiugGCfee2AS5jEAAAAKX7RhFU983.txt
wKiugGCfee2AS5jEAAAAKX7RhFU983_big.txt-m  wKiugGCfee2AS5jEAAAAKX7RhFU983.txt-m
[root@localhost 00]#
[root@localhost 00]#
[root@localhost 00]#
[root@localhost 00]# fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKiugGCfee2AS5jEAAAAKX7RhFU983.txt
[root@localhost 00]# ls
wKiugGCfee2AS5jEAAAAKX7RhFU983_big.txt  wKiugGCfee2AS5jEAAAAKX7RhFU983_big.txt-m
[root@localhost 00]#

可以看到文件已经被删除了。

注意

  • 没有搭建集群默认只有一个组group1。
  • 后缀名包含-m的为属性文件(meta)。
  • 在Linux中并没有磁盘一说,是虚拟的。
  • 1
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

融极

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值