分布式文件存储FastDFS使用
功能包括:文件存储、文件同步、文件上传下载等适合中小文件(建议范围:4KB < file_size <500MB),对以文件为载体的在线服务,如相册网站、视频网站等等具有显著的效果。
一、FastDFS架构
Tracker
FastDFS的协调者,负责管理所有的storage server和group。tracker根据storage的心跳信息,建立group==>[storage serverlist]的映射表。
Storage
以group为单位组织,所有group的文件容量累加就是整个存储系统中的文件容量。当存储空间不足或即将耗尽时,可以动态添加group。一个group内包含多台storage机器,数据互为备份,storage尽量配置相同。以group为单位组织存储能方便的进行应用隔离、负载均衡、冗余备份。storage的存储依赖于本地文件系统,可配置多个数据存储目录(挂载磁盘)。
client
客户端,作为业务请求的发起方,通过专有接口,使用TCP/IP协议与跟踪器服务器或存储节点进行数据交互。
二、FastDFS的文件同步
写文件时,客户端将文件写至group内一个storage server即认为写文件成功,storage server写完文件后,会由后台线程将文件同步至同group内其他的storage server。
每个storage写文件后,同时会写一份binlog,binlog里不包含文件数据,只包含文件名等元信息,这份binlog用于后台同步,storage会记录向group内其他storage同步的进度,以便重启后能接上次的进度继续同步;进度以时间戳的方式进行记录,所以最好能保证集群内所有server的时钟保持同步。
三、fastdfs集群安装使用
192.168.1.11 tracker、nginx、client
192.168.1.12 tracker、nginx
192.168.1.13 storage、Group 1-1、nginx
192.168.1.14 storage、Group 1-2、nginx
192.168.1.15 storage、Group 2-1、nginx
192.168.1.16 storage、Group 2-2、nginx
192.168.1.100 VIP 192.168.1.10,keepalived高可用、nginx
192.168.1.200 VIP 192.168.1.10,keepalived高可用、nginx
1、准备依赖(所有集群节点)
[root@tracker1 ~]# yum -y install libevent libevent-devel perl make gcc zlib zlib-devel pcre pcre-devel gcc-c++ openssl-devel
#安装libfastcommon(公共C函数库)
[root@tracker1 ~]# wget https://github.com/happyfish100/libfastcommon/archive/V1.0.43.tar.gz
[root@tracker1 ~]# tar xf libfastcommon-1.0.43.tar.gz
[root@tracker1 ~]# cd libfastcommon-1.0.43 && ./make.sh && ./make.sh install
#安装fastdfs
[root@tracker1 ~]# wget https://github.com/happyfish100/fastdfs/archive/V6.06.tar.gz
[root@tracker1 ~]# tar xf V6.06.tar.gz
[root@tracker1 ~]# cd fastdfs-6.06 && ./make.sh && ./make.sh install
[root@tracker1 ~]# cp -r conf/* /etc/fdfs
[root@tracker1 fastdfs-6.06]# ls /etc/fdfs/
client.conf.sample storage.conf.sample storage_ids.conf.sample tracker.conf.sample
[root@tracker1 fastdfs-6.06]# ls conf/
anti-steal.jpg client.conf http.conf mime.types storage.conf storage_ids.conf tracker.conf
2、配置FastDFS跟踪器Tracker
[root@tracker1 ~]# vim /etc/fdfs/tracker.conf
...
disabled=false #启用配置文件
port = 22122 #默认服务器端口22122,一般不修改
base_path =/usr/local/fastdfs/tracker #存储日志和数据的根目录
...
[root@tracker1 ~]# mkdir -p /usr/local/fastdfs/{storage,tracker,client}
[root@tracker1 ~]# /etc/init.d/fdfs_trackerd start
Reloading systemd: [ OK ]
Starting fdfs_trackerd (via systemctl): [ OK ]
[root@tracker1 ~]# ps -ef | grep tracker
[root@tracker1 ~]# tail /usr/local/fastdfs/tracker/logs/trackerd.log
[root@tracker1 ~]# /etc/init.d/fdfs_trackerd --help
Usage: /etc/init.d/fdfs_trackerd {start|stop|status|restart|condrestart}
#设置 FastDFS 跟踪器开机启动:
[root@tracker1 ~]# vim /etc/rc.d/rc.local
## FastDFS Tracker
/etc/init.d/fdfs_trackerd start
3、配置 FastDFS 存储
[root@storage1 ~]# vim /etc/fdfs/storage.conf
disabled=false #启用配置文件
group_name=group1 #组名(第一组为 group1,第二组为 group2)
port=23000 #storage的端口号,同一个组的storage端口号必须相同
base_path=/usr/local/fastdfs/storage #设置 storage 的日志目录
store_path0=/usr/local/fastdfs/storage #存储路径
store_path_count=1 #存储路径个数,需要和store_path个数匹配
tracker_server=192.168.1.11:22122 #tracker服务器的IP地址和端口
tracker_server=192.168.1.12:22122 #多个tracker直接添加多条配置
http.server_port=8888 #设置http端口号,后面会配合nginx使用
[root@storage1 ~]# /etc/init.d/fdfs_storaged start
Reloading systemd: [ OK ]
Starting fdfs_storaged (via systemctl): [ OK ]
[root@storage1 ~]# tail -f /usr/local/fastdfs/storage/logs/storaged.log
[root@storage1 ~]# /usr/bin/fdfs_monitor /etc/fdfs/storage.conf #查看storage节点
...
server_count=1, server_index=0
tracker server is 192.168.1.11:22122
group count: 2
Group 1:
group name = group1
disk total space = 17,394 MB
disk free space = 13,453 MB
....
Storage 1:
id = 192.168.1.13
ip_addr = 192.168.1.13 ACTIVE
http domain =
version = 6.06
...
[root@storage1 ~]# /usr/bin/fdfs_monitor /etc/fdfs/storage.conf | grep ip_addr
ip_addr = 192.168.1.13 ACTIVE
ip_addr = 192.168.1.14 ACTIVE
ip_addr = 192.168.1.15 ACTIVE
ip_addr = 192.168.1.16 ACTIVE #一定要active
设置 FastDFS 存储器开机启动:
[root@storage1 ~]# vim /etc/rc.d/rc.local
## FastDFS Storage
/etc/init.d/fdfs_storaged start
4、client文件上传测试
[root@tracker1 ~]# vim /etc/fdfs/client.conf #tracker1上测试即可
base_path = /usr/local/fastdfs/client
tracker_server = 192.168.1.11:22122
tracker_server = 192.168.1.12:22122
[root@tracker1 ~]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /root/fdfs6.06.tar.gz
group1/M00/00/00/wKgBC2OdJ7eARD-YAAxZcPR00vw.tar.gz
#返回文件ID,说明文件上传成功
#返回的文件ID由group、存储目录、两级子目录、fileid、文件后缀名(由客户端指定,主要用于区分文件类型)拼接而成。
[root@tracker1 ~]# ls /usr/local/fastdfs/storage/data/
00 0D 1A 27 34 41 4E 5B 68 75 82 8F 9C A9 B6 C3 D0 DD EA F7
01 0E 1B 28 35 42 4F 5C 69 76 83 90 9D AA B7 C4 D1 DE EB F8
...
[root@tracker1 ~]# ls /usr/local/fastdfs/storage/data/00/00/wKgBC2OdJ7eARD-YAAxZcPR00vw.tar.gz
/usr/local/fastdfs/storage/data/00/00/wKgBC2OdJ7eARD-YAAxZcPR00vw.tar.gz
5、storage节点实现http访问
[root@storage1 ~]# wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.22.tar.gz
[root@storage1 ~]# tar xf nginx-1.16.1.tar.gz && tar xf V1.22.tar.gz
[root@storage1 ~]# cd nginx-1.16.1/ && useradd -s /sbin/nologin nginx
#配置安装fastdfs-nginx-module模块到nginx
[root@storage1 ~]# ./configure --user=nginx --group=nginx --prefix=/home/nginx --with-http_ssl_module --with-http_stub_status_module --with-stream --add-module=/root/fastdfs-nginx-module-1.22/src/
[root@storage1 ~]# make && make install
[root@storage1 ~]# /home/nginx/sbin/nginx -V
[root@storage1 ~]# cp fastdfs-nginx-module-1.22/src/mod_fastdfs.conf /etc/fdfs/
[root@storage1 ~]# vim /etc/fdfs/mod_fastdfs.conf
#第一组Storage,第二组group_name改为group2,其他一样
connect_timeout=10
base_path=/tmp
tracker_server=192.168.1.11:22122
tracker_server=192.168.1.12:22122
storage_server_port=23000
group_name=group1 #第二组改为group2,其他一样
url_have_group_name = true
store_path0=/usr/local/fastdfs/storage
group_count = 2
[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/usr/local/fastdfs/storage
[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/usr/local/fastdfs/storage
[root@storage1 conf]# vim /home/nginx/conf/nginx.conf
[root@storage1 conf]# /home/nginx/sbin/nginx -t
ngx_http_fastdfs_set pid=17225
nginx: the configuration file /home/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /home/nginx/conf/nginx.conf test is successful
[root@storage1 conf]# /home/nginx/sbin/nginx
ngx_http_fastdfs_set pid=17226
[root@storage1 conf]# ss -antup | grep 8888
tcp LISTEN 0 128 *:8888 *:* users:(("nginx"...
[root@tracker1 ~]# fdfs_upload_file /etc/fdfs/client.conf /root/nginx-1.16.1.tar.gz
group1/M00/00/00/wKgBDmOfNBiANMwzAA_BttUvEqM.tar.gz #上传测试
浏览器访问即可下载
http://192.168.1.13:8888/group1/M00/00/00/wKgBDmOfNBiANMwzAA_BttUvEqM.tar.gz
6、tracker节点安装 Nginx反向代理
主要为了提供 http 访问的反向代理、负载均衡以及缓存服务。
[root@tracker1 ~]# wget http://labs.frickle.com/files/ngx_cache_purge-2.3.tar.gz
[root@tracker1 ~]# tar xf ngx_cache_purge-2.3.tar.gz && tar xf nginx-1.16.1.tar.gz
[root@tracker1 ~]# cd nginx-1.16.1
[root@tracker1 ~]# ./configure --prefix=/usr/local/nginx --add-module=/root/ngx_cache_purge-2.3
[root@tracker1 ~]# make && make install
[root@tracker1 nginx-1.16.1]# /usr/local/nginx/sbin/nginx -V
configure arguments: --prefix=/usr/local/nginx --add-module=/root/ngx_cache_purge-2.3
[root@tracker1 ~]# vim /usr/local/nginx/conf/nginx.conf
user root;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
events {
worker_connections 1024;
use epoll;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
tcp_nopush on;
keepalive_timeout 65;
#gzip on;
#设置缓存
server_names_hash_bucket_size 128;
client_header_buffer_size 32k;
large_client_header_buffers 4 32k;
client_max_body_size 300m;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 16k;
proxy_buffers 4 64k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 128k;
#设置缓存存储路径、存储方式、分配内存大小、磁盘最大空间、缓存期限
proxy_cache_path /usr/local/fastdfs/cache/nginx/proxy_cache levels=1:2 keys_zone=http-cache:200m max_size=1g inactive=30d;
proxy_temp_path /usr/local/fastdfs/cache/nginx/proxy_cache/tmp;
#设置 group1 的服务器
upstream fdfs_group1 {
server 192.168.1.13:8888 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.1.14:8888 weight=1 max_fails=2 fail_timeout=30s;
}
#设置 group2 的服务器
upstream fdfs_group2 {
server 192.168.1.15:8888 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.1.16:8888 weight=1 max_fails=2 fail_timeout=30s;
}
server {
listen 8000;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
#设置 group 的负载均衡参数
location /group1/M00 {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_cache http-cache;
proxy_cache_valid 200 304 12h;
proxy_cache_key $uri$is_args$args;
proxy_pass http://fdfs_group1;
expires 30d;
}
location /group2/M00 {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_cache http-cache;
proxy_cache_valid 200 304 12h;
proxy_cache_key $uri$is_args$args;
proxy_pass http://fdfs_group2;
expires 30d;
}
#设置清除缓存的访问权限
location ~/purge(/.*) {
allow 127.0.0.1;
allow 192.168.1.0/24;
deny all;
proxy_cache_purge http-cache $1$is_args$args;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
#按以上 nginx 配置文件的要求,创建对应的缓存目录:
[root@tracker1 ~]# mkdir -p /usr/local/fastdfs/cache/nginx/proxy_cache
[root@tracker1 ~]# mkdir -p /usr/local/fastdfs/cache/nginx/proxy_cache/tmp
[root@tracker1 ~]# /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@tracker1 ~]# /usr/local/nginx/sbin/nginx
浏览器访问即可下载
http://192.168.1.11:8000/group1/M00/00/00/wKgBDmOfNBiANMwzAA_BttUvEqM.tar.gz
http://192.168.1.12:8000/group1/M00/00/00/wKgBDmOfNBiANMwzAA_BttUvEqM.tar.gz
7、Keepalived + Nginx 做高可用集群
192.168.100、192.168.1.200
user root;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
## FastDFS Tracker Proxy
upstream fastdfs_tracker {
server 192.168.1.11:8000 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.1.12:8000 weight=1 max_fails=2 fail_timeout=30s;
}
server {
listen 8888;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
## FastDFS Proxy
location /dfs {
root html;
index index.html index.htm;
proxy_pass http://fastdfs_tracker/;
proxy_set_header Host $http_host;
proxy_set_header Cookie $http_cookie;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
client_max_body_size 300m;
}
}
}
keepalived配置
! Configuration File for keepalived
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.10
}
}
! Configuration File for keepalived
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.10
}
}
浏览器访问vip即可下载
http://192.168.1.10:8888/group1/M00/00/00/wKgBDmOfNBiANMwzAA_BttUvEqM.tar.gz