centos7FastDFS分布式安装部署

1. 环境准备

我们用四台服务器做测试，IP分别为
虚拟IP（VIP）：10.50.10.111
tracker1：10.50.10.3 hostname：cq03
tracker2：10.50.10.4 hostname：cq04
storage1：10.50.10.11 hostname：cq11
storage2：10.50.10.12 hostname：cq12

下面是我们的简单的构架，通过访问虚拟IP10.50.10.111，来实现文件服务器的读写，此方案最大的容灾程度是一台tracker和一台storage同时宕机。

Nginx和keepalived安装部署高可用这里不写了，回头有专门的博客文章

这里列一下我们要用到的软件版本，
fastdfs-6.06.tar.gz
fastdfs-nginx-module-1.22.tar.gz
libfastcommon-1.0.43.tar.gz
nginx-1.18.0.tar.gz

这几个除了NGINX，别的三个软件是互相对应的，更换其他版本的话可能会出问题，下面是我用的这几个软件的百度网盘下载链接：

链接：https://pan.baidu.com/s/10cxHRT8hFckTcnvWr3p8hQ 
提取码：okmu 

我们先创建一个目录专门放软件

mkdir /upload
cd /upload

将这几个软件复制给其他机器,四台机器全都传过去

2. 安装配置

2.1 所有服务器节点操作

安装依赖环境

yum install make cmake gcc gcc-c++ -y

解压并进入libfastcommon-master

tar -zxvf libfastcommon-1.0.43.tar.gz
cd libfastcommon-1.0.43

编译

./make.sh

编译完成后如下显示

cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -c -o fast_timer.o fast_timer.c  
cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -c -o process_ctrl.o process_ctrl.c  
cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -c -o fast_mblock.o fast_mblock.c  
cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -c -o connection_pool.o connection_pool.c  
cc -Wall -D_FILE_OFFSET_BITS=64 -g -DDEBUG_FLAG -c -o fast_mpool.o fast_mpool.c  
ar rcs libfastcommon.a hash.o chain.o shared_func.o ini_file_reader.o logger.o sockopt.o base64.o sched_thread.o http_func.o md5.o pthread_func.o local_ip_func.o avl_tree.o ioevent.o ioevent_loop.o fast_task_queue.o fast_timer.o process_ctrl.o fast_mblock.o connection_pool.o fast_mpool.o
[root@localhost libfastcommon-1.0.43]#

安装

./make.sh install

安装完成后如下显示

[root@localhost libfastcommon-master]# ./make.sh install
mkdir -p /usr/lib64
install -m 755 libfastcommon.so /usr/lib64
mkdir -p /usr/include/fastcommon
install -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h /usr/include/fastcommon
[root@localhost libfastcommon-master]#

返回上级目录

cd ../

解压fastdfs

tar -zxvf fastdfs-6.06.tar.gz

进入解压目录

cd fastdfs-6.06

编译和安装

./make.sh
./make.sh install

完成后显示如下

cp -f ../common/fdfs_define.h ../common/fdfs_global.h ../common/mime_file_parser.h ../common/fdfs_http_shared.h ../tracker/tracker_types.h ../tracker/tracker_proto.h ../tracker/fdfs_shared_func.h ../storage/trunk_mgr/trunk_shared.h tracker_client.h storage_client.h storage_client1.h client_func.h client_global.h fdfs_client.h /usr/include/fastdfs
if [ ! -f /etc/fdfs/client.conf.sample ]; then cp -f ../conf/client.conf /etc/fdfs/client.conf.sample; fi
[root@localhost fastdfs-5.05]# 

添加系统服务

mv /etc/init.d/fdfs_trackerd /etc/init.d/trackerd
mv /etc/init.d/fdfs_storaged /etc/init.d/storaged

修改/etc/init.d/trackerd和storaged

vim /etc/init.d/trackerd

将打开文件中的所有/usr/local/bin替换成/usr/bin，一次性替换的方式为输入冒号： %s+/usr/local/bin+/usr/bin，回车后保存退出。如

vim /etc/init.d/storaged

同样 将打开文件中的所有/usr/local/bin替换成/usr/bin，一次性替换的方式为输入冒号： %s+/usr/local/bin+/usr/bin，回车后保存退出

所有服务器同样的配置到此结束。接下来我们单独配置两台tracker

2.2 在两台tracker上操作（Cq03、Cq04）

复制tracker配置文件

cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf

修改tracker配置文件

vim /etc/fdfs/tracker.conf

10.50.10.3修改如下内容，内容太多只是部分要修改的地方，比对后再做修改
10.50.10.4也是一样，只是把ip成10.50.10.4就可以了

# is this config file disabled
# false for enabled
# true for disabled
disabled = false

# bind an address of this host
# empty for bind all addresses of this host
bind_addr = 10.50.10.3

# the tracker server port
port = 22122


# the base path to store data and log files
#base_path = /home/yuqing/fastdfs
base_path = /home/fdfs/fastdfs/fastdfs_tracker


# if use connection pool
# default value is false
# since V4.05
use_connection_pool = true


# HTTP port on this tracker server
http.server_port = 7788

增加相应路径

mkdir -p /home/fastdfs/tracker

防火墙放开22122、7788端口
有问题请查看另一篇笔记【CentOS7 防火墙（firewall）的操作命令】

firewall-cmd --zone=public --add-port=22122/tcp --permanent
firewall-cmd --zone=public --add-port=7788/tcp --permanent
firewall-cmd --reload

复制client配置文件

cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf

修改

vim /etc/fdfs/client.conf

修改位如下面配置文件

[admin@cq03 ~]$ cat /etc/fdfs/client.conf
# connect timeout in seconds
# default value is 30s
# Note: in the intranet network (LAN), 2 seconds is enough.
connect_timeout = 5

# network timeout in seconds
# default value is 30s
network_timeout = 60

# the base path to store log files
base_path = /home/fdfs/fastdfs/fastdfs_tracker

# tracker_server can ocur more than once for multi tracker servers.
# the value format of tracker_server is "HOST:PORT",
#   the HOST can be hostname or ip address,
#   and the HOST can be dual IPs or hostnames seperated by comma,
#   the dual IPS must be an inner (intranet) IP and an outer (extranet) IP,
#   or two different types of inner (intranet) IPs.
#   for example: 192.168.2.100,122.244.141.46:22122
#   another eg.: 192.168.1.10,172.17.4.21:22122

#tracker_server = 192.168.0.196:22122
#tracker_server = 192.168.0.197:22122
tracker_server = 10.50.10.3:22122
tracker_server = 10.50.10.4:22122
#standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level = info

# if use connection pool
# default value is false
# since V4.05
use_connection_pool = false

# connections whose the idle time exceeds this time will be closed
# unit: second
# default value is 3600
# since V4.05
connection_pool_max_idle_time = 3600

# if load FastDFS parameters from tracker server
# since V4.05
# default value is false
load_fdfs_parameters_from_tracker = false

# if use storage ID instead of IP address
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# default value is false
# since V4.05
use_storage_id = false

# specify storage ids filename, can use relative or absolute path
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# since V4.05
storage_ids_filename = storage_ids.conf


#HTTP settings
http.tracker_server_port = 7788

#use "#include" directive to include HTTP other settiongs
##include http.conf

启动tracker服务

service trackerd start

查看是否存在trackerd进程

ps -ef|grep trackerd

[root@localhost fastdfs-5.05]# ps -ef|grep trackerd
root 3558 1 0 16:17 ? 00:00:00 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root 3576 2038 0 16:17 pts/0 00:00:00 grep trackerd
[root@localhost fastdfs-5.05]# 

添加trackerd服务开机自启动

chkconfig trackerd on

tracker配置暂时结束，接下来配置storage

2.3 在两台storage上操作（Cq11、Cq12）

复制storage配置文件

cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf

修改

vim /etc/fdfs/storage.conf

cq11修改为以下内容
cq12更改ip地址为10.50.10.12

[admin@cq11 ~]$ cat /etc/fdfs/storage.conf
# is this config file disabled
# false for enabled
# true for disabled
disabled = false

# the name of the group this storage server belongs to
#
# comment or remove this item for fetching from tracker server,
# in this case, use_storage_id must set to true in tracker.conf,
# and storage_ids.conf must be configured correctly.
group_name = group1

# bind an address of this host
# empty for bind all addresses of this host
bind_addr = 10.50.10.11

# if bind an address of this host when connect to other servers 
# (this storage server as a client)
# true for binding the address configured by the above parameter: "bind_addr"
# false for binding any address of this host
client_bind = true

# the storage server port
port = 23000

# connect timeout in seconds
# default value is 30
# Note: in the intranet network (LAN), 2 seconds is enough.
connect_timeout = 5

# network timeout in seconds for send and recv
# default value is 30
network_timeout = 60

# the heart beat interval in seconds
# the storage server send heartbeat to tracker server periodically
# default value is 30
heart_beat_interval = 30

# disk usage report interval in seconds
# the storage server send disk usage report to tracker server periodically
# default value is 300
stat_report_interval = 60

# the base path to store data and log files
# NOTE: the binlog files maybe are large, make sure
#       the base path has enough disk space,
#       eg. the disk free space should > 50GB
#base_path = /home/yuqing/fastdfs
base_path = /home/fdfs/fastdfs/fastdfs_storage
# max concurrent connections the server supported,
# you should set this parameter larger, eg. 10240
# default value is 256
max_connections = 1024

# the buff size to recv / send data from/to network
# this parameter must more than 8KB
# 256KB or 512KB is recommended
# default value is 64KB
# since V2.00
buff_size = 256KB

# accept thread count
# default value is 1 which is recommended
# since V4.07
accept_threads = 1

# work thread count
# work threads to deal network io
# default value is 4
# since V2.00
work_threads = 4

# if disk read / write separated
##  false for mixed read and write
##  true for separated read and write
# default value is true
# since V2.00
disk_rw_separated = true

# disk reader thread count per store path
# for mixed read / write, this parameter can be 0
# default value is 1
# since V2.00
disk_reader_threads = 1

# disk writer thread count per store path
# for mixed read / write, this parameter can be 0
# default value is 1
# since V2.00
disk_writer_threads = 1

# when no entry to sync, try read binlog again after X milliseconds
# must > 0, default value is 200ms
sync_wait_msec = 50

# after sync a file, usleep milliseconds
# 0 for sync successively (never call usleep)
sync_interval = 0

# storage sync start time of a day, time format: Hour:Minute
# Hour from 0 to 23, Minute from 0 to 59
sync_start_time = 00:00

# storage sync end time of a day, time format: Hour:Minute
# Hour from 0 to 23, Minute from 0 to 59
sync_end_time = 23:59

# write to the mark file after sync N files
# default value is 500
write_mark_file_freq = 500

# disk recovery thread count
# default value is 1
# since V6.04
disk_recovery_threads = 3

# store path (disk or mount point) count, default value is 1
store_path_count = 1

# store_path#, based on 0, to configure the store paths to store files
# if store_path0 not exists, it's value is base_path (NOT recommended)
# the paths must be exist.
#
# IMPORTANT NOTE:
#       the store paths' order is very important, don't mess up!!!
#       the base_path should be independent (different) of the store paths

#store_path0 = /home/yuqing/fastdfs
store_path0 = /home/fdfs/fastdfs/fastdfs_storage_data
#store_path1 = /home/yuqing/fastdfs2

# subdir_count  * subdir_count directories will be auto created under each 
# store_path (disk), value can be 1 to 256, default value is 256
subdir_count_per_path = 256

# tracker_server can ocur more than once for multi tracker servers.
# the value format of tracker_server is "HOST:PORT",
#   the HOST can be hostname or ip address,
#   and the HOST can be dual IPs or hostnames seperated by comma,
#   the dual IPS must be an inner (intranet) IP and an outer (extranet) IP,
#   or two different types of inner (intranet) IPs.
#   for example: 192.168.2.100,122.244.141.46:22122
#   another eg.: 192.168.1.10,172.17.4.21:22122

#tracker_server = 192.168.209.121:22122
#tracker_server = 192.168.209.122:22122
tracker_server = 10.50.10.3:22122
tracker_server = 10.50.10.4:22122

#standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level = info

#unix group name to run this program, 
#not set (empty) means run by the group of current user
run_by_group =

#unix username to run this program,
#not set (empty) means run by current user
run_by_user =

# allow_hosts can ocur more than once, host can be hostname or ip address,
# "*" (only one asterisk) means match all ip addresses
# we can use CIDR ips like 192.168.5.64/26
# and also use range like these: 10.0.1.[0-254] and host[01-08,20-25].domain.com
# for example:
# allow_hosts=10.0.1.[1-15,20]
# allow_hosts=host[01-08,20-25].domain.com
# allow_hosts=192.168.5.64/26
allow_hosts = *

# the mode of the files distributed to the data path
# 0: round robin(default)
# 1: random, distributted by hash code
file_distribute_path_mode = 0

# valid when file_distribute_to_path is set to 0 (round robin).
# when the written file count reaches this number, then rotate to next path.
# rotate to the first path (00/00) after the last path (such as FF/FF).
# default value is 100
file_distribute_rotate_count = 100

# call fsync to disk when write big file
# 0: never call fsync
# other: call fsync when written bytes >= this bytes
# default value is 0 (never call fsync)
fsync_after_written_bytes = 0

# sync log buff to disk every interval seconds
# must > 0, default value is 10 seconds
sync_log_buff_interval = 1

# sync binlog buff / cache to disk every interval seconds
# default value is 60 seconds
sync_binlog_buff_interval = 1

# sync storage stat info to disk every interval seconds
# default value is 300 seconds
sync_stat_file_interval = 300

# thread stack size, should >= 512KB
# default value is 512KB
thread_stack_size = 512KB

# the priority as a source server for uploading file.
# the lower this value, the higher its uploading priority.
# default value is 10
upload_priority = 10

# the NIC alias prefix, such as eth in Linux, you can see it by ifconfig -a
# multi aliases split by comma. empty value means auto set by OS type
# default values is empty
if_alias_prefix =

# if check file duplicate, when set to true, use FastDHT to store file indexes
# 1 or yes: need check
# 0 or no: do not check
# default value is 0
check_file_duplicate = 0

# file signature method for check file duplicate
## hash: four 32 bits hash code
## md5: MD5 signature
# default value is hash
# since V4.01
file_signature_method = hash

# namespace for storing file indexes (key-value pairs)
# this item must be set when check_file_duplicate is true / on
key_namespace = FastDFS

# set keep_alive to 1 to enable persistent connection with FastDHT servers
# default value is 0 (short connection)
keep_alive = 0

# you can use "#include filename" (not include double quotes) directive to 
# load FastDHT server list, when the filename is a relative path such as 
# pure filename, the base path is the base path of current/this config file.
# must set FastDHT server list when check_file_duplicate is true / on
# please see INSTALL of FastDHT for detail
##include /home/yuqing/fastdht/conf/fdht_servers.conf

# if log to access log
# default value is false
# since V4.00
use_access_log = false

# if rotate the access log every day
# default value is false
# since V4.00
rotate_access_log = false

# rotate access log time base, time format: Hour:Minute
# Hour from 0 to 23, Minute from 0 to 59
# default value is 00:00
# since V4.00
access_log_rotate_time = 00:00

# if compress the old access log by gzip
# default value is false
# since V6.04
compress_old_access_log = false

# compress the access log days before
# default value is 1
# since V6.04
compress_access_log_days_before = 7

# if rotate the error log every day
# default value is false
# since V4.02
rotate_error_log = false

# rotate error log time base, time format: Hour:Minute
# Hour from 0 to 23, Minute from 0 to 59
# default value is 00:00
# since V4.02
error_log_rotate_time = 00:00

# if compress the old error log by gzip
# default value is false
# since V6.04
compress_old_error_log = false

# compress the error log days before
# default value is 1
# since V6.04
compress_error_log_days_before = 7

# rotate access log when the log file exceeds this size
# 0 means never rotates log file by log file size
# default value is 0
# since V4.02
rotate_access_log_size = 0

# rotate error log when the log file exceeds this size
# 0 means never rotates log file by log file size
# default value is 0
# since V4.02
rotate_error_log_size = 0

# keep days of the log files
# 0 means do not delete old log files
# default value is 0
log_file_keep_days = 0

# if skip the invalid record when sync file
# default value is false
# since V4.02
file_sync_skip_invalid_record = false

# if use connection pool
# default value is false
# since V4.05
use_connection_pool = true

# connections whose the idle time exceeds this time will be closed
# unit: second
# default value is 3600
# since V4.05
connection_pool_max_idle_time = 3600

# if compress the binlog files by gzip
# default value is false
# since V6.01
compress_binlog = true

# try to compress binlog time, time format: Hour:Minute
# Hour from 0 to 23, Minute from 0 to 59
# default value is 01:30
# since V6.01
compress_binlog_time = 01:30

# if check the mark of store path to prevent confusion
# recommend to set this parameter to true
# if two storage servers (instances) MUST use a same store path for
# some specific purposes, you should set this parameter to false
# default value is true
# since V6.03
check_store_path_mark = true

# use the ip address of this storage server if domain_name is empty,
# else this domain name will ocur in the url redirected by the tracker server
http.domain_name =

# the port of the web server on this storage server
http.server_port = 7788

修改/etc/fdfs/storage_ids.conf分组配置文件

vim /etc/fdfs/storage_ids.conf

修改为以下内容

[admin@cq11 ~]$ cat /etc/fdfs/storage_ids.conf
# <id>  <group_name>  <ip_or_hostname[:port]>
#
# id is a natural number (1, 2, 3 etc.),
# 6 bits of the id length is enough, such as 100001
#
# storage ip or hostname can be dual IPs seperated by comma,
# one is an inner (intranet) IP and another is an outer (extranet) IP,
# or two different types of inner (intranet) IPs
# for example: 192.168.2.100,122.244.141.46
# another eg.: 192.168.1.10,172.17.4.21
#
# the port is optional. if you run more than one storaged instances
# in a server, you must specified the port to distinguish different instances.

100001   group1  10.50.10.11
100002   group1  10.50.10.12

添加目录

mkdir -p /home/fdfs/fastdfs/fastdfs_storage
mkdir -p /home/fdfs/fastdfs/fastdfs_storage_data

防火墙放开23000和7788端口

firewall-cmd --zone=public --add-port=22122/tcp --permanent
firewall-cmd --zone=public --add-port=7788/tcp --permanent
firewall-cmd --reload

启动storage

service storaged start

查看storaged进程

ps -ef|grep storaged

添加storaged服务开机自启动

chkconfig storaged on

storage配置完成，接下来在所有tracker服务器上重启tracker,

service trackerd restart

如果执行完命令shell串口没退出，直接敲回车就可以。

在所有storage服务器上重启storage

service storaged restart

如果执行完命令shell串口没退出，直接敲回车就可以。

查看一下storage的日志

[2020-09-06 16:10:22] INFO - file: storage_func.c, line: 273, tracker_client_ip: 10.50.10.11, my_server_id_str: 10.50.10.11, g_server_id_in_filename: 185217546
[2020-09-06 16:10:22] INFO - file: tracker_client_thread.c, line: 299, successfully connect to tracker server 10.50.10.4:22122, as a tracker client, my ip is 10.50.10.1
1[2020-09-06 16:10:22] INFO - file: tracker_client_thread.c, line: 2044, tracker server: #0. 10.50.10.3:22122, my_report_result: -1
[2020-09-06 16:10:22] INFO - file: tracker_client_thread.c, line: 1411, tracker server 10.50.10.4:22122, set tracker leader: 10.50.10.4:22122
[2020-09-06 16:10:22] INFO - file: storage_sync_func.c, line: 114, successfully connect to storage server 10.50.10.12:23000
[2020-09-06 16:10:23] INFO - file: tracker_client_thread.c, line: 299, successfully connect to tracker server 10.50.10.3:22122, as a tracker client, my ip is 10.50.10.1
1

从日志可以第三四行可以看出，这台storage成功连接了两台tracker10.50.10.3和10.50.10.4从第五行可以看出，在两个tracker的选举中10.50.10.4被选举为leader.从第六行可以看出，这台storage已经成功连接上了另外一台storage10.50.10.3。

2.4 测试上传文件

在所有tracker上操作
我们在两台tracker的/home目录下放一张ht.jpg的图片，我们测试将这张图片上传到storage上面，通过如下命令上传

##先随便上传一个图片文件放到/home/下，我的是ht.jpg
/usr/bin/fdfs_upload_file /etc/fdfs/client.conf /home/ht.jpg

我们查看两个tracker的返回

tracker1 10.50.10.3

[root@localhost home]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /home/ht.jpg
group1/M00/00/00/wKgB1lnbObeAOljuAATBl8xofxY969.jpg

tracker2 10.50.10.4

[root@localhost home]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /home/ht.jpg
group1/M00/00/00/wKgB1VnbObeAek4HAATBl8xofxY816.jpg

从返回可以看出两台tracker各自执行了一次上传操作，并得到了storage返回的存储路径+文件名

group1/M00/00/00/wKgB1lnbObeAOljuAATBl8xofxY969.jpg
group1/M00/00/00/wKgB1VnbObeAek4HAATBl8xofxY816.jpg

现在我们去storage上查看是否有这两个图片
在所有storage上操作

ll /home/fastdfs/storage/data/00/00

storage1

[root@localhost 00]# ll /home/fdfs/fastdfs/fastdfs_storage_data/data/00/00
total 616
-rw-r--r-- 1 root root 311703 Oct 9 16:56 wKgB1lnbObeAOljuAATBl8xofxY969.jpg
-rw-r--r-- 1 root root 311703 Oct 9 16:56 wKgB1VnbObeAek4HAATBl8xofxY816.jpg

storaged2

[root@localhost logs]# ll /home/fdfs/fastdfs/fastdfs_storage_data/data/00/00
total 616
-rw-r--r-- 1 root root 311703 Oct 9 16:56 wKgB1lnbObeAOljuAATBl8xofxY969.jpg
-rw-r--r-- 1 root root 311703 Oct 9 16:56 wKgB1VnbObeAek4HAATBl8xofxY816.jpg

从这个可以看出，两台tracker上传的文件，分别存到了两台storage上，并且两台storage会自动实现文件同步。

3. 配置Ngnix

到目前为止，我们还是不能够使用http方式来上传或下载文件的，因此我们需要配置下nginx来达到这个目的。达到这个目的。

cd /upload
ll ##查看目录下文件存不存在

3.1 两台storage都要操作

解压fastdfs-nginx-module-1.22.tar.gz到/usr/local/fast/目录下

tar -zxvf fastdfs-nginx-module-1.22.tar.gz -C /usr/local/fast/

进入目录修改配置文件

cd /usr/local/fast/fastdfs-nginx-module-1.22/src/

修改config为以下内容

[admin@cq12 src]$ cat config 
ngx_addon_name=ngx_http_fastdfs_module

if test -n "${ngx_module_link}"; then
    ngx_module_type=HTTP
    ngx_module_name=$ngx_addon_name
    ngx_module_incs="/usr/include"
    ngx_module_libs="-lfastcommon -lfdfsclient"
    ngx_module_srcs="$ngx_addon_dir/ngx_http_fastdfs_module.c"
    ngx_module_deps=
    CFLAGS="$CFLAGS -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='\"/etc/fdfs/mod_fastdfs.conf\"'"
    . auto/module
else
    HTTP_MODULES="$HTTP_MODULES ngx_http_fastdfs_module"
    NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_fastdfs_module.c"
    CORE_INCS="$CORE_INCS /usr/include"
    CORE_LIBS="$CORE_LIBS -lfastcommon -lfdfsclient"
    CFLAGS="$CFLAGS -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='\"/etc/fdfs/mod_fastdfs.conf\"'"
fi

安装nginx依赖包
把下面四条命令执行一遍。

yum install pcre
yum install pcre-devel
yum install zlib
yum install zlib-devel

安装nginx
进入到/upload目录下

解压nginx文件到指定目录

tar -zxvf nginx-1.18.0.tar.gz -C /usr/local/
cd /usr/local/nginx-1.18.0/
./configure --add-module=/usr/local/fast/fastdfs-nginx-module-1.22/src/ ##加入下模块并进行检查

检查信息如下图所示

检查完之后，输入命令make && make install，回车，两台设备都要执行编译安装操作。

安装信息中没有出现错误信息，说明我们的安装成功，安装完之后，我们到/usr/local/目录下，可以看到多了一个nginx的文件夹，如下图所示。

下面我们到/usr/local/fast/fastdfs-nginx-module-1.22/src/目录下

cd /usr/local/fast/fastdfs-nginx-module-1.22/src/

我们将mod_fastdfs.conf复制到/etc/fdfs/目录下

cp mod_fastdfs.conf /etc/fdfs

复制完之后，我们修改/etc/fdfs/目录下的mod_fastdfs.conf，我们使用命令

vim /etc/fdfs/mod_fastdfs.conf

修改/etc/fdfs/mod_fastdfs.conf配置文件，两台storage都要操作修改
修改为以下内容

[admin@cq11 ~]$ cat /etc/fdfs/mod_fastdfs.conf 
# connect timeout in seconds
# default value is 30s
connect_timeout=10

# network recv and send timeout in seconds
# default value is 30s
network_timeout=30

# the base path to store log files
base_path=/home/fdfs/fastdfs/fastdfs_storage/logs

# if load FastDFS parameters from tracker server
# since V1.12
# default value is false
load_fdfs_parameters_from_tracker=true

# storage sync file max delay seconds
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# since V1.12
# default value is 86400 seconds (one day)
storage_sync_file_max_delay = 86400

# if use storage ID instead of IP address
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# default value is false
# since V1.13
use_storage_id = false

# specify storage ids filename, can use relative or absolute path
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# since V1.13
storage_ids_filename = storage_ids.conf

# FastDFS tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
# valid only when load_fdfs_parameters_from_tracker is true
#tracker_server=tracker:22122
tracker_server=10.50.10.3:22122
tracker_server=10.50.10.4:22122
# the port of the local storage server
# the default value is 23000
storage_server_port=23000

# the group name of the local storage server
group_name=group1

# if the url / uri including the group name
# set to false when uri like /M00/00/00/xxx
# set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx
# default value is false
url_have_group_name = true

# path(disk or mount point) count, default value is 1
# must same as storage.conf
store_path_count=1

# store_path#, based 0, if store_path0 not exists, it's value is base_path
# the paths must be exist
# must same as storage.conf
#store_path0=/home/yuqing/fastdfs
store_path0=/home/fdfs/fastdfs/fastdfs_storage_data
#store_path1=/home/yuqing/fastdfs1

# standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level=info

# set the log filename, such as /usr/local/apache2/logs/mod_fastdfs.log
# empty for output to stderr (apache and nginx error_log file)
log_filename=

# response mode when the file not exist in the local file system
## proxy: get the content from other storage server, then send to client
## redirect: redirect to the original storage server (HTTP Header is Location)
response_mode=proxy

# the NIC alias prefix, such as eth in Linux, you can see it by ifconfig -a
# multi aliases split by comma. empty value means auto set by OS type
# this paramter used to get all ip address of the local host
# default values is empty
if_alias_prefix=

# use "#include" directive to include HTTP config file
# NOTE: #include is an include directive, do NOT remove the # before include
#include http.conf


# if support flv
# default value is false
# since v1.15
flv_support = true

# flv file extension name
# default value is flv
# since v1.15
flv_extension = flv


# set the group count
# set to none zero to support multi-group on this storage server
# set to 0  for single group only
# groups settings section as [group1], [group2], ..., [groupN]
# default value is 0
# since v1.14
group_count = 1

# group settings for group #1
# since v1.14
# when support multi-group on this storage server, uncomment following section
#[group1]
#group_name=group1
#storage_server_port=23000
#store_path_count=2
#store_path0=/home/yuqing/fastdfs
#store_path1=/home/yuqing/fastdfs1
[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/home/fdfs/fastdfs/fastdfs_storage_data

# group settings for group #2
# since v1.14
# when support multi-group, uncomment following section as neccessary
#[group2]
#group_name=group2
#storage_server_port=23000
#store_path_count=1
#store_path0=/home/yuqing/fastdfs

接下来，我们需要把/upload/fastdfs-6.06/conf/目录下的http.conf和mime.types两个文件复制到/etc/fdfs/目录下

cd /upload/fastdfs-6.06/conf/
cp http.conf mime.types /etc/fdfs/

复制完之后，我们给两个节点都创建一下软链接
创建软链接命令：

ln -s /home/fdfs/fastdfs/fastdfs_storage_data/data/ /home/fdfs/fastdfs/fastdfs_storage_data/M00

创建完软链接之后，我们来配置下nginx
进入到/usr/local/nginx/conf/这个目录下，并使用命令vim nginx.conf来编辑该文件，我们需要修改的地方有listen端口，把它由80改成7788，至于原因，是因为我们在上篇博客搭建storage的时候使用的端口是7788，因此这里也需要使用7788。另一个修改的地方是location，修改代码如下，路径采用正则表达式来匹配，匹配group0到group9下的M00前缀。由于fastdfs与nginx的模块结合，因此需要在location当中添加该模块。

修改位置如下，两台storage都要修改，注意两台节点ip地址不一样

server {
        listen       7788;
        server_name  10.50.10.11;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location ~/group([0-9])/M00 {
            root /home/fdfs/fastdfs/fastdfs_storage_data;
            ngx_fastdfs_module;
        }

启动nginx，两台storage都要启动

/usr/local/nginx/sbin/nginx

检查一遍防火墙是否放开23000和7788端口

##检查防火墙是否放开23000和7788端口
查看所有打开的端口： firewall-cmd --zone=public --list-ports
没有放开就用下面命令放开
firewall-cmd --zone=public --add-port=22122/tcp --permanent
firewall-cmd --zone=public --add-port=7788/tcp --permanent
firewall-cmd --reload

启动完nginx之后，我们现在便可以通过http的方式访问上传到FastDFS上的文件了，比如我们现在再把那个3.jpg文件上传一次并拿回显地址去访问，如下所示（注意：上传是在10.50.10.3上进行的）。

[root@cq03 admin]# /usr/bin/fdfs_upload_file  /etc/fdfs/client.conf 123.jpg 
group1/M00/00/04/CjIKDF-BUP2AYXG1AAA-7zaJVQw438.jpg

我们在地址栏输入
http://10.50.10.111:7788/group1/M00/00/04/CjIKDF-BUP2AYXG1AAA-7zaJVQw438.jpg

至此，fastdfs安装部署完毕