分布式图片服务器FastDFS搭建，以及在Django项目中的使用

FastDFS简介：

FastDFS 架构包括 Tracker server 和 Storage server。客户端请求 Tracker server 进行文 件上传、下载，通过 Tracker server 调度最终由 Storage server 完成文件上传和下载。

• Tracker server 作用是负载均衡和调度，通过 Tracker server 在文件上传时可以根据一些 策略找到 Storage server 提供文件上传服务。可以将 tracker 称为追踪服务器或 调度服务器。
• Storage server 作用是文件存储，客户端上传的文件最终存储在 Storage 服务器上， Storageserver 没有实现自己的文件系统而是利用操作系统 的文件系统来管理文件。可以将 storage 称为 存储服务器。

服务端两个角色:

• Tracker:管理集群，tracker 也可以实现集群。每个 tracker 节点地位平等。收集 Storage 集群的状态。

• Storage:实际保存文件 Storage 分为多个组，每个组之间保存的文件是不同的。每 个组内部可以有多个成员，组成员内部保存的内容是一样的，组成员的地位是一致的，没有 主从的概念。

FastDFS安装

1. 安装fastdfs依赖包

1. 解压缩libfastcommon-master.zip

unzip libfastcommon-master.zip

2. 进入到libfastcommon-master的目录中

cd libfastcommon-master

3. 执行 ./make.sh

 ./make.sh

4. 执行 sudo ./make.sh install

sudo ./make.sh install

2. 安装fastdfs

1. 解压缩fastdfs-master.zip

unzip fastdfs-master.zip

2. 进入到 fastdfs-master目录中

cd fastdfs-master

3. 执行 ./make.sh

./make.sh

4. 执行 sudo ./make.sh install

sudo ./make.sh install

3. 配置跟踪服务器tracker

1. sudo cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf

sudo cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf

2. 在/home/python/目录中创建目录 fastdfs/tracker

mkdir -p /home/python/fastdfs/tracker

3. 编辑/etc/fdfs/tracker.conf配置文件

sudo vim /etc/fdfs/tracker.conf

base_path=/home/python/fastdfs/tracker   # 你自建的目录

4. 配置存储服务器storage

1. sudo cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf

sudo cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf

2. 在/home/python/fastdfs/ 目录中创建目录 storage

mkdir –p /home/python/fastdfs/storage

3. 编辑/etc/fdfs/storage.conf配置文件

sudo vim /etc/fdfs/storage.conf

base_path=/home/python/fastdfs/storage
store_path0=/home/python/fastdfs/storage
tracker_server=自己ubuntu虚拟机的ip地址:22122

5. 启动tracker 和 storage

/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start

6. 测试是否安装成功

1. 编辑配置文件

sudo cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf

sudo vim /etc/fdfs/client.conf

base_path=/home/python/fastdfs/tracker
tracker_server=自己ubuntu虚拟机的ip地址:22122

3. 上传文件测试：

fdfs_upload_file /etc/fdfs/client.conf 要上传的图片文件路径

如果返回类似group1/M00/00/00/wKiZhFxtZi2ADsBcAAC-P_xJgL8512.jpg的文件id则说明文件上传成功

以上5步只完成了图片上传功能：

接下来我们继续实现下载、安装和配置nginx相关的组件，实现图片的下载

1. 安装nginx及fastdfs-nginx-module

1. 解压缩 nginx-1.8.1.tar.gz

unzip nginx-1.8.1.tar.gz

2. 解压缩 fastdfs-nginx-module-master.zip

unzip fastdfs-nginx-module-master.zip

3. 进入nginx-1.8.1目录中

cd nginx-1.8.1

4. 新建文件夹作为nginx安装目录

sudo mkdir -r /usr/local/nginx

5. 安装nginx，执行以下命令

sudo ./configure --prefix=/usr/local/nginx/ --add-module=fastdfs-nginx-module-master（解压后的目录的绝对路径）/src

6. 如果这一步有报错，有些文件显示not found错误，根据提示进行安装以下依赖包

sudo apt-get insatll build-essential libtool
sudo apt-get install g++-5
sudo apt-get install openssl libssl-dev
sudo apt-get install zlib1g-dev
sudo apt-get install openssl libssl-dev

7. 编译

sudo make

• 如果出现以下错误：

make -f objs/Makefile
make[1]: Entering directory '/usr/local/nginx/nginx-1.8.1'
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/nginx.o \
	src/core/nginx.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_log.o \
	src/core/ngx_log.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_palloc.o \
	src/core/ngx_palloc.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_array.o \
	src/core/ngx_array.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_list.o \
	src/core/ngx_list.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_hash.o \
	src/core/ngx_hash.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_buf.o \
	src/core/ngx_buf.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_queue.o \
	src/core/ngx_queue.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_output_chain.o \
	src/core/ngx_output_chain.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_string.o \
	src/core/ngx_string.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_parse.o \
	src/core/ngx_parse.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_inet.o \
	src/core/ngx_inet.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_file.o \
	src/core/ngx_file.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_crc32.o \
	src/core/ngx_crc32.c
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_murmurhash.o \
	src/core/ngx_murmurhash.c
src/core/ngx_murmurhash.c: In function ‘ngx_murmur_hash2’:
src/core/ngx_murmurhash.c:37:11: error: this statement may fall through [-Werror=implicit-fallthrough=]
         h ^= data[2] << 16;
         ~~^~~~~~~~~~~~~~~~
src/core/ngx_murmurhash.c:38:5: note: here
     case 2:
     ^~~~
src/core/ngx_murmurhash.c:39:11: error: this statement may fall through [-Werror=implicit-fallthrough=]
         h ^= data[1] << 8;
         ~~^~~~~~~~~~~~~~~
src/core/ngx_murmurhash.c:40:5: note: here
     case 1:
     ^~~~
cc1: all warnings being treated as errors
objs/Makefile:440: recipe for target 'objs/src/core/ngx_murmurhash.o' failed
make[1]: *** [objs/src/core/ngx_murmurhash.o] Error 1
make[1]: Leaving directory '/usr/local/nginx/nginx-1.8.1'
Makefile:8: recipe for target 'build' failed
make: *** [build] Error 2
fatpuffer@ubuntu:/usr/local/nginx/nginx-1.8.1$ sudo make install
make -f objs/Makefile install
make[1]: Entering directory '/usr/local/nginx/nginx-1.8.1'
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused -Werror -g  -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='"/etc/fdfs/mod_fastdfs.conf"' -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/fastdfs -I /usr/include/fastcommon/ -I objs \
	-o objs/src/core/ngx_murmurhash.o \
	src/core/ngx_murmurhash.c
src/core/ngx_murmurhash.c: In function ‘ngx_murmur_hash2’:
src/core/ngx_murmurhash.c:37:11: error: this statement may fall through [-Werror=implicit-fallthrough=]
         h ^= data[2] << 16;
         ~~^~~~~~~~~~~~~~~~
src/core/ngx_murmurhash.c:38:5: note: here
     case 2:
     ^~~~
src/core/ngx_murmurhash.c:39:11: error: this statement may fall through [-Werror=implicit-fallthrough=]
         h ^= data[1] << 8;
         ~~^~~~~~~~~~~~~~~
src/core/ngx_murmurhash.c:40:5: note: here
     case 1:
     ^~~~
cc1: all warnings being treated as errors
objs/Makefile:440: recipe for target 'objs/src/core/ngx_murmurhash.o' failed
make[1]: *** [objs/src/core/ngx_murmurhash.o] Error 1
make[1]: Leaving directory '/usr/local/nginx/nginx-1.8.1'
Makefile:12: recipe for target 'install' failed
make: *** [install] Error 2

-解决办法如下：

1. 找到你nginx解压缩之后的文件夹（例如以下：）

cd Downloads/nginx-1.8.1/objs
sudo vim Makefile 
找到 -Werror 直接删除 在重新make即可

8. 安装

sudo make install

7. 配置 fastdfs-nginx-module 和 Nginx，复制配置文件到/etc/fdfs目录下

sudo cp fastdfs-nginx-module-master(解压后的目录路径)/src/mod_fastdfs.conf  /etc/fdfs/mod_fastdfs.conf

8. 修改配置文件

sudo vim /etc/fdfs/mod_fastdfs.conf

9. 修改内容

connect_timeout=10
tracker_server=自己ubuntu虚拟机的ip地址:22122
url_have_group_name=true
store_path0=/home/python/fastdfs/storage

10. 拷贝其他配置文件到/etc/fdfs目录下

sudo cp 解压缩的fastdfs-master目录下的conf路径中的http.conf /etc/fdfs/http.conf
sudo cp 解压缩的fastdfs-master目录下的conf路径中的mime.types /etc/fdfs/mime.types

11. 修改nginx配置

sudo vim /usr/local/nginx/conf/nginx.conf

12. 修改如下

server {
            listen       8888;
            server_name  localhost;
            # 访问地址以group0-9开头，我们将会调用ngx_fastdfs_module处理
            location ~/group[0-9]/ {
                ngx_fastdfs_module;
            }
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
            root   html;
            }
        }

13. 启动nginx

sudo /usr/local/nginx/sbin/nginx

# sudo /usr/local/nginx/sbin/nginx -s stop  停止nginx

14. 使用python客户端上传测试（首先进入你项目的虚拟环境，然后安装客户端依赖包）

workon 虚拟环境名
进入fdfs_client-py-master.zip所在目录
pip install fdfs_client-py-master.zip

15. 测试如下

>>> from fdfs_client.client import Fdfs_client
>>> client = Fdfs_client('/etc/fdfs/client.conf')
>>> ret = client.upload_by_filename('test')
>>> ret
{'Group name':'group1','Status':'Upload successed.', 'Remote file_id':'group1/M00/00/00/
	wKjzh0_xaR63RExnAAAaDqbNk5E1398.py','Uploaded size':'6.0KB','Local file name':'test'
	, 'Storage IP':'192.168.243.133'}

获取上述压缩包请前往以下路径：
点击获取资源

FastDFS在Django中的使用

第一步：修改Django默认上传方式，在settings.py文件中做以下配置

# 设置Django的文件存储类
DEFAULT_FILE_STORAGE = 'utils.fdfs.storage.FDFSStorage'

# 设置fdfs使用的client.conf文件路径
FDFS_CLIENT_CONF = os.path.join(BASE_DIR, 'utils/fdfs/client.conf')

FDFS_URL = "http://192.168.153.132:8888/"

说明：

• utils是我项目中用来保存一些公用的方法的包（比如celery，登录验证模块，邮件发送模块等 ），与项目应用平级别
• fdfs是我项目中用来存放文件上传类（Fast dfs）的包
• storage是文件上传类脚本文件
• FDFSStorage是我们用来上传文件所使用的类对象

第二步：创建fdfs包（Python Package），将client.conf文件拷贝进来，并创建storage.py文件，说明：文件在上述链接中可以获得，需要修改以下两个地方：

# 需要修改成你想要保存文件的路径
base_path=/home/fatpuffer/Desktop

# 修改为你搭建文件存储服务器的ip地址
tracker_server=192.168.153.132:22122

第三步：在storage.py文件中写入以下内容

from django.core.files.storage import Storage
from fdfs_client.client import Fdfs_client
from django.conf import settings


class FDFSStorage(Storage):
    """fast dfs文件存储类"""
    def __init__(self, client_conf=None, base_url=None):
        """初始化"""
        if client_conf is None:
            client_conf = settings.FDFS_CLIENT_CONF

        self.client_conf = client_conf

        if base_url is None:
            base_url = settings.FDFS_URL

        self.base_url = base_url

    def _open(self):
        """打开文件时使用"""
        pass

    def _save(self, name, content, max_length=None):
        """保存文件时使用"""
        # name：你选择上传文件的名字
        # content：包含你上传文件内容的file对象

        # 创建 Fdfs_client 对象
        client = Fdfs_client(self.client_conf)

        # 上传文件到 fast dfs 存储系统中
        res = client.upload_by_buffer(content.read())

        """
        dict {
            'Group name'      : group_name,
            'Remote file_id'  : remote_file_id,
            'Status'          : 'Upload successed.',
            'Local file name' : '',
            'Uploaded size'   : upload_size,
            'Storage IP'      : storage_ip
        }
        """

        if res.get('Status') != 'Upload successed.':
            """上传失败"""
            raise Exception("上传文件到fast dfs失败")

        # 获取返回的文件ID
        filename = res.get('Remote file_id')

        return filename

    def exists(self, name):
        """Django判断上传文件名是否可用"""
        # 因为文件存储到Fast dfs系统上了，所以永远不会出错
        return False

    def url(self, name):
        """返回访问文件的url路径"""
        return self.base_url+name

可能出现的错误如下：

（1）缺少mutagen模块

    from fdfs_client.utils import *
  File "/home/fatpuffer/.virtualenvs/dailyfresh/lib/python3.6/site-packages/fdfs_client/utils.py", line 8, in <module>
    from mutagen._compat import StringIO
ModuleNotFoundError: No module named 'mutagen'

解决办法：

pip3 install mutagen

（2）缺少requests模块

  File "/home/fatpuffer/.virtualenvs/dailyfresh/lib/python3.6/site-packages/fdfs_client/utils.py", line 10, in <module>
    from requests.compat import basestring
ModuleNotFoundError: No module named 'requests'

解决办法：

pip3 install requests