Django搭建FastDFS分布式文件存储系统

前言

django admin的ImageField默认会把文件存到settings.py配置的MEDIA_ROOT目录下，所以存储图片的数量受限于django服务器的硬盘大小。为了支持海量的图片存储，更易于拓展存储空间，使用FastDFS分布式存储系统。如果用户上传的文件重复(文件指纹一样)，那么系统只有存储一份数据。

环境：

1. 云服务器下的ubuntu16.04

工具：

1. docker
2. pip
3. pycharm
4. python3.6

docker安装FastFDS

•  获取镜像

sudo docker image pull delron/fastdfs

也可以直接使用镜像备份文件

sudo docker load i 文件路径/fastdfs_docker.tar

• 运行tracker

将fastDFS tracker运行目录映射到本机的 /var/fdfs/tracker目录中

sudo docker run -dti --network=host --name tracker -v /var/fdfs/tracker:/var/fdfs delron/fastdfs tracker

• 运行storage

将fastDFS storage运行目录映射到本机的/var/fdfs/storage目录中

sudo docker run -dti --network=host --name storage -e TRACKER_SERVER=192.168.211.132:22122 -v /var/fdfs/storage:/var/fdfs delron/fastdfs storage

这里要注意的是，TRACKER_SERVER后面跟的ip地址需要填写自己的。我使用的是阿里云的服务器，所以我这里填写自己的公网ip。如果你是自己本地的linux，那就填写自己的ip，可通过ifconfig查看，但要注意的是，这里不要填成本地回环地址127.0.0.1

• 查看容器是否已经启动

sudo docker ps -a

注意：如果无法重新运行，可以删除/var/fdfs/storage/data目录下的fdfs_storaged.pid 文件，然后重新运行storage。

安装FastDFS的Python客户端

为了可以让django可以使用FastDFS系统，我们需要一个可以使用python语言调用FastDFS系统的库，我们可以直接从github上下载下来，再通过pip安装。

• 使用pip安装

pip install fdfs_client-py-master.zip
pip install mutagen
pip install requests

• 在django下配置fastdfs的配置文件

在项目/utils目录下新建fastdfs目录，新建client.conf配置文件

# connect timeout in seconds
# default value is 30s
connect_timeout=30

# network timeout in seconds
# default value is 30s
network_timeout=60

# the base path to store log files
#base_path=FastDFS客户端存放日志文件的目录

# tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
tracker_server=172.17.0.1:22122

#standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level=info

# if use connection pool
# default value is false
# since V4.05
use_connection_pool = false

# connections whose the idle time exceeds this time will be closed
# unit: second
# default value is 3600
# since V4.05
connection_pool_max_idle_time = 3600

# if load FastDFS parameters from tracker server
# since V4.05
# default value is false
load_fdfs_parameters_from_tracker=false

# if use storage ID instead of IP address
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# default value is false
# since V4.05
use_storage_id = false

# specify storage ids filename, can use relative or absolute path
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# since V4.05
storage_ids_filename = storage_ids.conf

#HTTP settings
http.tracker_server_port=80

#use "#include" directive to include HTTP other settiongs
##include http.conf

tracker_server后面的ip设置成自己的。

端口转发的设置

需要进行的端口有三个，分别是tracker_serverd的端口22122、storage_serverd的端口23000、以及最后生成图片地址的端口8888.

测试

进入django项目的shell，进行下面的测试。下面表示测试成功。

(wl_dj) bd@pyvip:~/project_wl$ python manage.py shell
Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from fdfs_client.client import Fdfs_client
>>> FDFS_Client = Fdfs_client('utils/fastdfs/client.conf')
>>> ret = FDFS_Client.upload_by_filename('media/2018.png')
getting connection
<fdfs_client.connection.Connection object at 0x7f2e7c5c8b38>
<fdfs_client.fdfs_protol.Tracker_header object at 0x7f2e7c5c8b00>
>>> ret
{'Status': 'Upload successed.', 'Remote file_id': 'group1/M00/00/00/rBMb_lzeUjSATnH1AAfh_rrm7jw829.png', 'Group name': 'group1', 'Uploaded size': '504.00KB', 'Local file name': 'media/2018.png', 'Storage IP': '106.14.148.168'}

django项目中的配置

• settings配置

# fastdfs服务的站点
FASTDFS_SERVER_DOMAIN = "http://106.14.148.168:8888/"

• view配置

from django.http import JsonResponse,Http404
class Image_upload(View):
    def post(self,request):
        """
         url: "/admin/news/images/"
         验证：
            1、图片对象是否为空
            2、请求内容是否是图片格式
            3、图片的后缀是否规范
        :param request:
        :return:
        """
        image_file = request.FILES.get('image_file')
        if not image_file:
            return Http404('图片不存在')
        if image_file.content_type not in ('image/jpeg', 'image/png', 'image/gif'):
            return Http404('不能上传非图片文件')
        try:
            image_ext_name = image_file.name.split('.')[-1]
        except Exception as e:
            image_ext_name = 'jpg'
        try:
            upload_res = FDFS_Client.upload_by_buffer(image_file.read(), file_ext_name=image_ext_name)
        except Exception as e:
            return Http404('图片上传异常')
        if upload_res.get('Status') != 'Upload successed.':
            return Http404('图片上传到服务器失败')
        image_name = upload_res.get('Remote file_id')
        image_url = settings.FASTDFS_SERVER_DOMAIN + image_name
        return JsonResponse({'image_url': image_url,'errmsg':'图片上传成功'})

• 路由配置

  path('news/images/',views.Image_upload.as_view(),name = 'image_updown'),

• 前端js

  let $upload_to_server = $("#upload-news-thumbnail"); //上传图片的input框的id
  $upload_to_server.change(function () {
    let file = this.files[0];   // 获取文件
    let oFormData = new FormData();  // 创建一个 FormData
    oFormData.append("image_file", file); // 把文件添加进去
    // 发送请求
    $.ajax({
      url: "/admin/news/images/",
      method: "POST",
      data: oFormData,
      processData: false,   // 定义文件的传输
      contentType: false,
    })
      .done(function (res) {
        if (res.errno === "0") {
          // 更新标签成功
          message.showSuccess("图片上传成功");
          let sImageUrl = res["image_url"];
          // console.log(thumbnailUrl);
          $thumbnailUrl.val('');
          $thumbnailUrl.val(sImageUrl);
        } else {
          message.showError(res.errmsg)
        }
      })
      .fail(function () {
        message.showError('服务器超时，请重试！');
      });

  });