分布式文件系统FastDFS最佳部署实践

版权声明:本文为博主John Lau原创文章,未经博主允许不得转载 https://blog.csdn.net/GreatElite/article/details/11020989

FastDFS安装

  • FastDFS简介

    FastDFS是一个国产开源的轻量级分布式文件系统,它对文件进行管理,功能包括:文件存储、文件同步、文件访问(文件上传、文件下载)等,解决了大容量存储和负载均衡的问题.特别适合以文件为载体的在线服务,如相册网站、视频网站等等.FastDFS服务端有两个角色:跟踪器(tracker)和存储节点(storage).跟踪器主要做调度工作,在访问上起负载均衡的作用.

    fastdfs_arch
  • 安装部署说明

    tracker : 192.168.1.190 (部署Nginx, Varnish, MySQL, PHP & fastdfs扩展, FastDFS Tracker)
    storage1: 192.168.1.191 (部署Nginx & fastdfs-nginx-module, FastDFS Storage)
    storage2: 192.168.1.191 (部署Nginx & fastdfs-nginx-module, FastDFS Storage)

  • FastDFS安装
    • 服务器所用的Linux系统:

      # cat /etc/issue
      CentOS release 6.3 (Final)
      

      安装FastDFS:

      # tar zxvf FastDFS_v4.06.tar.gz
      # cd FastDFS
      # ./make.sh
      # ./make.sh install
      # cp init.d/* /etc/init.d/

    FastDFS配置与测试

    FastDFS安装后,默认配置文件放在/etc/fdfs目录下.

    • 跟踪器Tracker配置

      修改/etc/fdfs/tracker.conf:

      # mkdir -p /data/fastdfs
      # sed -i 's:base_path=.*:base_path=/data/fastdfs:g' /etc/fdfs/tracker.conf

      调整防火墙的规则,开放端口:22122

      # iptables -I INPUT 5 -p tcp -m state --state NEW -m tcp --dport 22122 -j ACCEPT
      # /etc/init.d/iptables save
      

      启动tracker进程:

      # /etc/init.d/fdfs_trackerd start
      Starting FastDFS tracker server:

      查看trackerd状态:

      # /etc/init.d/fdfs_trackerd status
      fdfs_trackerd (pid 3098) is running...

      查看trackerd监听的端口:

      # netstat -plantu | grep tracker
      tcp        0      0 0.0.0.0:22122               0.0.0.0:*                   LISTEN      3098/fdfs_trackerd
    • 存储节点Storage配置

      修改/etc/fdfs/tracker.conf:

      # mkdir -p /data/fastdfs
      # sed -i 's:base_path=.*:base_path=/data/fastdfs:g' /etc/fdfs/storage.conf
      # sed -i 's:store_path0=.*:store_path0=/data/fastdfs:g' /etc/fdfs/storage.conf
      # sed -i 's/tracker_server=.*/tracker_server=192.168.1.190:22122/g' /etc/fdfs/storage.conf
      

      调整防火墙的规则,开放端口:23000

      # iptables -I INPUT 5 -p tcp -m state --state NEW -m tcp --dport 23000 -j ACCEPT
      # /etc/init.d/iptables save
      

      启动Storage进程:

      # /etc/init.d/fdfs_storaged start
      

      注:我启动时候遇到了错误:error while loading shared libraries: libevent-2.0.so.5…
      解决方法:
      查找libevent-2.0.so.5有出现/usr/local/lib/目录下.因此:

      ln -s /usr/local/lib/libevent-2.0.so.5 /usr/lib64/libevent-2.0.so.5
      

      查看Storage进程:

      # ps -ef | grep storaged | grep -v grep
      root      3032     1  0 23:16 pts/0    00:00:00 /usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf
      

      查看Storage服务器状态:

      # /usr/local/bin/fdfs_monitor /etc/fdfs/storage.conf
      

      删除无效的Storage服务器:

      /usr/local/bin/fdfs_monitor /etc/fdfs/client.conf delete group1 192.168.1.192
      

      注:如果被删除的Storage服务器状态为ACTIVE的话(即仍然在线提供服务),删除无法完成.

    • 客户端Client配置

      修改/etc/fdfs/client.conf:

      # sed -i 's:base_path=.*:base_path=/data/fastdfs:g' /etc/fdfs/client.conf
      # sed -i 's/tracker_server=.*/tracker_server=192.168.1.190:22122/g' /etc/fdfs/client.conf 
      
    • FastDFS测试

      上传测试:

      # fdfs_upload_file /etc/fdfs/client.conf test.jpg 
      group1/M00/00/00/wKgBv1FRyIiAdxkRAAO1XLFK2Tw138.jpg
      

      查看文件信息:

      # fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/wKgBv1FRyIiAdxkRAAO1XLFK2Tw138.jpg
      source storage id: 0
      source ip address: 192.168.1.191
      file create timestamp: 2013-03-27 00:10:48
      file size: 243036
      file crc32: 2974472508 (0xB14AD93C)
      

      下载测试:

      # fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/wKgBv1FRyIiAdxkRAAO1XLFK2Tw138.jpg downtest.jpg
      # ls
      downtest.jpg  test.jpg

    FastDFS客户端

    • PHP扩展安装

      客户端PHP扩展安装:

      # cd /data/install/FastDFS/php_client
      # ./configure --with-php-config=/usr/local/php5410/bin/php-config --with-fastdfs_client
      # make && make install
      

      编辑php.ini,添加如下内容:

      [fastdfs]
      extension = fastdfs_client.so
      
      ; the base path
      fastdfs_client.base_path =/data/fastdfs
      
      ; connect timeout in seconds
      ; default value is 30s
      fastdfs_client.connect_timeout = 2
      
      ; network timeout in seconds
      ; default value is 30s
      fastdfs_client.network_timeout = 60
      
      ; standard log level as syslog, case insensitive, value list:
      ;;; emerg for emergency
      ;;; alert
      ;;; crit for critical
      ;;; error
      ;;; warn for warning
      ;;; notice
      ;;; info
      ;;; debug
      fastdfs_client.log_level = info
      
      ; set the log filename, such as /usr/local/fastdfs/logs/fastdfs_client.log
      ; empty for output to stderr
      fastdfs_client.log_filename =/data/fastdfs/logs/fastdfs_client.log
      
      ; secret key to generate anti-steal token
      ; this parameter must be set when http.anti_steal.check_token set to true
      ; the length of the secret key should not exceed 128 bytes
      fastdfs_client.http.anti_steal_secret_key =
      
    • PHP扩展使用
      <?php
      // fastdfs.php
      class FDFS {
          private $fdfs, $tracker, $storage;
          public function __construct() {
              $this->fdfs = new FastDFS();
      
              // get a connected tracker server
              $this->tracker = $this->fdfs->tracker_get_connection();
              if (!$this->tracker) {
                  throw new Exception('cannot connect to tracker server:[' .  $this->fdfs->get_last_error_no() . '] ' . $this->fdfs->get_last_error_info());
              }
      
              // get the storage server info and connect to it
              $this->storage = $this->fdfs->tracker_query_storage_store();
              $this->server = $this->fdfs->connect_server($this->storage['ip_addr'], $this->storage['port']);
              if ($this->server === false) {
                  throw new Exception('cannot connect to storage server' . $this->storage['ip_addr'] . ':' . $this->storage['port'] .  ' :[' .  $this->fdfs->get_last
              }
              $this->storage['sock'] = $this->server['sock'];
          }
         
          public function upload($localfile) {
              //$info = $this->fdfs->storage_upload_by_filename($localfile);
              $info = $this->fdfs->storage_upload_by_filename($localfile, null, array(), null, $this->tracker, $this->storage);
              if (is_array($info)) {
                  $group_name = $info['group_name'];
                  $remote_filename = $info['filename'];
                  $source_info = $this->fdfs->get_file_info($group_name, $remote_filename);
                  $source_ip = $source_info['source_ip_addr'];
                  $file_size = $source_info['file_size'];
                  return compact('group_name', 'remote_filename', 'source_ip', 'file_size');
              }
              return false;
          }
      
          public function download_to_buff($group_name, $remote_filename) {
              $content = $this->fdfs->storage_download_file_to_buff($group_name, $remote_filename);
              return $content;
          }
      
          public function download_to_file($group_name, $remote_filename, $dst_localfile) {
              return $this->fdfs->storage_download_file_to_file($group_name, $remote_filename, $dst_localfile);
          }
      
          public function delete($group_name, $remote_filename) {
              return $this->fdfs->storage_delete_file($group_name, $remote_filename);
          }
      
          public function exists($group_name, $remote_filename) {
              return $this->fdfs->storage_file_exist($group_name, $remote_filename);
          }
      
          public function get_file_info($group_name, $remote_filename) {
              return $this->fdfs->get_file_info($group_name, $remote_filename);
          }
      }
      

      PHP客户端上传图片测试:

      # cat test.php 
      <?php
      require_once('fastdfs.php');
      
      $fdfs = new FDFS();
      
      $localfile = './test.jpg';
      $fileinfo = $fdfs->upload($localfile);
      
      if ($fileinfo) {
        // update file info in the database etc
      }
      
      var_dump($fileinfo);
      
      # php test.php 
      array(3) {
        ["group_name"]=>
        string(6) "group1"
        ["remote_filename"]=>
        string(44) "M00/00/00/wKgBwFFSZwKANJKgAAO1XLFK2Tw609.jpg"
        ["source_ip"]=>
        string(13) "192.168.1.192"
      }
      

    FastDFS与Nginx集成

    • Storage服务器安装fastdfs的nginx模块

      查看之前部署的Nginx编译参数:

      # /usr/local/nginx/sbin/nginx -V
      nginx version: nginx/1.3.9
      built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) 
      TLS SNI support enabled
      configure arguments: --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --with-pcre --with-http_gzip_static_module --with-openssl=/data/install/openssl-1.0.1c
      

      重新编译Nginx,增加fastdfs模块:

      # wget https://fastdfs.googlecode.com/files/fastdfs-nginx-module_v1.15.tar.gz
      # tar zxvf fastdfs-nginx-module_v1.15.tar.gz
      # cd /data/install/nginx-1.3.9
      # ./configure --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --with-pcre --with-http_gzip_static_module --with-openssl=/data/install/openssl-1.0.1c --add-module=/data/install/fastdfs-nginx-module/src/
      # make
      # mv /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx.bak
      # cp objs/nginx /usr/local/nginx/sbin/
      
    • Nginx及fastdfs模块配置

      fastdfs模块配置:

      # cp /data/install/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
      # touch /data/fastdfs/logs/mod_fastdfs.log
      # chown www:www /data/fastdfs/logs/mod_fastdfs.log
      # vi /etc/fdfs/mod_fastdfs.conf
      

      修改:

      base_path=/data/fastdfs
      tracker_server=192.168.1.190:22122
      store_path0=/data/fastdfs
      log_filename=/data/fastdfs/logs/mod_fastdfs.log
      http.need_find_content_type=true
      

      Storage服务器nginx配置(Server配置内增加如下内容):

      ...
           location /M00 {
                 alias /data/fastdfs/data;
                 ngx_fastdfs_module;
          }
      ...
      

      重启Nginx:

      # /usr/local/nginx/sbin/nginx -s stop
      # /usr/local/nginx/sbin/nginx
      

      通过Nginx访问PHP客户端上传的图片:

      fastdfs_nginx

    前端部署HTTP缓存

    HTTP缓存方案可以选择Squid,Varnish,Nginx Proxy Cache,Apache Traffic Server等.
    以下是两种前端缓存方案供参考:

    • 部署Nginx Proxy Cache
      • 安装Nginx + ngx_cache_purge:
        # wget -c http://nginx.org/download/nginx-1.2.8.tar.gz
        # git clone https://github.com/FRiCKLE/ngx_cache_purge.git
        # tar zxvf nginx-1.2.8.tar.gz
        # cd nginx-1.2.8
        # ./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_realip_module --with-http_gzip_static_module --with-pcre --with-openssl=/data/install/openssl-1.0.1c --add-module=../nginx-http-concat/ --add-module=../ngx_cache_purge/
        # make && make install
        
      • 配置Nginx Proxy Cache:
        [root@centos190 conf]# cat /usr/local/nginx/conf/nginx.conf 
        
        user www www;
        worker_processes 2;
        worker_cpu_affinity 0001 0010;
        pid logs/nginx.pid;
        error_log logs/error.log crit;
        worker_rlimit_nofile 65535;
        events {
            worker_connections 65535;
            use epoll;
        }
        http {
            include       mime.types;
            default_type  application/octet-stream;
            charset  utf-8;
              
            server_names_hash_bucket_size 128;
            client_header_buffer_size 32k;
            large_client_header_buffers 4 32k;
            client_max_body_size 300m;
              
            sendfile on;
            tcp_nopush     on;
            keepalive_timeout 60;
            tcp_nodelay on;
        
            proxy_redirect off;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_connect_timeout 90;
            proxy_send_timeout 90;
            proxy_read_timeout 90;
            proxy_buffer_size 16k;  
            proxy_buffers 4 64k;
            proxy_busy_buffers_size 128k;  
            proxy_temp_file_write_size 128k;
        
            proxy_cache_path /var/cache/nginx/proxy_cache levels=1:2 keys_zone=http-cache:10m max_size=1000m inactive=600m;
            proxy_temp_path /var/cache/nginx/proxy_cache/tmp;
        
            upstream fdfs {
                server   192.168.1.191:80 weight=1 max_fails=2 fail_timeout=30s;
                server   192.168.1.192:80 weight=1 max_fails=2 fail_timeout=30s;
            }
        
            server {
                listen 80;
                server_name a.qingluobo.com;
                
                root /data/www/qingluobo;
                index index.php index.html;
        
                ## Proxy Cache for FDFS images
                location /M00 {
                    proxy_next_upstream http_502 http_504 error timeout invalid_header;
                    proxy_cache http-cache;
                    proxy_cache_valid  200 304 12h;
                    proxy_cache_key $uri$is_args$args;
                    proxy_pass http://fdfs;
                    expires 1d;
                }
                ## Purge Nginx Cache
                location ~ /purge(/.*) {
                    #auth_basic "Restricted Access";
                    #auth_basic_user_file access_nginx_auth;
                    allow 127.0.0.1;
                    allow 192.168.1.0/24;
                    deny all;
                    proxy_cache_purge http-cache  $1$is_args$args;
                }
                location ~ \.php$ {
                    fastcgi_pass   127.0.0.1:9000;
                    fastcgi_index  index.php;
                    fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
                    include fastcgi_params;
                }
                #access_log off;
            }
        }
        
      • 清除Nginx缓存:
        访问指定URL的对应purge地址即可清除其缓存:

        nginx_proxy_cache_purge

阅读更多
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页