分布式文件系统-FastDFS+阿里OSS
目录
一、配置FastDFS环境准备工作
1.1、环境准备
-
Centos7.x两台,分别安装tracker与storage
-
下载安装包:
libfastcommon
:FastDFS分离出的一些公用函数包FastDFS
:FastDFS本体fastdfs-nginx-module
:FastDFS和nginx的关联模块nginx
:发布访问服务
安装包的下载地址:
https://github.com/happyfish100
官方安装过程:https://github.com/happyfish100/fastdfs/wiki
1.2、安装步骤(tracker虚拟机与storage虚拟机都要安装)
1.2.1、安装基础库
yum install -y gcc gcc-c++
yum -y install libevent
1.2.2、安装libfastcommon函数库
# 解压
tar -zxvf libfastcommon-1.0.42.tar.gz
-
进入libfastcommon文件夹,编译并且安装
./make.sh ./make.sh install
安装的目录从控制台可以看见
1.2.3、安装fastdfs主程序文件
# 解压
tar -zxvf fastdfs-6.04.tar.gz
-
进入到fastdfs目录,查看fastdfs安装配置
cd fastdfs-6.0.04/ vim make.sh
TARGET_PREFIX+$DESTDIR/usr TARGET_CONF_PATH=$DESTDIR/etc/fdfs TARGET_INIT_PATH=$/DESTDIR/etc/init.d
-
安装fastdfs
./make.sh ./make.sh install
1.2.4、拷贝配置文件到/etc/fdfs中
-
进入fastdfs/conf目录
cd fastdfs-6.0.04/conf
-
拷贝配置文件
cp * /etc/fdfs
二、配置tracker服务
- tracker和storage都是同一个fastdfs的主程序的两个不同概念,配置不同的配置文件就可以设定为tracker或者storage
2.1、配置tracker
-
/etc/fdfs下都是一些配置文件,配置tracker即可
vim tracker.config
-
修改tracker配置文件,此为tracker的工作目录,保存数据以及日志
base_path=/usr/local/fastdfs/tracker
mkdir /usr/local/fastdfs/tracker -p
2.2、启动tracker服务
/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
-
检查进程
ps -ef|grep tracker
-
停止tracker
/usr/bin/stop.sh /etc/fdfs/tracker.conf
三、配置Storage服务
3.1、修改storage配置文件
# 修改组名(任意)
group_name=zzm
# 修改storage的工作空间
base_path=/usr/local/fastdfs/storage
# 修改storage的存储空间
store_path0=/usr/local/fastdfs/storage
# 修改tracker的地址和端口号,用于心跳
tracker_server=192.168.64.137:22122
# 后续结合nginx的一个对外服务端口号
http.server_port=8888
-
创建目录
mkdir /usr/local/fastdfs/storeage -p
3.2、启动storage
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf
前提:必须首先启动tracker
-
检查进程
ps -ef|grep storage
3.3、测试上传
3.3.1、修改client配置文件
base_path=/usr/local/fastdfs/client
tracker_server=192.168.64.137:22122
3.3.2、创建目录
mkdir /usr/local/fastdfs/client -p
3.3.3、测试文件上传
/usr/bin/fdfs_test /etc/fdfs/client.conf upload logo.jpg
logo.jpg
为上传的图片资源
四、配置nginx fastdfs 实现文件服务器
- fastdfs安装号以后是无法通过http访问的,这个时候就需要借助nginx了,所有需要安装fastddfs的第三方模块到nginx,就能使用了
注:nginx需要和storage在同一个节点
4.1、安装nginx插件
4.1.1、解压nginx的fastdfs解压包
tar -zxvf fastdfs-nginx-module-1.22.tar.gz
4.1.2、复制配置文件到指定位置
cd fastdfs-nginx-module-1.22/src
cp mod_fastdfs.conf /etc/fdfs
4.1.3、修改fastdfs-nginx-module/src/config文件
- 主要是修改路径,把local删除,因为fastdfs安装的时候没有修改路径,原路径是
/usr/include
vim config
ngx_module_incs="/usr/include"
CORE_INCS="$CORE_INCS /usr/include"
4.2、安装nginx
- 安装步骤没有区别,唯一区别是,创建makefile文件命令,命令如下
./configure \
--prefix=/usr/local/nginx \
--pid-path=/var/run/nginx/nginx.pid \
--lock-path=/var/locak/nginx.lock \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/temp/nginx/client \
--http-proxy-temp-path=/var/temp/nginx/proxy \
--http-fastcgi-temp-path=/var/temp/nginx/fastcgi \
--http-uwsgi-temp-path=/var/temp/nginx/uwsgi \
--http-scgi-temp-path=/var/temp/nginx/scgi
--add-module=/home/software/FastDFS/fastdfs-nginx-module-1.22/src
4.3、配置mod_fastdfs.conf
cd /etc/fdfs/
vim mod_fastdfs.conf
-
修改内容如下
base_path=/usr/local/fastdfs/tmp tracker_server=192.168.64.137:22122 group_name=zzm url_have_group_name=true store_path0=/usr/local/fastdfs/tmp
-
创建目录
mkdir /usr/local/fastdfs/tmp -p
4.4、配置nginx的nginx.conf配置文件
-
配置文件内容如下
#user nobody; worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; server { listen 8888; server_name localhost; location /zzm/M00 { ngx_fastdfs_module; } } }
-
启动
cd /usr/local/nginx/sbin ./nginx
4.5、资源访问
-
访问之前测试上传的logo.jpg
cd /usr/local/fastdfs/storage/data/00/00
-
查看上传的图片名称,然后浏览器访问如下地址
http://192.168.64.138:8888/zzm/M00/00/00/wKhAil6MNNuAfIG3AABGpXU6GsA963.jpg
如何访问不了,关闭防火墙,两台服务器防火墙都要关闭
五、FastDFS整合SpringBoot
- 创建springboot工程,不会的网上搜一下
5.1、pom.xml配置文件
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>eat_shop-dev</artifactId>
<groupId>com.zzm</groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>eat_shop-dev-fs</artifactId>
<dependencies>
<dependency>
<groupId>com.zzm</groupId>
<artifactId>eat_shop-dev-service</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.tobato</groupId>
<artifactId>fastdfs-client</artifactId>
<version>1.26.7</version>
</dependency>
<dependency>
<groupId>com.aliyun.oss</groupId>
<artifactId>aliyun-sdk-oss</artifactId>
<version>3.8.0</version>
</dependency>
</dependencies>
</project>
5.2、配置文件
-
application.yml配置文件
server: port: 8086 ######################################################################### # # 配置数据源信息 # ######################################################################### spring: datasource: # 数据源的相关配置 type: com.zaxxer.hikari.HikariDataSource # 数据源类型:HikariCP driver-class-name: com.mysql.cj.jdbc.Driver # mysql驱动 url: jdbc:mysql://127.0.0.1:3306/eat_shop?useSSL=false&serverTimezone=UTC username: root password: 你的密码 hikari: connection-timeout: 30000 # 等待连接池分配连接的最大时长(毫秒),超过这个时长还没可用的连接则发生SQLException,默认:30秒 minimum-idle: 5 # 最小连接 maximum-pool-size: 20 # 最大连接 auto-commit: true # 自动提交 idle-timeout: 600000 # 连接超时的最大时长(毫秒),超时则被释放(retired),默认:10分钟 pool-name: DateSourceHikariCP # 连接池名字 max-lifetime: 1800000 # 连接的生命时长(毫秒),超时而且没被使用则被释放,默认:30分钟 connection-test-query: SELECT 1 redis: database: 1 host: 你的ip port: 6379 password: 你的密码 servlet: multipart: max-file-size: 512000 # 文件上传大小限制为500kb = 500 * 1024 max-request-size: 512000 # 请求大小限制为500kb ######################################################################### # # mybatis 配置 # ######################################################################### mybatis: type-aliases-package: com.zzm.pojo # 所有POJO类所在包路径 mapper-locations: classpath:mapper/*.xml # mapper映射文件 # configuration: # log-impl: org.apache.ibatis.logging.stdout.StdOutImpl ######################################################################### # # mybatis mapper配置 # ######################################################################### mapper: mappers: com.zzm.my.mapper.MyMapper not-empty: false identity: MYSQL #分页插件配置 pagehelper: helper-dialect: mysql support-methods-arguments: true ######################################################################### # # fdfs 配置 # ######################################################################### fdfs: connect-timeout: 5000 # 连接的超时时间 so-timeout: 5000 # 读取的超时时间 tracker-list: 192.168.64.137:22122 # tracker服务所在的ip地址和端口号
-
file.properties配置文件
file.host=http://192.168.64.138:8888/
5.3、解决跨域
-
创建CorsConfig.java文件,代码如下
package com.zzm.fs.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.web.cors.CorsConfiguration; import org.springframework.web.cors.UrlBasedCorsConfigurationSource; import org.springframework.web.filter.CorsFilter; @Configuration public class CorsConfig { public CorsConfig() { } @Bean public CorsFilter corsFilter() { // 1. 添加cors配置信息 CorsConfiguration config = new CorsConfiguration(); config.addAllowedOrigin("http://localhost:8888"); config.addAllowedOrigin("http://localhost:8080"); config.addAllowedOrigin("*"); // 设置是否发送cookie信息 config.setAllowCredentials(true); // 设置允许请求的方式 config.addAllowedMethod("*"); // 设置允许的header config.addAllowedHeader("*"); // 2. 为url添加映射路径 UrlBasedCorsConfigurationSource corsSource = new UrlBasedCorsConfigurationSource(); corsSource.registerCorsConfiguration("/**", config); // 3. 返回重新定义好的corsSource return new CorsFilter(corsSource); } }
5.4、构建service层
-
FdfsService.java代码如下
package com.zzm.fs.service.impl; import com.aliyun.oss.OSS; import com.aliyun.oss.OSSClientBuilder; import com.github.tobato.fastdfs.domain.fdfs.StorePath; import com.github.tobato.fastdfs.service.FastFileStorageClient; import com.zzm.fs.resourse.FileResourse; import com.zzm.fs.service.FdfsService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import org.springframework.web.multipart.MultipartFile; import java.io.IOException; import java.io.InputStream; import java.net.URL; @Service public class FdfsServiceImpl implements FdfsService { @Autowired private FileResourse fileResourse; @Autowired private FastFileStorageClient fastFileStorageClient; @Override public String upload(MultipartFile file, String fileExtName) { try { StorePath storePath = fastFileStorageClient.uploadFile( file.getInputStream(), file.getSize(), fileExtName, null); String fullPath = storePath.getFullPath(); return fullPath; } catch (IOException e) { e.printStackTrace(); } return null; } }
5.5、构建controller层
- FdfsController.java代码如下
package com.zzm.fs.controller;
import com.alibaba.fastjson.JSONObject;
import com.zzm.fs.resourse.FileResourse;
import com.zzm.fs.service.FdfsService;
import com.zzm.pojo.Users;
import com.zzm.pojo.vo.UsersVO;
import com.zzm.service.center.CenterUserService;
import com.zzm.utils.CookieUtils;
import com.zzm.utils.JSONResult;
import com.zzm.utils.JsonUtils;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiParam;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
@RestController
@RequestMapping("/fdfs")
public class FdfsController{
@Autowired
private FdfsService fdfsService;
@Autowired
private FileResourse fileResourse;
@Autowired
private CenterUserService centerUserService;
@ApiOperation(value = "用户头像修改", notes = "用户头像修改", httpMethod = "POST")
@PostMapping("/uploadFace")
public JSONResult uploadFace(
@ApiParam(name= "userId", value = "用户id", required = true)
@RequestParam String userId,
@ApiParam(name = "file", value = "用户头像", required = true)
MultipartFile file,
HttpServletRequest request,
HttpServletResponse response){
String path = "";
// 开始文件上传
if(file != null){
try {
// 获得文件上传的文件名称
String fileName = file.getOriginalFilename();
if (StringUtils.isNotBlank(fileName)){
// 文件重命名 abc-face.png -> ["abc-face", "png"]
String fileNameArr[] = fileName.split("\\.");
// 获取文件的后缀名
String suffix = fileNameArr[fileNameArr.length - 1];
if (!suffix.equalsIgnoreCase("png") &&
!suffix.equalsIgnoreCase("jpg") &&
!suffix.equalsIgnoreCase("jpeg")){
return JSONResult.errorMsg("上传图片格式错误");
}
path = fdfsService.upload(file, suffix);
}
} catch (Exception e) {
e.printStackTrace();
}
}else{
return JSONResult.errorMsg("上传的文件不能为空");
}
if (StringUtils.isNotBlank(path)){
// 由于浏览器可能存在缓存的情况,所以在这里,需要加上时间戳来保证更新后的图片可以及时刷新
String finalUserFaceUrl = fileResourse.getHost() + path;
// 更新用户头像到数据库
Users user = centerUserService.updateUserFace(userId, finalUserFaceUrl);
return JSONResult.ok(user);
}else{
return JSONResult.errorMsg("上传的文件不能为空");
}
}
}
5.6、构建resourse层
-
FileResource.java代码如下
package com.zzm.fs.resourse; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.context.annotation.PropertySource; import org.springframework.stereotype.Component; @Component @PropertySource("classpath:file.properties") @ConfigurationProperties(prefix = "file") @Data public class FileResourse { private String host; }
六、阿里OSS整合SpringBoot
- 去阿里网站申请阿里oss并创建
Buckets
和Access Key
6.1、获取重要参数
- endpoint
- accessKeyId
- accessKeySecret
- bucketName:你创建bucketName的名称
- objectName:你保存文件的路径
- ossHost
6.2、pom.xml配置文件
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>eat_shop-dev</artifactId>
<groupId>com.zzm</groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>eat_shop-dev-fs</artifactId>
<dependencies>
<dependency>
<groupId>com.zzm</groupId>
<artifactId>eat_shop-dev-service</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.aliyun.oss</groupId>
<artifactId>aliyun-sdk-oss</artifactId>
<version>3.8.0</version>
</dependency>
</dependencies>
</project>
6.3、配置文件
-
application.yml配置文件
server: port: 8086 ######################################################################### # # 配置数据源信息 # ######################################################################### spring: datasource: # 数据源的相关配置 type: com.zaxxer.hikari.HikariDataSource # 数据源类型:HikariCP driver-class-name: com.mysql.cj.jdbc.Driver # mysql驱动 url: jdbc:mysql://127.0.0.1:3306/eat_shop?useSSL=false&serverTimezone=UTC username: root password: 你的密码 hikari: connection-timeout: 30000 # 等待连接池分配连接的最大时长(毫秒),超过这个时长还没可用的连接则发生SQLException,默认:30秒 minimum-idle: 5 # 最小连接 maximum-pool-size: 20 # 最大连接 auto-commit: true # 自动提交 idle-timeout: 600000 # 连接超时的最大时长(毫秒),超时则被释放(retired),默认:10分钟 pool-name: DateSourceHikariCP # 连接池名字 max-lifetime: 1800000 # 连接的生命时长(毫秒),超时而且没被使用则被释放,默认:30分钟 connection-test-query: SELECT 1 redis: database: 1 host: 你的ip port: 6379 password: 你的密码 servlet: multipart: max-file-size: 512000 # 文件上传大小限制为500kb = 500 * 1024 max-request-size: 512000 # 请求大小限制为500kb ######################################################################### # # mybatis 配置 # ######################################################################### mybatis: type-aliases-package: com.zzm.pojo # 所有POJO类所在包路径 mapper-locations: classpath:mapper/*.xml # mapper映射文件 # configuration: # log-impl: org.apache.ibatis.logging.stdout.StdOutImpl ######################################################################### # # mybatis mapper配置 # ######################################################################### mapper: mappers: com.zzm.my.mapper.MyMapper not-empty: false identity: MYSQL #分页插件配置 pagehelper: helper-dialect: mysql support-methods-arguments: true
-
file.properties配置文件
file.endpoint=http://oss-cn-beijing.... file.accessKeyId=LTAI4FvqK2f... file.accessKeySecret=AtqZfc3oY9vEIU... file.bucketName=eatshop file.objectName=images file.ossHost=https://eatshop.oss-cn-beijing....
6.4、解决跨域
-
创建CorsConfig.java文件,代码如下
package com.zzm.fs.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.web.cors.CorsConfiguration; import org.springframework.web.cors.UrlBasedCorsConfigurationSource; import org.springframework.web.filter.CorsFilter; @Configuration public class CorsConfig { public CorsConfig() { } @Bean public CorsFilter corsFilter() { // 1. 添加cors配置信息 CorsConfiguration config = new CorsConfiguration(); config.addAllowedOrigin("http://localhost:8888"); config.addAllowedOrigin("http://localhost:8080"); config.addAllowedOrigin("*"); // 设置是否发送cookie信息 config.setAllowCredentials(true); // 设置允许请求的方式 config.addAllowedMethod("*"); // 设置允许的header config.addAllowedHeader("*"); // 2. 为url添加映射路径 UrlBasedCorsConfigurationSource corsSource = new UrlBasedCorsConfigurationSource(); corsSource.registerCorsConfiguration("/**", config); // 3. 返回重新定义好的corsSource return new CorsFilter(corsSource); } }
6.5、构建service层
-
FdfsServiceImpl.java代码如下
package com.zzm.fs.service.impl; import com.aliyun.oss.OSS; import com.aliyun.oss.OSSClientBuilder; import com.github.tobato.fastdfs.domain.fdfs.StorePath; import com.github.tobato.fastdfs.service.FastFileStorageClient; import com.zzm.fs.resourse.FileResourse; import com.zzm.fs.service.FdfsService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import org.springframework.web.multipart.MultipartFile; import java.io.IOException; import java.io.InputStream; import java.net.URL; @Service public class FdfsServiceImpl implements FdfsService { @Autowired private FileResourse fileResourse; @Override public String uploadOSS(MultipartFile file, String userId, String fileExtName) { // Endpoint以杭州为例,其它Region请按实际情况填写。 String endpoint = fileResourse.getEndpoint(); // 阿里云主账号AccessKey拥有所有API的访问权限,风险很高。强烈建议您创建并使用RAM账号进行API访问或日常运维,请登录 https://ram.console.aliyun.com 创建RAM账号。 String accessKeyId = fileResourse.getAccessKeyId(); String accessKeySecret = fileResourse.getAccessKeySecret(); String myObjectName = fileResourse.getObjectName() + "/" + userId + "/" + userId + "." + fileExtName; OSS ossClient = null; try { // 创建OSSClient实例。 ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret); // 上传网络流。 InputStream inputStream = file.getInputStream(); ossClient.putObject(fileResourse.getBucketName(), myObjectName, inputStream); } catch (IOException e) { e.printStackTrace(); } // 关闭OSSClient。 ossClient.shutdown(); return myObjectName; } }
6.6、构建controller层
- FdfsController.java代码如下
package com.zzm.fs.controller;
import com.alibaba.fastjson.JSONObject;
import com.zzm.fs.resourse.FileResourse;
import com.zzm.fs.service.FdfsService;
import com.zzm.pojo.Users;
import com.zzm.pojo.vo.UsersVO;
import com.zzm.service.center.CenterUserService;
import com.zzm.utils.CookieUtils;
import com.zzm.utils.JSONResult;
import com.zzm.utils.JsonUtils;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiParam;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
@RestController
@RequestMapping("/fdfs")
public class FdfsController extends BaseController{
@Autowired
private FdfsService fdfsService;
@Autowired
private FileResourse fileResourse;
@Autowired
private CenterUserService centerUserService;
@ApiOperation(value = "用户头像修改", notes = "用户头像修改", httpMethod = "POST")
@PostMapping("/uploadFace")
public JSONResult uploadFace(
@ApiParam(name= "userId", value = "用户id", required = true)
@RequestParam String userId,
@ApiParam(name = "file", value = "用户头像", required = true)
MultipartFile file,
HttpServletRequest request,
HttpServletResponse response){
String path = "";
// 开始文件上传
if(file != null){
try {
// 获得文件上传的文件名称
String fileName = file.getOriginalFilename();
if (StringUtils.isNotBlank(fileName)){
// 文件重命名 abc-face.png -> ["abc-face", "png"]
String fileNameArr[] = fileName.split("\\.");
// 获取文件的后缀名
String suffix = fileNameArr[fileNameArr.length - 1];
if (!suffix.equalsIgnoreCase("png") &&
!suffix.equalsIgnoreCase("jpg") &&
!suffix.equalsIgnoreCase("jpeg")){
return JSONResult.errorMsg("上传图片格式错误");
}
path = fdfsService.uploadOSS(file, userId, suffix);
}
} catch (Exception e) {
e.printStackTrace();
}
}else{
return JSONResult.errorMsg("上传的文件不能为空");
}
if (StringUtils.isNotBlank(path)){
// 由于浏览器可能存在缓存的情况,所以在这里,需要加上时间戳来保证更新后的图片可以及时刷新
String finalUserFaceUrl = fileResourse.getOssHost() + path;
// 更新用户头像到数据库
Users user = centerUserService.updateUserFace(userId, finalUserFaceUrl);
return JSONResult.ok(user);
}else{
return JSONResult.errorMsg("上传的文件不能为空");
}
}
}
6.7、构建resourse层
-
FileResource.java代码如下
package com.zzm.fs.resourse; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.context.annotation.PropertySource; import org.springframework.stereotype.Component; @Component @PropertySource("classpath:file.properties") @ConfigurationProperties(prefix = "file") @Data public class FileResourse { private String endpoint; private String accessKeyId; private String accessKeySecret; private String bucketName; private String objectName; private String ossHost; }