场景:从磁盘某个地方获取文件然后上传到服务器,或者从服务器上传到外部系统
考虑因素:网络因素,主要是网络带宽(实际环境遇到过,因为服务部署在不同省份)
单线程: 按一秒一张每天传递:24*60*60*N(N为机器数); 服务端带宽大的情况下配置为多线程
具体QPS设计,需要结合网络情况
另外考虑到windows电脑做成服务启动应用,具体可以参考github的项目:Releases · winsw/winsw · GitHubA wrapper executable that can run any executable as a Windows service, in a permissive license. - Releases · winsw/winswhttps://github.com/winsw/winsw/releases
win+R进入:services.msc 可查看最终部署的应用
具体操作见最后面
设计思路:队列+多线程
其他方式:redis队列或者mq方式实现,主体思路消费队列
需要注意的是:生产者生产的速度控制,在生产和消费之间找到一个对应的平衡点
适用于一个生产者多个消费者模式,主要是在消费瓶颈上
最终实现思路,文章代码待完善(阉割版),实际项目中已实现设计图内容
临时map的作用是把上传文件后将本地文件删除,避免删除不掉本地文件的情况(此处考虑过一段时间得到最终处理方案)
监控中心作用:定期将未从临时map中清除的文件清理掉,防止意外情况,正确情况下不会存在此情况发生
另外:实际项目中,本地到远程服务再到远程服务,中间搭建了一个中继服务
具体可参考模式:本地项目-----中继服务------远程服务器
maven依赖
<dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.12</version> <scope>provided</scope> </dependency>
工程结构图
1.工具类获取bean
/**
* description: 获取bean
*/
@Component
public class SpringBeanUtils implements ApplicationContextAware {
private static ApplicationContext applicationContext;
@Override
public void setApplicationContext(ApplicationContext context) throws BeansException {
applicationContext = context;
}
/**
* 通过beanName获取bean
* @param beanName
* @return
*/
public static Object getBean(String beanName){
return applicationContext.getBean(beanName);
}
public static ApplicationContext getApplicationContext() {
return applicationContext;
}
}
2.静态数据中心
/**
* description:
*/
@Component
public class MonitorThread {
@Value("${parent_file_path}")
private String parent_file_path;
@Value("${max_thread}")
private int max_thread;
@PostConstruct
public void init() {
MiddleUtils.setParent_file_path(parent_file_path);
MiddleUtils.setMax_thread(max_thread);
new ImageProducer().start();
new ImageMonitorThread().start();
}
}
@Component
public class MiddleUtils {
private static volatile ConcurrentLinkedQueue<String> CONSUMER_FILE = new ConcurrentLinkedQueue<>(); //文件消费队列
private static volatile ConcurrentHashMap<String, Long> FILE_MAP = new ConcurrentHashMap<>(); //计数器
private static String parent_file_path = null;
private static int max_thread = 1;
public static int getMax_thread() {
return max_thread;
}
public static void setMax_thread(int max_thread) {
MiddleUtils.max_thread = max_thread;
}
public static ConcurrentLinkedQueue<String> getConsumerFile() {
return CONSUMER_FILE;
}
public static void setConsumerFile(ConcurrentLinkedQueue<String> consumerFile) {
CONSUMER_FILE = consumerFile;
}
public static ConcurrentHashMap<String, Long> getFileMap() {
return FILE_MAP;
}
public static void setFileMap(ConcurrentHashMap<String, Long> fileMap) {
FILE_MAP = fileMap;
}
public static String getParent_file_path() {
return parent_file_path;
}
public static void setParent_file_path(String parent_file_path) {
MiddleUtils.parent_file_path = parent_file_path;
}
}
3.线程池
@Component
@Slf4j
public class ImageThreadPool {
@Value("${max_thread}")
private int max_thread;
@Bean("imageThread")
public ThreadPoolTaskExecutor initImagePool() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(max_thread);
threadPoolTaskExecutor.setMaxPoolSize(i * 2);
threadPoolTaskExecutor.setQueueCapacity(max_thread * 2);
threadPoolTaskExecutor.setThreadNamePrefix("image_pool");
threadPoolTaskExecutor.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy());
threadPoolTaskExecutor.initialize();
return threadPoolTaskExecutor;
}
}
4.生产者
@Slf4j
public class ImageProducer extends Thread {
private static ImageService imageService;
@Override
public void run() {
while (true) {
/**
* https://www.ip138.com/ascii/ 中文路径转为ASCII码
* 做成windows应用
* https://github.com/winsw/winsw/releases
*/
getImage(MiddleUtils.getParent_file_path());
}
}
public void getImage(String filePath) {
if (Objects.isNull(filePath)) {
log.info("根路径为空");
return;
}
File file = new File(filePath);
File[] files = file.listFiles();
if (files == null || files.length < 1) {
return;
}
for (File f : files) {
if (f.isDirectory()) {
getImage(f.getAbsolutePath());
} else if (f.isFile()) {
/**
* 自旋等待
*/
while (MiddleUtils.getFileMap().size() > MiddleUtils.getConsumerFile().size()) {
try {
TimeUnit.MILLISECONDS.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
/**
* 将数据加入到队列中
*/
Long aLong = MiddleUtils.getFileMap().get(f.getAbsolutePath());
if (aLong != null) {
return;
}
MiddleUtils.getFileMap().put(f.getAbsolutePath(), System.currentTimeMillis());
if (Objects.isNull(imageService)) {
imageService = (ImageService) SpringBeanUtils.getBean("imageService");
}
try {
TimeUnit.SECONDS.sleep(2);
} catch (InterruptedException e) {
e.printStackTrace();
}
imageService.uploadImage(f.getAbsolutePath());
}
}
}
}
5.消费者(可以将图片转成base64字符串,后端做解密操作完成上传服务端)
@Service
@Slf4j
public class ImageService {
@Async("imageThread")
public void uploadImage(String filePath) {
log.info(filePath);
if (Objects.isNull(filePath)) {
return;
}
try {
/**
* 具体业务
*/
log.info("具体业务逻辑");
} catch (Exception e) {
e.printStackTrace();
} finally {
MiddleUtils.getFileMap().remove(filePath);
}
}
}
pom文件依赖
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>image_upload</artifactId>
<version>1.0-SNAPSHOT</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.6</version>
</parent>
<packaging>pom</packaging>
<modules>
<module>image_upload_local</module>
<module>image_upload_server</module>
</modules>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.12</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
local端
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>image_upload</artifactId>
<groupId>org.example</groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>image_upload_local</artifactId>
<packaging>jar</packaging>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<build>
<finalName>image_upload_local</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>com.unique.local.ImageUploadLocal</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>
server端也一样
如果部署windows应用的话可以参考如下做法
1.github上下载对应windows位数的exe文件
Releases · winsw/winsw · GitHub
2.将exe文件名和jar文件名配置一样
原始
修改后
xml文件配置,其中BASE为当前exe文件所在目录
<service>
<!-- ID of the service. It should be unique across the Windows system-->
<id>image_upload_local</id>
<!-- Display name of the service -->
<name>image_upload_local</name>
<!-- Service description -->
<description>image_upload_local upload</description>
<!-- Path to the executable, which should be started -->
<executable>java</executable>
<arguments>-jar %BASE%\image_upload_local.jar --server.port=9002</arguments>
<logpath>%BASE%\logs</logpath>
</service>
安装:image_upload_local.exe install
更新:先停服务 win+r进入服务右键停止服务
卸载程序:image_upload_local.exe uninstall
C:\Users\haoha\Desktop\新建文件夹 (2)\新建文件夹>image_upload_local.exe install
2021-11-13 21:19:20,382 INFO - Installing service 'image_upload_local (image_upload_local)'...
2021-11-13 21:19:20,406 INFO - Service 'image_upload_local (image_upload_local)' was installed successfully.
C:\Users\haoha\Desktop\新建文件夹 (2)\新建文件夹>image_upload_local.exe uninstall
2021-11-13 21:21:37,650 INFO - Uninstalling service 'image_upload_local (image_upload_local)'...
2021-11-13 21:21:37,655 INFO - Service 'image_upload_local (image_upload_local)' was uninstalled successfully.
C:\Users\haoha\Desktop\新建文件夹 (2)\新建文件夹>