三:多个文件下载的管理
这一节我们主要来讲一下如何对多个文件的下载进行管理
首先来看一下整个系统的UML图
从最下面开始说起:
Download代表一个下载类,对每一个文件都需要创建一个Download实例,用于对该文件下载线程的管理。其中每个Download中都有以下几个对象:
private ConcurrentLinkedQueue<DownloadBlock> blockQueue;
private ConcurrentLinkedQueue<DownloaBlock> blockCache;
private ConcurrentHashMap<Long, Long> blockCounts;
private ConcurrentLinkedQueue<DownloadThread> activeThreads;
- blockQueue是一个队列,用于存储当前需要下载的DownloadBlock。Download对文件进行切分形成的DownloadBlock会被放入到放入到blockQueue中,供以后的下载。
- blockCache为block内存缓存池,主要是为了能够复用已经建立好的DownloadBlock。
- blockCounts为一个Map,其中key为每个block的start值,而value为该block已经下载完的段D。主要作用是统计出当前已经每个Block已经下载完的段D,以计算实时下载速度
- activeThreads 主要是为了存储该Thread中所有的活跃线程。
DownloadBlock是一个下载块,其里面有3个成员变量
private Download download; //其所属的Download
private long start; //下载文件起始处
private long length; //下载文件的长度
DownloadThread是指下载进程,每个DownloadBlock都需要启动一个DownloadThread去进行下载。即
new DownloadThread(block).start()
DownloadDeamon为了一个守护线程。其内部主要为了下载所有的需要下载DownloadBlock
private DownloadList downloads; //当前系统中所有的下载列表
private ExecutorService threadPool; //线程池
Downloader 代表整个下载系统,整个系统中只有一个实例对象,因此我们需要保证系统中只有一个实例对象。
private DownloaderConfig config; // Downloader配置
private DownloadList downloads; //当前系统所有的下载列表private Thread deamon; //守护进程
private ConcurrentLinkedQueue<DownloadBlock> blockCache; //当前系统的缓存
private Timer timer; //
看了上面一大堆的东西,我保证你现在很晕,OK,我们从使用的角度来看整个系统是如何运行的。
下面是示例代码。public static void main(String[] args) {
Downloader downloader = Downloader.getInstance();
//下载第一个文件
String url1 = "https://tmsvm.googlecode.com/files/tmsvm_src_v1.1.0.rar";
String saveFile1 = "data/tmsvm_src_v1.1.0.rar";
DownloadConfig config = new DownloadConfig();
try {
config.setUrl(new URL(url1));
config.setFile(new File(saveFile1));
config.setNthread(new Integer(5));
config.setPriority(new Integer(6));
//将第一个下载加入到下载列表中
downloader.addDownload(new Download(config, downloader.getTimer()));
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
//下载第二个文件
String url2 = "https://tmsvm.googlecode.com/files/Tmsvm%E5%8F%82%E8%80%83%E6%96%87%E6%A1%A3%28v1.1.0%29.rar";
String saveFile2 = "data/Tmsvm参考文档(v1.1.0).rar";
try {
config.setUrl(new URL(url2));
config.setFile(new File(saveFile2));
config.setNthread(new Integer(5));
config.setPriority(new Integer(6));
//将第二个下载加入到下载列表中
downloader.addDownload(new Download(config, downloader.getTimer()));
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
1. 系统初始化
Downloader downloader = Downloader.getInstance();
Downloader是这个下载器的总调度师,一山不容二虎,当然在系统运行过程中,只能有一个Downloader的实例,因此我们需要用单例模式来保证这一点。首先要取得downloader实例,即系统的初始化。我们看系统初始化需要做什么?
public static Downloader getInstance(){
if(downloader == null)
downloader = new Downloader(new DownloaderConfig());
return downloader;
}
private Downloader(DownloaderConfig config) {
super();
this.config = config;
start();
}
上面的代码中的start()函数中到底做了什么呢?
- 初始化blockCache缓存,其中blockCache为ConcurrentLinkedQueue<DownloadBlock>类型。
- 启动守护进程DownloadDeamon
具体代码如下:
private void start(){
blockCache = new ConcurrentLinkedQueue<DownloadBlock>(); //初始化缓存
downloads = new DownloadList();
deamon = new Thread(new DownloadDeamon(downloads)); //初始化守护进程
deamon.setDaemon(true);
deamon.start();
timer = new Timer(Constants.TIMER_NAME, true);
}
上面代码中启动了一个守护进程。那么这个守护进程在启动的时候在做什么事情呢?
我们来看一下他的run()函数
public void run() {
System.out.println("Create thread pool");
threadPool = Executors.newCachedThreadPool(); //初始化线程池
DownloadBlock block;
while(true){
block = getDownloadBlock(); //不断从当前系统中获取待下载的DownloadBlock
if(block != null){
log.info("Create new download thread for " + block);
//启动线程执行下载
threadPool.execute(new DownloadThread(block));
//将当前Block从其所在的Download中移除
block.getDownload().removeDownloadBlock(block);
}
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
System.out.println("Download deamon stoped by user");
break;
}
}
}
守护进程所做的事情就是不断获取将要进行下载的Block,然后启动线程去进行下载。
来看一下获取Block的策略:这里不断的从当前下载列表中获取所有的Download,然后从里面选取最需要下载的文件,“最需要下载”定义为剩余的待下载量最多。其具体的代码看下方:
private DownloadBlock getDownloadBlock(){
downs= downloads.toArray();
if(downs== null || downs.length == 0)
return null;
Downloaddownload = downs[0];
for(Downloaddown : downs){//找最需要下载的Download进行下载 if(down.getRemainThread()> download.getRemainThread())
download= down;
}
download.descRemainThread();
returndownload.getDownloadBlock();
}
2. 新建下载
上面讲解的是系统初始化所做的事情。那么当我们把开始下载一个文件时系统是怎么运行的呢?
//下载第一个文件
String url1 = "https://tmsvm.googlecode.com/files/tmsvm_src_v1.1.0.rar";
String saveFile1 = "data/tmsvm_src_v1.1.0.rar";
DownloadConfig config = new DownloadConfig();
try {
config.setUrl(new URL(url1));
config.setFile(new File(saveFile1));
config.setNthread(new Integer(5));
config.setPriority(new Integer(6));
//将第一个下载加入到下载列表中
downloader.addDownload(new Download(config, downloader.getTimer()));
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
我们重点来看这一句:
//将第一个下载加入到下载列表中
downloader.addDownload(new Download(config, downloader.getTimer()));
addDownload的定义如下public boolean addDownload(Download download){
new Thread(download).start();
return downloads.add(download);
}
这段代码做了两件事情:
- 为Download启动一下线程。Download线程所做的事情就是把当前的文件根据线程数目进行切分。
- 把当前Download加入到DownloadList中。
OK,我们看看,Download是切分文件时是如何与整个系统联系在一起。
public void run(){
try {
begin = System.currentTimeMillis();
// get length
log.info("Begin download " + config.getUrl());
length = config.getUrl().openConnection().getContentLength();
log.info("Total size : " + length);
// create file
log.info("Create file " + config.getFile());
RandomAccessFile file = new RandomAccessFile(config.getFile(), "rw");
file.setLength(length);
log.info("Created with length = " + length);
file.close();
int size = length / config.getNthread();
// add initial blocks
log.debug("Add initial " + config.getNthread() + " download blocks");
for(int i = 0; i < config.getNthread(); i++){
int start = i * size;
int len;
if(i == config.getNthread() - 1)
len = length - start;
else len = size;
addDownloadBlock(getDownloadBlock(start, len));
}
// set task that checks speed every 1 second
log.debug("Set task for speed check");
checkSpeedTask = new CheckSpeedTask(this, System.currentTimeMillis()-10, blockCounts);
//timer.schedule(checkSpeedTask, 1000, 1000);
// set task that creates new blocks every 1 minute
log.debug("Set task for split blocks");
splitBlockTask = new SplitBlockTask(this, System.currentTimeMillis()-10, blockCounts, activeThreads);
timer.schedule(splitBlockTask, 60*1000, 60*1000);
// wait for all blocks complete
log.debug("Waiting for all blocks to complete");
while(activeThreads.size() > 0 || blockQueue.size() > 0){
Thread.sleep(1000);
checkSpeed();
}
// stop the tasks
checkSpeedTask.cancel();
splitBlockTask.cancel();
long total = System.currentTimeMillis() - begin;
speed = length/(total/1000);
log.info("Complete download " + config.getUrl() + "\n"
+ "Total time : " + total + " ms" + "\n"
+ "Average speed: " + speed + "Byte/s"
);
log.debug(this + " put all block in blockCache back to downloader system");
for(DownloadBlock block : blockCache){
Downloader.getInstance().putDownloadBlock(block);
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
上面的代码,我们在第一篇中已经讲解过了,这里我们会重点看
addDownloadBlock(getDownloadBlock(start, len));
其意思是将当前的切分出来的Block放入到待下载队列中去。
而我们在守护进程那里,看到他不断的会从当前系统中找最需要下载的Download,然后再从Download中取出下载队列的Block进行下载
//DownloadDemen不断的获取DownloadBlock
while(true){
block = getDownloadBlock();
if(block != null){
System.out.println("Create new download thread for " + block);
threadPool.execute(new DownloadThread(block));
block.getDownload().removeDownloadBlock(block);
}
}
//getDownloadBlock()定义如下:
private DownloadBlock getDownloadBlock(){
downs= downloads.toArray();
if(downs== null || downs.length == 0)
return null;
Downloaddownload = downs[0];
for(Downloaddown : downs){
if(down.getRemainThread()> download.getRemainThread())
download= down;
}
download.descRemainThread();
return download.getDownloadBlock();
}
//而download.getDownloadBlock()定义如下所示:
public DownloadBlock getDownloadBlock(){
return blockQueue.peek();
}
写到这里,整个的系统框架目录就非常清晰了:Downloader, DownloadDemen, Download 之间是通过DownloadBlock联系起来的。
当有一个文件需要下载时,Downloader 把该Download加入到DownloadList中。而Download自身会通过切分文件创建出多个DownloadBlock。DownloadDemen每时每刻都在获取DownloadBlock,赋予其线程进行下载。