本demo通过RandomAccessFile, URLConnection和多线程机制实现了Http下载功能.从这里可以下载到完整的java代码工程: http://download.csdn.net/detail/hejiangtao/4029935. 相对于别的网上的例子来看, 本demo是可运行的, 可以判断网络资源是否支持分段下载. 你是否遇到了java下载的图片显示不出来或者RAR解压不了的情况, 可以参考本demo的解决方案
设计思路:
1. 首先读取文件的长度, 并判断网站是否支持分段下载
2. 如果支持分段下载则创建多个线程同时下载该文件,否则使用单线程下载
3. 在各个线程中,分别使用RandomAccessFile对象写入对应的文件位置
具体实现解析:
1. 为了方便理解,先把util类贴出来.
SysValue.java:
在实际应用中,这些参数应该可以在界面配置或者配置文件配置.
HttpProcess.java:package com.ross.httpdownload.util; /** * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com * Date: 2012-1-18 * Since: MyJavaExpert v1.0 * Description: */ public class SysValue { //constant values //maximum number of the download thread public static int MAX_Num_Of_Thread = 3; //download file store path public static String Save_As_Path = "E:/myspace/download/"; //buffer size public static int Buffer_Size = 1024; }将Http相关的公共操作放在这个类里面,避免写重复代码.
package com.ross.httpdownload.util; import java.io.IOException; import java.net.*; /** * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com * Date: 2012-1-18 * Since: MyJavaExpert v1.0 * Description: This class will provide the common process of http */ public class HttpProcess { /** * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com * Description: get an object of opened URL connection * @param sURL: completed URL string * @return oHttpURLCon: an object of opened URL connection * @throws IOException */ public URLConnection getURLConnection(String sURL) throws IOException { URLConnection oURLCon = null; // replace space with '+' in the URL string if (null != sURL) { sURL = sURL.replaceAll("\\s", "+"); // generate URL URL oURL = new URL(sURL); oURLCon = (URLConnection) oURL.openConnection(); } return oURLCon; } }
2. 看我们的多线程下载实现
a. 首先看下如何判断网站资源是否支持分段下载
很多网络资源,即使在java里面设置了http的InputStream流的范围后, 取的的流的长度还是整个文件的长度. 针对这种情况我就认为该资源不支持分段下载,也就是无法多线程下载了(使用原来的方式下载你就发现下载的图片只能显示一半,或者不能显示,RAR也是损坏的, 字节不对吗 :)). 我也试过通过使用InputStream流的skip方法跳过前面内容的方式实现多线程,但是事实证明,skip太不靠谱,跳的步长并不是一定的,哎什么玩意,只好放弃.
/** * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com * Description: Check whether the server support multiple thread download or not * @param sURL: the download file URL * @return bRet: true - support, false - not support * @throws IOException */ public boolean isSupportMultipleThreadDown(String sURL) throws IOException { boolean bRet = true; // get URL connection URLConnection oURLCon = oHttpProcess.getURLConnection(sURL); // set the resource range oURLCon.setRequestProperty("Range", "bytes=" + 0 + "-" + 1023); if (1024 < oURLCon.getContentLength()) { bRet = false; } return bRet; }
b. 然后我们看下如何,将文件分段下载1) 首先是预处理
先是做了个简单的判断,防止线程过多或者小于1; 然后判断网络资源是否支持分段下载,如果不支持则将线程数重置为1; 在获取整个文件长度后, 使用总长度除以线程数得出每个线程负责下载的字节范围.
2) 使用线程池管理并启动多线程下载
DownloadTread就是我们的线程下载实现类, 后面会单独说明. 众所周知java下标是从0开始的,所以各个线程分配字节范围的时候,要考虑进去. 将对应分配的字节范围,URL,文件名信息设置给DownloadTread对象后,使用ExecutorService启动线程.
3)当然不要忘记在finally里面调用线程池的关闭方法.
/** * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com * Description: download tool entry method * @param sURL: the completed URL of the file, which is wanted to download. * @param Num: Number of concurrent download thread * @param sSaveAsName: the the file name saved in disk, like: myphoto.jpg */ public void download(String sURL, int iNumOfThread, String sSaveAsName) { long lFileLen = 0; long lFileSegementLen = 0; long iLeft = 0; URLConnection oURLCon = null; String sFullFileName = SysValue.Save_As_Path + sSaveAsName; if (iNumOfThread > SysValue.MAX_Num_Of_Thread) { iNumOfThread = SysValue.MAX_Num_Of_Thread; } if (iNumOfThread < 1) { iNumOfThread = 1; } // build the thread pool to manage the thread ExecutorService oExecService = Executors .newFixedThreadPool(iNumOfThread); try { if (!isSupportMultipleThreadDown(sURL)) { System.out .println("This site do not support multiple thread download, reset the thread to 1"); iNumOfThread = 1; } // get URL connection oURLCon = oHttpProcess.getURLConnection(sURL); // get the length of the file lFileLen = oURLCon.getContentLength(); // split the length of the file according to the number of thread. lFileSegementLen = lFileLen / iNumOfThread; iLeft = lFileLen % iNumOfThread; for (int i = 0; i < iNumOfThread; i++) { // initial the thread data DownloadThread oDownloadThread = new DownloadThread(); oDownloadThread.setIThreadId(i); // Start index of the current thread oDownloadThread.setLStartIndex(i * lFileSegementLen); // last thread will take all the rest content if (i == (iNumOfThread - 1)) { oDownloadThread.setLEndIndex((i + 1) * lFileSegementLen + iLeft - 1); } else { oDownloadThread .setLEndIndex((i + 1) * lFileSegementLen - 1); } // set save file name with path oDownloadThread.setSFullFileName(sFullFileName); // set URL oDownloadThread.setSFileURL(sURL); // execute the download thread oExecService.execute(oDownloadThread); } System.out.println("the file is stored as \"" + sFullFileName + "\"."); System.out.println("the file length is " + lFileLen + "."); } catch (IOException e) { System.out.println("download the file failed, IO Exception" + e.getMessage()); e.printStackTrace(); } finally { oExecService.shutdown(); } }
c.最后就是我们的核心下载线程实现类了1) DownloadTread类实现了Runnalbe抽象接口, 实现了其run()方法.
2) 两个关键的地方一是通过使用URLConnection 的setRequestProperty方法设置本线程读取的流的范围, 另外一个是通过RandomAccessFile 的seek方法或者当前线程写入文件的位置, 一个字节都不能错哦,错了你的文件就有差错了, 还是要记住java下标是从0开始的.
3)然后通过循环从URLConnection的InputStream里面通过buffer一组一组的读出来, 通过RandomAccessFile 对象一组一组的写入到目标文件里面(代码里面判断实际读取长度部分是不需要的, 可以去掉, 是我探索不支持分段下载网络资源的多线程下载方式时留下的).
4)需要提一下的是run()方法是没有输入参数的,所以我们通过增加类字段的方式将参数传入该方法.
package com.ross.httpdownload; import java.io.*; import java.net.*; import java.text.*; import java.util.Date; import com.ross.httpdownload.util.*; /** * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com * Date: 2012-1-18 * Since: MyJavaExpert v1.0 * Description: download thread implementation. */ public class DownloadThread implements Runnable { private String sFileURL; private long lStartIndex; private long lEndIndex; private int iThreadId; private String sFullFileName; private HttpProcess oHttpProcess; /** * Description: default constructor, it is used for initializing the fields. */ public DownloadThread() { oHttpProcess = new HttpProcess(); } /** * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com * Description: it will implement the download logic */ public void run() { System.out.println("The download thread " + this.iThreadId + " is started at " + (new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")) .format(new Date())); System.out.println("The download index of " + this.iThreadId + " is " + this.lStartIndex + " to " + this.lEndIndex); // content length long lContentlen = 0; // used to store the new url connection URLConnection oURLCon = null; // used to store the input stream of the response BufferedInputStream oBIn = null; // used to store the output writer RandomAccessFile oRAFile = null; // create the buffer byte[] bBuffer = new byte[SysValue.Buffer_Size]; // get URL connection try { // create a link for each thread oURLCon = oHttpProcess.getURLConnection(this.sFileURL); // allow the user interaction, for example: a verify pop window oURLCon.setAllowUserInteraction(true); // set the resource range oURLCon.setRequestProperty("Range", "bytes=" + this.lStartIndex + "-" + this.lEndIndex); // get the url connection input stream oBIn = new BufferedInputStream(oURLCon.getInputStream()); // initialize the random access file object oRAFile = new RandomAccessFile(this.sFullFileName, "rw"); oRAFile.seek(this.lStartIndex); // read the stream from http connection int iLen = 0; int iActualLen = 0; while (iActualLen < (this.lEndIndex - this.lStartIndex + 1)) { iLen = oBIn.read(bBuffer, 0, SysValue.Buffer_Size); if (-1 == iLen) { break; } // write the read data to file oRAFile.write(bBuffer, 0, iLen); // move the position mark iActualLen = iActualLen + iLen; } System.out.println("Thread " + this.iThreadId + " download is finished, totaly: " + iActualLen); } catch (IOException e) { System.out.println("download the file failed, IO Exception" + e.getMessage()); e.printStackTrace(); } finally { if (null != oRAFile) { try { oRAFile.close(); } catch (IOException e) { System.out .println("close object of RandomAccessFile failed, IO Exception" + e.getStackTrace()); } } if (null != oBIn) { try { oBIn.close(); } catch (IOException e) { System.out .println("close input stream of http connection failed, IO Exception" + e.getStackTrace()); } } } System.out.println("The download thread " + this.iThreadId + " is ended at " + (new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")) .format(new Date())); } //省略了set/get方法 }
来测试下我们的实现:
看下我们的main函数, 我们就通过3个例子来测试我们的实现: 一个可以多线程下载的zip文件,一个可以多线程下载的图片,一个不能多线程下载图片.
MyMain.java:
package com.ross.httpdownload; import com.ross.httpdownload.util.*; /** * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com * Date: 2012-1-19 * Since: MyJavaExpert v1.0 * Description: Test the download tool class */ public class MyMain { public static void main(String[] args) { // set system configuration, in real software, it can be configured // through GUI, file,etc. SysValue.MAX_Num_Of_Thread = 3; SysValue.Save_As_Path = "E:/myspace/download/"; SysValue.Buffer_Size = 1024; //file web address //String sFileURL = "http://xinsheng-image.huawei.com/cn/forumimage/showimage-991441-20990f56c5b11261fa63e969d8d5580c-self.jpg"; //String sFileURL = "http://lh5.googleusercontent.com/-F7a0loCDyoQ/AAAAAAAAAAI/AAAAAAAAAFY/TrDxSDdQuhQ/s512-c/photo.jpg"; //String sFileURL = "http://xinsheng-image.huawei.com/cn/forumimage/showimage-816607-f61c8fe14fc359d49b044376a0956acd-self.jpg"; //String sFileURL = "http://3.bp.blogspot.com/-oVO24VW8F6M/TxJ9O0opP3I/AAAAAAAAANY/anpu8S4FAC4/s640/100_9223.JPG"; String sFileURL = "http://hi.csdn.net/attachment/201201/1/0_1325433530Bg5e.gif"; sFileURL = "http://west263.newhua.com:82/down/jia-audio-converter.zip"; //file save as name String sFileName = sFileURL.substring(sFileURL.lastIndexOf("/") + 1); // create a object of download tool HttpDownloadTool oHttpDownloadTool = new HttpDownloadTool(); // download the file oHttpDownloadTool.download(sFileURL, 3, sFileName); } }
a. 可以多线程下载的zip文件测试
URL: http://west263.newhua.com:82/down/jia-audio-converter.zip
线程数: 3
控制台打印信息:
下载的文件抓图:the file is stored as "E:/myspace/download/jia-audio-converter.zip". the file length is 5720120. The download thread 1 is started at 2012-01-21 00:49:23 The download index of 1 is 1906706 to 3813411 The download thread 0 is started at 2012-01-21 00:49:23 The download index of 0 is 0 to 1906705 The download thread 2 is started at 2012-01-21 00:49:23 The download index of 2 is 3813412 to 5720119 Thread 0 download is finished, totaly: 1906706 The download thread 0 is ended at 2012-01-21 00:58:09 Thread 1 download is finished, totaly: 1906706 The download thread 1 is ended at 2012-01-21 01:06:17 Thread 2 download is finished, totaly: 1906708 The download thread 2 is ended at 2012-01-21 01:07:05
b.可以多线程下载的图片测试
URL: http://hi.csdn.net/attachment/201201/1/0_1325433530Bg5e.gif
线程数: 3
控制台打印信息:
下载的文件抓图:the file is stored as "E:/myspace/download/0_1325433530Bg5e.gif". the file length is 23304. The download thread 2 is started at 2012-01-21 02:03:00 The download thread 0 is started at 2012-01-21 02:03:00 The download index of 2 is 15536 to 23303 The download index of 0 is 0 to 7767 The download thread 1 is started at 2012-01-21 02:03:00 The download index of 1 is 7768 to 15535 Thread 2 download is finished, totaly: 7768 The download thread 2 is ended at 2012-01-21 02:03:01 Thread 1 download is finished, totaly: 7768 The download thread 1 is ended at 2012-01-21 02:03:01 Thread 0 download is finished, totaly: 7768 The download thread 0 is ended at 2012-01-21 02:03:02
c. 不能多线程下载图片
URL: http://3.bp.blogspot.com/-oVO24VW8F6M/TxJ9O0opP3I/AAAAAAAAANY/anpu8S4FAC4/s640/100_9223.JPG
线程数: 3
控制台打印信息:
下载的文件抓图:This site do not support multiple thread download, reset the thread to 1 the file is stored as "E:/myspace/download/100_9223.JPG". the file length is 170398. The download thread 0 is started at 2012-01-21 02:06:54 The download index of 0 is 0 to 170397 Thread 0 download is finished, totaly: 170398 The download thread 0 is ended at 2012-01-21 02:06:55
到这里整个demo完成了.
注: 转载请注明出处: http://hejiangtao.iteye.com/, 用于商业得给我分成
777
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
