java网络编程之Http多线程下载应用实例

最新推荐文章于 2024-09-19 15:31:35 发布

~诸行无常~

最新推荐文章于 2024-09-19 15:31:35 发布

阅读量3.3k

点赞数

分类专栏： Java RandomAccessFile java网络编程文章标签：多线程 java 网络编程 download thread

本文链接：https://blog.csdn.net/hejiangtao/article/details/7212965

版权

Java 同时被 3 个专栏收录

18 篇文章 0 订阅

订阅专栏

RandomAccessFile

1 篇文章 0 订阅

订阅专栏

java网络编程

1 篇文章 0 订阅

订阅专栏

本demo通过RandomAccessFile, URLConnection和多线程机制实现了Http下载功能.从这里可以下载到完整的java代码工程: http://download.csdn.net/detail/hejiangtao/4029935. 相对于别的网上的例子来看, 本demo是可运行的, 可以判断网络资源是否支持分段下载. 你是否遇到了java下载的图片显示不出来或者RAR解压不了的情况, 可以参考本demo的解决方案

设计思路:

1. 首先读取文件的长度, 并判断网站是否支持分段下载

2. 如果支持分段下载则创建多个线程同时下载该文件,否则使用单线程下载

3. 在各个线程中,分别使用RandomAccessFile对象写入对应的文件位置

具体实现解析:

1. 为了方便理解,先把util类贴出来.

SysValue.java:

在实际应用中,这些参数应该可以在界面配置或者配置文件配置.

package com.ross.httpdownload.util;
/**
 * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com
 * Date: 2012-1-18
 * Since: MyJavaExpert v1.0
 * Description:
 */
public class SysValue
{
    //constant values
    //maximum number of the download thread
    public static int MAX_Num_Of_Thread = 3;
    //download file store path
    public static String Save_As_Path = "E:/myspace/download/";
    //buffer size
    public static int Buffer_Size = 1024;    
}

HttpProcess.java:

将Http相关的公共操作放在这个类里面,避免写重复代码.

package com.ross.httpdownload.util;
import java.io.IOException;
import java.net.*;
/**
 * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com
 * Date: 2012-1-18
 * Since: MyJavaExpert v1.0
 * Description: This class will provide the common process of http
 */
public class HttpProcess
{
    /**
     * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com
     * Description: get an object of opened URL connection
     * @param sURL: completed URL string
     * @return oHttpURLCon: an object of opened URL connection
     * @throws IOException
     */
    public URLConnection  getURLConnection(String sURL) throws IOException
    {
        URLConnection  oURLCon = null;
        // replace space with '+' in the URL string
        if (null != sURL)
        {
            sURL = sURL.replaceAll("\\s", "+");
            // generate URL
            URL oURL = new URL(sURL);
            oURLCon = (URLConnection) oURL.openConnection();
        }
        return oURLCon;
    }

}

2. 看我们的多线程下载实现

a. 首先看下如何判断网站资源是否支持分段下载

很多网络资源,即使在java里面设置了http的InputStream流的范围后, 取的的流的长度还是整个文件的长度. 针对这种情况我就认为该资源不支持分段下载,也就是无法多线程下载了(使用原来的方式下载你就发现下载的图片只能显示一半,或者不能显示,RAR也是损坏的, 字节不对吗 :)). 我也试过通过使用InputStream流的skip方法跳过前面内容的方式实现多线程,但是事实证明,skip太不靠谱,跳的步长并不是一定的,哎什么玩意,只好放弃.

    /**
     * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com
     * Description: Check whether the server support multiple thread download or not
     * @param sURL: the download file URL
     * @return bRet: true - support, false - not support
     * @throws IOException 
     */
    public boolean isSupportMultipleThreadDown(String sURL) throws IOException
    {
        boolean bRet = true;
       // get URL connection
        URLConnection oURLCon = oHttpProcess.getURLConnection(sURL);
       // set the resource range
        oURLCon.setRequestProperty("Range", "bytes=" + 0 + "-" + 1023);

        if (1024 < oURLCon.getContentLength())
        {
            bRet = false;
        }
        return bRet;
    }

b. 然后我们看下如何,将文件分段下载

1) 首先是预处理

先是做了个简单的判断,防止线程过多或者小于1; 然后判断网络资源是否支持分段下载,如果不支持则将线程数重置为1; 在获取整个文件长度后, 使用总长度除以线程数得出每个线程负责下载的字节范围.

2) 使用线程池管理并启动多线程下载

DownloadTread就是我们的线程下载实现类, 后面会单独说明. 众所周知java下标是从0开始的,所以各个线程分配字节范围的时候,要考虑进去. 将对应分配的字节范围,URL,文件名信息设置给DownloadTread对象后,使用ExecutorService启动线程.

3)当然不要忘记在finally里面调用线程池的关闭方法.

    /**
     * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com
     * Description: download tool entry method
     * @param sURL: the completed URL of the file, which is wanted to download. 
     * @param Num: Number of concurrent download thread
     * @param sSaveAsName: the the file name saved in disk, like: myphoto.jpg
     */
    public void download(String sURL, int iNumOfThread, String sSaveAsName)
    {
        long lFileLen = 0;
        long lFileSegementLen = 0;
        long iLeft = 0;
        URLConnection oURLCon = null;
        String sFullFileName = SysValue.Save_As_Path + sSaveAsName;

        if (iNumOfThread > SysValue.MAX_Num_Of_Thread)
        {
            iNumOfThread = SysValue.MAX_Num_Of_Thread;
        }
        if (iNumOfThread < 1)
        {
            iNumOfThread = 1;
        }

        // build the thread pool to manage the thread
        ExecutorService oExecService = Executors
                .newFixedThreadPool(iNumOfThread);
        try
        {
            if (!isSupportMultipleThreadDown(sURL))
            {
                System.out
                        .println("This site do not support multiple thread download, reset the thread to 1");
                iNumOfThread = 1;
            }
            // get URL connection
            oURLCon = oHttpProcess.getURLConnection(sURL);
            // get the length of the file
            lFileLen = oURLCon.getContentLength();
            // split the length of the file according to the number of thread.
            lFileSegementLen = lFileLen / iNumOfThread;
            iLeft = lFileLen % iNumOfThread;

            for (int i = 0; i < iNumOfThread; i++)
            {
                // initial the thread data
                DownloadThread oDownloadThread = new DownloadThread();
                oDownloadThread.setIThreadId(i);
                // Start index of the current thread
                oDownloadThread.setLStartIndex(i * lFileSegementLen);

                // last thread will take all the rest content
                if (i == (iNumOfThread - 1))
                {
                    oDownloadThread.setLEndIndex((i + 1) * lFileSegementLen
                            + iLeft - 1);
                }
                else
                {
                    oDownloadThread
                            .setLEndIndex((i + 1) * lFileSegementLen - 1);
                }
                // set save file name with path
                oDownloadThread.setSFullFileName(sFullFileName);
                // set URL
                oDownloadThread.setSFileURL(sURL);
                // execute the download thread
                oExecService.execute(oDownloadThread);
            }
            System.out.println("the file is stored as \"" + sFullFileName
                    + "\".");
            System.out.println("the file length is " + lFileLen + ".");
        }
        catch (IOException e)
        {
            System.out.println("download the file failed, IO Exception"
                    + e.getMessage());
            e.printStackTrace();
        }
        finally
        {
            oExecService.shutdown();
        }
    }

c.最后就是我们的核心下载线程实现类了

1) DownloadTread类实现了Runnalbe抽象接口, 实现了其run()方法.

2) 两个关键的地方一是通过使用URLConnection 的setRequestProperty方法设置本线程读取的流的范围, 另外一个是通过RandomAccessFile 的seek方法或者当前线程写入文件的位置, 一个字节都不能错哦,错了你的文件就有差错了, 还是要记住java下标是从0开始的.

3)然后通过循环从URLConnection的InputStream里面通过buffer一组一组的读出来, 通过RandomAccessFile 对象一组一组的写入到目标文件里面(代码里面判断实际读取长度部分是不需要的, 可以去掉, 是我探索不支持分段下载网络资源的多线程下载方式时留下的).

4)需要提一下的是run()方法是没有输入参数的,所以我们通过增加类字段的方式将参数传入该方法.

package com.ross.httpdownload;
import java.io.*;
import java.net.*;
import java.text.*;
import java.util.Date;
import com.ross.httpdownload.util.*;
/**
 * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com
 * Date: 2012-1-18
 * Since: MyJavaExpert v1.0
 * Description: download thread implementation.
 */
public class DownloadThread implements Runnable
{
    private String sFileURL;
    private long lStartIndex;
    private long lEndIndex;
    private int iThreadId;
    private String sFullFileName;
    private HttpProcess oHttpProcess;
    /**
     * Description: default constructor, it is used for initializing the fields.
     */
    public DownloadThread()
    {
        oHttpProcess = new HttpProcess();
    }
    /**
     * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com
     * Description: it will implement the download logic
     */
    public void run()
    {
        System.out.println("The download thread "
                + this.iThreadId
                + " is started at "
                + (new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"))
                        .format(new Date()));
        System.out.println("The download index of " + this.iThreadId + " is "
                + this.lStartIndex + " to " + this.lEndIndex);

        // content length
        long lContentlen = 0;
        // used to store the new url connection
        URLConnection oURLCon = null;
        // used to store the input stream of the response
        BufferedInputStream oBIn = null;
        // used to store the output writer
        RandomAccessFile oRAFile = null;
        // create the buffer
        byte[] bBuffer = new byte[SysValue.Buffer_Size];

        // get URL connection
        try
        {
            // create a link for each thread
            oURLCon = oHttpProcess.getURLConnection(this.sFileURL);
            // allow the user interaction, for example: a verify pop window
            oURLCon.setAllowUserInteraction(true);
            // set the resource range
            oURLCon.setRequestProperty("Range", "bytes=" + this.lStartIndex
                    + "-" + this.lEndIndex);

            // get the url connection input stream
            oBIn = new BufferedInputStream(oURLCon.getInputStream());
 
            // initialize the random access file object
            oRAFile = new RandomAccessFile(this.sFullFileName, "rw"); 
            oRAFile.seek(this.lStartIndex);

            // read the stream from http connection
            int iLen = 0;
            int iActualLen = 0;
            while (iActualLen < (this.lEndIndex - this.lStartIndex + 1))
            {
                iLen = oBIn.read(bBuffer, 0, SysValue.Buffer_Size);
                if (-1 == iLen)
                {
                    break;
                }

                // write the read data to file
                oRAFile.write(bBuffer, 0, iLen);

                // move the position mark
                iActualLen = iActualLen + iLen;
            }
            System.out.println("Thread " + this.iThreadId
                    + " download is finished, totaly: " + iActualLen);
        }
        catch (IOException e)
        {
            System.out.println("download the file failed, IO Exception"
                    + e.getMessage());
            e.printStackTrace();
        }
        finally
        {
            if (null != oRAFile)
            {
                try
                {
                    oRAFile.close();
                }
                catch (IOException e)
                {
                    System.out
                            .println("close object of RandomAccessFile failed, IO Exception"
                                    + e.getStackTrace());
                }
            }

            if (null != oBIn)
            {
                try
                {
                    oBIn.close();
                }
                catch (IOException e)
                {
                    System.out
                            .println("close input stream of http connection failed, IO Exception"
                                    + e.getStackTrace());
                }
            }
        }
        System.out.println("The download thread "
                + this.iThreadId
                + " is ended at "
                + (new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"))
                        .format(new Date()));
    }
   //省略了set/get方法  
}

来测试下我们的实现:

看下我们的main函数, 我们就通过3个例子来测试我们的实现: 一个可以多线程下载的zip文件,一个可以多线程下载的图片,一个不能多线程下载图片.

MyMain.java:

package com.ross.httpdownload;
import com.ross.httpdownload.util.*;
/**
 * Author: Jiangtao He; Email: ross.jiangtao.he@gmail.com
 * Date: 2012-1-19
 * Since: MyJavaExpert v1.0
 * Description: Test the download tool class
 */
public class MyMain
{
    public static void main(String[] args)
    {
        // set system configuration, in real software, it can be configured
        // through GUI, file,etc.
        SysValue.MAX_Num_Of_Thread = 3;
        SysValue.Save_As_Path = "E:/myspace/download/";
        SysValue.Buffer_Size = 1024;
        //file web address
        //String sFileURL = "http://xinsheng-image.huawei.com/cn/forumimage/showimage-991441-20990f56c5b11261fa63e969d8d5580c-self.jpg";
        //String sFileURL = "http://lh5.googleusercontent.com/-F7a0loCDyoQ/AAAAAAAAAAI/AAAAAAAAAFY/TrDxSDdQuhQ/s512-c/photo.jpg";
        //String sFileURL = "http://xinsheng-image.huawei.com/cn/forumimage/showimage-816607-f61c8fe14fc359d49b044376a0956acd-self.jpg";
        //String sFileURL = "http://3.bp.blogspot.com/-oVO24VW8F6M/TxJ9O0opP3I/AAAAAAAAANY/anpu8S4FAC4/s640/100_9223.JPG";
        String sFileURL = "http://hi.csdn.net/attachment/201201/1/0_1325433530Bg5e.gif";
        sFileURL = "http://west263.newhua.com:82/down/jia-audio-converter.zip";
        //file save as name
        String sFileName = sFileURL.substring(sFileURL.lastIndexOf("/") + 1);
       
        // create a object of download tool
        HttpDownloadTool oHttpDownloadTool = new HttpDownloadTool();
        // download the file
        oHttpDownloadTool.download(sFileURL, 3, sFileName);
    }
}

a. 可以多线程下载的zip文件测试

URL: http://west263.newhua.com:82/down/jia-audio-converter.zip

线程数: 3

控制台打印信息:

the file is stored as "E:/myspace/download/jia-audio-converter.zip".
the file length is 5720120.
The download thread 1 is started at 2012-01-21 00:49:23
The download index of 1 is 1906706 to 3813411
The download thread 0 is started at 2012-01-21 00:49:23
The download index of 0 is 0 to 1906705
The download thread 2 is started at 2012-01-21 00:49:23
The download index of 2 is 3813412 to 5720119
Thread 0 download is finished, totaly: 1906706
The download thread 0 is ended at 2012-01-21 00:58:09
Thread 1 download is finished, totaly: 1906706
The download thread 1 is ended at 2012-01-21 01:06:17
Thread 2 download is finished, totaly: 1906708
The download thread 2 is ended at 2012-01-21 01:07:05

下载的文件抓图:

b.可以多线程下载的图片测试

URL: http://hi.csdn.net/attachment/201201/1/0_1325433530Bg5e.gif

线程数: 3

控制台打印信息:

the file is stored as "E:/myspace/download/0_1325433530Bg5e.gif".
the file length is 23304.
The download thread 2 is started at 2012-01-21 02:03:00
The download thread 0 is started at 2012-01-21 02:03:00
The download index of 2 is 15536 to 23303
The download index of 0 is 0 to 7767
The download thread 1 is started at 2012-01-21 02:03:00
The download index of 1 is 7768 to 15535
Thread 2 download is finished, totaly: 7768
The download thread 2 is ended at 2012-01-21 02:03:01
Thread 1 download is finished, totaly: 7768
The download thread 1 is ended at 2012-01-21 02:03:01
Thread 0 download is finished, totaly: 7768
The download thread 0 is ended at 2012-01-21 02:03:02

下载的文件抓图:

c. 不能多线程下载图片

URL: http://3.bp.blogspot.com/-oVO24VW8F6M/TxJ9O0opP3I/AAAAAAAAANY/anpu8S4FAC4/s640/100_9223.JPG

线程数: 3

控制台打印信息:

This site do not support multiple thread download, reset the thread to 1
the file is stored as "E:/myspace/download/100_9223.JPG".
the file length is 170398.
The download thread 0 is started at 2012-01-21 02:06:54
The download index of 0 is 0 to 170397
Thread 0 download is finished, totaly: 170398
The download thread 0 is ended at 2012-01-21 02:06:55

下载的文件抓图: