MultiThreadedHttpConnectionManager 创建新线程失败

 

写的一个爬虫工具，多次因为抛出这样的异常而退出。

java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:597)
        at java.util.Timer.<init>(Timer.java:154)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.<init>(MultiThreadedHttpConnectionManager.java:255)

  。。。。

 

后来终于明白，

每抓一次网页就新建一个MultiThreadedHttpConnectionManager对象，如此的低级错误，这个bug太严重了！！每回都三次握手。。。

实在不该啊！！！只需要新建一个MultiThreadedHttpConnectionManager对象就可以了。

 

 

 

        HttpState initialState = new HttpState();

        MultiThreadedHttpConnectionManager connectionManager;
        connectionManager = new MultiThreadedHttpConnectionManager();

        //handshake 3 times
        connectionManager.getParams().setMaxTotalConnections(1000);
        connectionManager.getParams().setConnectionTimeout(5 * 3 * 1000);
        connectionManager.getParams().setSoTimeout(15 * 1000);
        connectionManager.getParams().setSendBufferSize(16 * 1024);
        connectionManager.getParams().setReceiveBufferSize(1024 * 512);

        hc = new HttpClient(connectionManager);
        hc.setState(initialState);

        // set agent string
        hc.getParams().setParameter(HttpClientParams.USER_AGENT, ApiConst.AGENT_STRING);
        // use compatibility cookie mode because the world is a cruel place
        hc.getParams().setParameter(HttpClientParams.COOKIE_POLICY,
                "COMPATIBILITY");

        hc.getParams().makeLenient();
        // no need to process retry
        hc.getParams().setParameter(HttpClientParams.RETRY_HANDLER,
                null);