在写网络爬虫的时候,需要根据链接来获取文件类型,将内容正确存储。之前我都是根据链接的后缀来判断的,比如:
http://img12.360buyimg.com/da/20120330/88_31_ZySDre.jpg
这个链接指向的文件就是个jpg文件。但是后来发现有诸如
http://jprice.360buyimg.com/getSkuPriceImgService.action?skuId=1850001109&origin=1&webSite=1&type=1的链接,这招就不灵了。后来谷歌百度了一下也没发现解决办法。后来机缘巧合在Java Network Programming上找到了一个办法:
URLConnection class provides two static methods to help programs figure out the MIME type of some data; you can use these if the content type just isn't available or if you have reason to believe that the content type you're given isn't correct。
就是说URLConnection提供了两种方法可以猜测(根据实测结果,这个猜测是相当的准)数据的MIME类型。
第一个是:
public static String guessContentTypeFromName(String name)
这个方法根据URL文件部分的后缀名来判断类型,跟之前我的方法一样。这个不能解决上面那个问题。
第二个是:
public static String guessContentTypeFromStream(InputStream in)
这个方法是根据流的前面几个字节来判断类型,这个就不需要文件后缀名了,完全可以解决上面那个问题。
测试代码如下:
以上
java.net.ProtocolException: Can't reset method: already connected
at java.net.HttpURLConnection.setRequestMethod(HttpURLConnection.java:314)
at MultiDowloader.run(MultiDowloader.java:33)
今天写多线程下载的时候总是遇到以上的问题
import java.net.*;
import java.io.*;
public class BadURLPost
{
public static void main(String args[])
{
// get an HTTP connection to POST to
try
{
// get the url as a string
String surl ="http://www.foshanshop.net/ejb3/ActivePort.exe";
URL url = new URL(surl);
URLConnection con = url.openConnection();
System.out.println("Received a : " + con.getClass().getName());
con.setDoInput(true);
con.setDoOutput(true);
con.setUseCaches(false);
String msg = "Hi HTTP SERVER! Just a quick hello!";
con.setRequestProperty("CONTENT_LENGTH", "5"); // Not checked
con.setRequestProperty("Stupid", "Nonsense");
System.out.println("Getting an output stream...");
OutputStream os = con.getOutputStream();
System.out.println("Getting an input stream...");
InputStream is = con.getInputStream();
/*
con.setRequestProperty("CONTENT_LENGTH", "" + msg.length());
Illegal access error - can't reset method.
*/
OutputStreamWriter osw = new OutputStreamWriter(os);
osw.write(msg);
osw.flush();
osw.close();
System.out.println("After flushing output stream. ");
// any response?
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ( (line = br.readLine()) != null)
{
System.out.println("line: " + line);
}
} catch (Throwable t)
{
t.printStackTrace();
}
}
}
原文转自:http://www.cnblogs.com/syncg/archive/2011/04/07/2008242.html