网络爬虫中动态的获取页面编码

动态获取页面编码有两种实现方式：根据下载的网页源代码的URL地址来获取 或者是 根据下载的网页源代码进行判断页面的编码。

这两种方式都用到了第三方的库：antlr.jar、chardet.jar和cpdetector.jar

第一种方式：

<span style="font-family:微软雅黑, Microsoft Yahei, Helvetica Neue, Hiragino Sans GB, 宋体, simsun, 黑体, Arial, sans-serif;">
</span><pre name="code" class="java" style="font-size: 16px;"><span style="font-family:Times New Roman;">public String getURLEncoding(String url) {
  java.nio.charset.Charset charset = null;</span>

<div style="word-wrap: break-word; line-height: 24px;"><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="font-family:Times New Roman;font-size:12px;"><span style="white-space: pre;"> </span>CodepageDetectorProxy detector = CodepageDetectorProxy.getInstance();</span></span></div></div><div style="word-wrap: break-word; line-height: 24px;"><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="white-space: pre;"><span style="font-family:Times New Roman;font-size:12px;"> //<span style="line-height: 24px;">ParsingDetector可用于检查HTML、XML等文件或字符流的编码,构造方法中的参数用于指示是否显示探测过程的详细信息，为false不显示。</span></span></span></span></div></div><div style="word-wrap: break-word; line-height: 24px;"><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="font-family:Times New Roman;font-size:12px;"><span style="white-space: pre;"> </span>detector.add(new ParsingDetector(false));</span></span></div><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="font-family:Times New Roman;"><span style="line-height: 24px;"><span style="font-size: 16px;">//JC</span><span style="font-size:12px;">hardetFacade封装了由Mozilla组织提供的JChardet，它可以完成大多数文件的编码</span></span><span style="font-size:12px;">
</span></span></span></div></div><div style="word-wrap: break-word; line-height: 24px;"><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="font-family:Times New Roman;font-size:12px;"><span style="white-space: pre;"> </span>detector.add(JChardetFacade.getInstance());</span></span></div></div><div style="word-wrap: break-word; line-height: 24px;"><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="white-space: pre;"><span style="font-family:Times New Roman;font-size:12px;"> //添加ASCII编码探测器</span></span></span></div></div><div style="word-wrap: break-word; line-height: 24px;"><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="font-family:Times New Roman;font-size:12px;"><span style="white-space: pre;"> </span>detector.add(ASCIIDetector.getInstance());</span></span></div></div><div style="word-wrap: break-word; line-height: 24px;"><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="white-space: pre;"><span style="font-family:Times New Roman;font-size:12px;">//添加Unicode编码探测器 </span></span></span></div></div><div style="word-wrap: break-word; line-height: 24px;"><div style="word-wrap: break-word;"><span style="line-height: 1.5;"><span style="font-family:Times New Roman;font-size:12px;"><span style="white-space: pre;"> </span>detector.add(UnicodeDetector.getInstance());</span></span></div></div><span style="font-size: 16px;"><span style="font-family:Times New Roman;">
 try {
 charset = detector.detectCodepage(new URL(url));
 } catch (Exception ex) {
 ex.printStackTrace();
 }
 if (charset != null){
 System.out.println(" charset--name:"+charset.name());
 return charset.name();
 }
 else
 return null;
 }</span><span style="font-family:微软雅黑, Microsoft Yahei, Helvetica Neue, Hiragino Sans GB, 宋体, simsun, 黑体, Arial, sans-serif;">
</span></span>

第二种方式：

<span style="font-family:Times New Roman;">public String getFileEncode(String path) {
 
  CodepageDetectorProxy detector = CodepageDetectorProxy.getInstance();
 detector.add(new ParsingDetector(false));
  detector.add(JChardetFacade.getInstance());
 detector.add(ASCIIDetector.getInstance());
 detector.add(UnicodeDetector.getInstance());
  java.nio.charset.Charset charset = null;
 File f = new File(path);
 try {
 charset = detector.detectCodepage(f.toURI().toURL());
 } catch (Exception ex) {
 ex.printStackTrace();
 }
 if (charset != null)
 return charset.name();
 else
 return null;
  }</span>

还有一种底层的实现方式就是根据各种字符编码的规则来判断各种字符编码，例如对于UTF-8编码格式的文本文件，其前3个字节的值就是-17、-69、-65，只需要将所判断文本转化为byte数组，判断前三个字节即可。

以上是三种网络爬虫中常用的页面编码获取方法，当然在正规的网站中首先查看

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />中的charset属性的值，此即为网页的编码。