http://blog.csdn.net/syhd142/article/details/8525603
用php的curl抓取网页遇到了问题,为阐述方便,将代码简化如下:
- <?php
- function curlGet($url) {
- $ch = curl_init();
- curl_setopt($ch, CURLOPT_URL, $url);
- curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
- curl_setopt($ch, CURLOPT_HEADER, true);
- return curl_exec($ch);
- }
- $url = 'http://144go.com';
- echo curlGet($url), "\n";
执行上述代码,得到的结果:
- HTTP/1.1 301 Moved Permanently
- Content-Length: 144
- Content-Type: text/html
- Location: http://www.144go.com/
- Server: Microsoft-IIS/6.0
- X-Powered-By: ASP.NET
- Date: Mon, 03 Sep 2012 04:25:22 GMT
- <head><title>Document Moved</title></head>
- <body><h1>Object Moved</h1>This document may be found <a HREF="http://www.144go.com/">here</a></body>
Location: http://www.144go.com/
可知 http://144go.com 被重定向到了 http://www.144go.com/
怎么办呢,要用正则分析出Location部分的链接,重复执行执行curlGet吗?行到是行,就是有点麻烦。
其实只要加一条语就可以了:
- <?php
- function curlGet($url) {
- $ch = curl_init();
- curl_setopt($ch, CURLOPT_URL, $url);
- curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
- curl_setopt($ch, CURLOPT_HEADER, true);
- //函数中加入下面这条语句
- curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
- return curl_exec($ch);
- }
CURLOPT_FOLLOWLOCATION指明:
让curl递归的抓取http头中Location中指明的url。
当抓取次数超过CURLOPT_MAXREDIRS时,递归将终止。
在抓取中任何跳转带来的问题,都可通过设置此参数解决。