PHP CURL抓取重定向页面内容

最新推荐文章于 2021-04-13 05:28:18 发布

保罗森

最新推荐文章于 2021-04-13 05:28:18 发布

阅读量2k

点赞数 1

分类专栏： PHP

本文链接：https://blog.csdn.net/slyjit/article/details/106501900

版权

PHP 专栏收录该内容

75 篇文章 1 订阅

订阅专栏

抓取某页面原始地址为：

http://store.xx.com/shop/view_shop.htm?user_number_id=692420117

初始curl请求方法：

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url); 
$ret = curl_exec($ch); 
$info = curl_getinfo($ch);

结果直接是301

后来直接访问地址发现会重定向到https地址修改为

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url); 
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_MAXREDIRS,20);        //设置最大的重定向次数
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);  //跟随重定向
$ret = curl_exec($ch); 
$info = curl_getinfo($ch);

发现页面直接输出，无法对返回结果进行操作

最后修改为

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url); 
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_AUTOREFERER, true); //自动设置header中的referer信息
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); //将数据保留到返回结果中 而非直接输出
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); //跟随重定向
$ret = curl_exec($ch); 
$info = curl_getinfo($ch); 
curl_close($ch);

/**
对$ret操作即可
**/

PS：如果要抓取的页面有很多重定向则可以先只抓取头部，直至状态码为200状态，获取$ret里面的location地址然后抓取最后一次location页面的内容即可，可以提高速度；也可以直接调用$info['url'] 也是可以的

curl_setopt($ch, CURLOPT_HEADER, TRUE);//抓取头部
curl_setopt($ch, CURLOPT_NOBODY, TRUE);//不抓取内容

$info内容如下：

Array ( 
[url] => https://yy.tmall.com/shop/view_shop.htm?user_number_id=xxx
[content_type] => text/html;charset=GBK 
[http_code] => 200 
[header_size] => 3394 
[request_size] => 396 
[filetime] => -1 
[ssl_verify_result] => 20 
[redirect_count] => 3 
[total_time] => 0.640332 
[namelookup_time] => 0.051478 
[connect_time] => 0.20533 
[pretransfer_time] => 0.470629 
[size_upload] => 0 
。。。
[redirect_time_us] => 444840 
[starttransfer_time_us] => 639978 
[total_time_us] => 640332 
)

$ret内容如下几次跳转都有很清晰的记录

HTTP/1.1 301 Moved Permanently
Date: Tue, 02 Jun 2020 10:17:02 GMT 
Content-Type: text/html 
Content-Length: 357 
Connection: keep-alive 
ufe-result: A6 
Location: https://store.xx.com/shop/view_shop.htm?user_number_id=xxx 
Server: Tengine/Aserver 
EagleEye-TraceId: 0b5218fa15910930221194685ed5bf 
Timing-Allow-Origin: * 

HTTP/2 302 
date: Tue, 02 Jun 2020 10:17:02 GMT 
content-type: text/html; charset=GBK 
content-length: 337 
location: https://shopXX.tb.com/shop/view_shop.htm?user_number_id=xxx 
ufe-result: A6 
set-cookie: cookie2=12012d6326c0ad6f4809f28835591ba9; Domain=.taobao.com; Path=/; HttpOnly
Timing-Allow-Origin: * 

HTTP/2 302 
date: Tue, 02 Jun 2020 10:17:02 GMT 
content-type: text/html; charset=GBK 
content-length: 337 
location: https://xx.tm.com/shop/view_shop.htm?user_number_id=xx 
ufe-result: A6 
url-hash: http://shopXX.tb.com/index.htm 
set-cookie: thw=cn; Path=/; Domain=.taobao.com; Expires=Wed, 02-Jun-21 10:17:02 GMT;
timing-allow-origin: * 

HTTP/2 200 
date: Tue, 02 Jun 2020 10:17:02 GMT 
content-type: text/html;charset=GBK 
vary: Accept-Encoding 
ufe-result: A6 
url-hash: http://xx.tm.com/index.htm 
eagleeye-traceid: 0b5205c015910930226692698e6f4d  
timing-allow-origin: *