求助爬虫大佬

最新推荐文章于 2024-07-12 16:42:46 发布

＇半世恋

最新推荐文章于 2024-07-12 16:42:46 发布

阅读量550

点赞数

文章标签：爬虫 python

本文链接：https://blog.csdn.net/qq_43620157/article/details/129772467

版权

2023-3-25 21:00
在一次心血来潮下,随便写了个脚本访问了下百度.

代码如下:


import urllib.request
import urllib.parse

import requests
from lxml import etree

coding = 'utf-8'

url = 'https://www.baidu.com/s?'
# url = 'http://www.baidu.com/s?'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'
}

params = {
    'ie': coding,
    'wd': '随便查'
}

resp = requests.get(url=url, params=params, headers=headers)

resp.encoding = coding

print(resp.text)

print('---------分隔符----------')

resp = urllib.request.urlopen(urllib.request.Request(url + urllib.parse.urlencode(params), headers=headers))

print(resp.read().decode(coding))

按照正常流程走,这样的请求和返回是没有问题的.期间试了各种头信息

比如说 Accept, Connection, Referer 等等.

都是返回:

<!DOCTYPE html>
<html lang="zh-CN">
<head>
    <meta charset="utf-8">
    <title>百度安全验证</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <meta name="apple-mobile-web-app-capable" content="yes">
    <meta name="apple-mobile-web-app-status-bar-style" content="black">
    <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0">
    <meta name="format-detection" content="telephone=no, email=no">
    <link rel="shortcut icon" href="https://www.baidu.com/favicon.ico" type="image/x-icon">
    <link rel="icon" sizes="any" mask href="https://www.baidu.com/img/baidu.svg">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta http-equiv="Content-Security-Policy" content="upgrade-insecure-requests">
    <link rel="stylesheet" href="https://ppui-static-wap.cdn.bcebos.com/static/touch/css/api/mkdjump_aac6df1.css" />
</head>
<body>
    <div class="timeout hide-callback">
        <div class="timeout-img"></div>
        <div class="timeout-title">网络不给力，请稍后重试</div>
        <button type="button" class="timeout-button">返回首页</button>
    </div>
    <div class="timeout-feedback hide-callback">
        <div class="timeout-feedback-icon"></div>
        <p class="timeout-feedback-title">问题反馈</p>
    </div>

<script src="https://ppui-static-wap.cdn.bcebos.com/static/touch/js/mkdjump_v2_2d634b8.js"></script>
</body>
</html>

header头内容中加入host参数,不报反扒页面了.但是也没有返回了

headers = {
	'Host': 'www.baidu.com',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'
}