python 爬虫（二）使用代理模拟浏览器

最新推荐文章于 2023-06-08 08:40:34 发布

sun-zhu

最新推荐文章于 2023-06-08 08:40:34 发布

阅读量454

点赞数

分类专栏： python 文章标签： python 爬虫代理防封IP 模拟浏览器

本文链接：https://blog.csdn.net/weixin_40938748/article/details/85276557

版权

python 专栏收录该内容

24 篇文章 0 订阅

订阅专栏

import urllib.request
import random

url = "http://www.baidu.com"

#建立一个代理列表，每次随机使用一个，防止被封IP
agnetsList = [
              "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5"
              "Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5"
              "Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"
              "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 UBrowser/4.0.3214.0 Safari/537.36"
              "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
              "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0"
              ]

#模拟请求头
headerStr = random.choice(agnetsList)
#header ={"User-Agent":headerStr}

#设置一个请求体
req = urllib.request.Request(url)
req.add_header("User-Agent",headerStr)

#发起请求
response = urllib.request.urlopen(req)

data = response.read().decode("utf-8")
print(data)