爬虫—request设置使用代理ip

最新推荐文章于 2024-09-18 18:31:48 发布

HQ_JSY

最新推荐文章于 2024-09-18 18:31:48 发布

阅读量1.5w

点赞数

分类专栏：爬虫

本文链接：https://blog.csdn.net/JSYhq/article/details/88706278

版权

爬虫专栏收录该内容

14 篇文章 1 订阅

订阅专栏

一、获取代理ip的网站

快代理
西祠代理
www.goubanjia.com

二、代理ip使用

#代理ip
import requests
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36',
}


url = 'https://www.baidu.com/s?wd=ip'
page_text = requests.get(url=url,headers=headers,proxies={'https':'134.209.13.16:8080'}).text
with open('./ip.html','w',encoding='utf-8') as fp:
    fp.write(page_text)


https_proxies = [
    {'https':'134.209.13.16:8080'},
    {'https':'134.209.13.14:8080'},
    {'https':'134.209.13.19:8080'}
]
http_proxies = [
    {'http':'134.209.13.16:8080'},
    {'http':'134.209.13.14:8080'},
    {'http':'134.209.13.19:8080'}
]


#获取请求的url
url = 'https://www.baidu.com/s?wd=ip'
response = requests.get(url=url,headers=headers)
response.url.split(':')[0]