爬取站大爷的免费ip代理

最新推荐文章于 2024-05-28 16:35:37 发布

liuy5277

最新推荐文章于 2024-05-28 16:35:37 发布

阅读量3w

点赞数 1

分类专栏： python 文章标签： python 爬虫

本文链接：https://blog.csdn.net/liuy5277/article/details/119429161

版权

python 专栏收录该内容

8 篇文章 0 订阅

订阅专栏

该博客展示了如何使用Python的requests和lxml库从指定网站抓取免费代理IP，并进行整理。通过设置User-Agent避免被检测，遍历多页获取IP和端口，最后将结果保存为HTTP和HTTPS代理。

摘要由CSDN通过智能技术生成

“”"
Created by linuxdba at 2021/8/2
mail: linuxdba@qq.com
“”"
import requests
from lxml import etree

base_url = ‘https://www.zdaye.com’
url = ‘https://www.zdaye.com/dayProxy.html’
header = {
‘User-Agent’: ‘Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Mobile Safari/537.36’}

proxies_list = []

res = requests.get(url, headers=header)
res.encoding = ‘utf-8’
dom = etree.HTML(res.text)
sub_urls = dom.xpath(’//h3[@class=“thread_title”]/a/@href’)

sub_pages = []
for sub_url in sub_urls:
# print(base_url + sub_url)
for i in range(1, 11):
sub_page = (base_url + sub_url).rstrip(’.html’) + ‘/’ + str(i) + ‘.html’
sub_pages.append(sub_page)

#不要刷太多页（否则会被检测到立马封ip），或者获取优化获取策略防屏蔽
sub_res = requests.get(sub_pages[0], headers=header)
sub_res.encoding = ‘utf-8’
sub_dom = etree.HTML(sub_res.text)
ips = sub_dom.xpath(’//tbody/tr/td[1]/text()’)
ports = sub_dom.xpath(’//tbody/tr/td[2]/text()’)

proxies_list = []

for ip, port in zip(ips, ports):
proxies = {}
http = ‘http://’ + ip + ‘:’ + port
https = ‘https://’ + ip + ‘:’ + port
proxies[‘http’] = http
proxies[‘https’] = https
proxies_list.append(proxies)

print(proxies_list)

效果如下：
在这里插入图片描述