![](https://img-blog.csdnimg.cn/20201014180756923.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
Spider_Practise
文章平均质量分 57
Z0o1010
这个作者很懒,什么都没留下…
展开
-
爬取某商城网站所有订单标题
from lxml import etreeimport requestsif __name__ == '__main__': for i in range(1, 3425): url = f"http://www.hebzfcgwssc.com/Mall/HeBei/gonggaoinfo.aspx?page={i}&danweiname=&citytype=" headers = { "User-Agent": "Mo原创 2021-10-30 13:38:41 · 183 阅读 · 0 评论 -
爬取百度热榜新闻概括
import requestsfrom bs4 import BeautifulSoupdef get_html(url, headers): r = requests.get(url, headers=headers) r.encoding = r.apparent_encoding return r.textdef get_pages(html): global s soup = BeautifulSoup(html, 'html.parser')原创 2020-12-17 09:54:47 · 149 阅读 · 0 评论 -
使用 aiohttp 进行异步爬取
import asyncioimport timeimport aiohttpst = time.time()urls = ["https://www.baidu.com/", "https://www.baidu.com/"]async def request(url): async with aiohttp.ClientSession() as session: # get(), post() # proxy="http://ip:port"原创 2020-11-09 14:19:34 · 230 阅读 · 0 评论 -
使用 xpath 爬取当前页面所有城市名称
from lxml import etreeimport requestsif __name__ == '__main__': url = "https://www.aqistudy.cn/historydata/" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/"原创 2020-10-28 09:24:04 · 267 阅读 · 0 评论 -
xpath 爬取某网站图片
from lxml import etreeimport requestsimport osif __name__ == '__main__': url = "http://pic.netbian.com/4kmeinv/" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/"原创 2020-10-27 22:33:21 · 154564 阅读 · 4 评论 -
xpath 爬取某网站二手房简介及价格
from lxml import etreeimport requestsif __name__ == '__main__': url = "https://bj.58.com/ershoufang/" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/"原创 2020-10-27 21:15:40 · 146 阅读 · 0 评论 -
bs4 爬取某网站三国演义标题及内容
from bs4 import BeautifulSoupimport requestsif __name__ == '__main__': url = "https://www.shicimingju.com/book/sanguoyanyi.html" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)原创 2020-10-24 16:57:55 · 240 阅读 · 0 评论 -
简易网页采集器
import requests# UA 检测:门户网站的服务器会检测对应请求的载体身份标识,若检测到请求的载体身份标识为某款浏览器,则说明为正常请求# 反之,则为爬虫,服务器段可能会拒绝此次请求# UA 伪装:让爬虫对应的请求载体身份标识伪装成某款浏览器if __name__ == '__main__': # UA 伪装:将对应的 User-Agent 封装到字典中 headers = { "User-Agent": "Mozilla/5.0 (W原创 2020-10-21 14:03:46 · 109 阅读 · 0 评论