前言:
爬虫框架scrapy学习笔记,练手-爬取丑事百科
参考有趣的教程
配置
conda 安装
conda install -c conda-forge scrapy
pip install Scrap
scrapy依赖一些相关的库:
lxml
parsel
w3lib
twisted
cryptography and pyOpenSSL
生成初始化项目文件:
cmd 到指定目录:
d:
cd Python/
scrapy startproject qiushibaike
实现代码:
具体问题具体分析:
与教材的地址明显不一样了。
在spiders文件下:
import scrapy
class QiushiSpider(scrapy.Spider):
name = "qiushibaike"
haders = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36'
}
def start_requests(self):
urls = [
'https://www.qiushibaike.com/text/page/12/',
'https://www.qiushibaike.com/text/page/2/',
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse, headers=self.haders)
def parse(self, response):
content_left_div = response.xpath('//div[@class="article block untagged mb15 typs_long"]')
for content_div in content_left_div:
yield {
'author': content_div.xpath('//div[@class="author clearfix"]/a[2]/h2/text()').get(),
'content': content_div.xpath('//a[@class="contentHerf"]/div/span/text()').getall(),
}