python爬虫scrapy框架爬取网页数据_Python爬虫框架Scrapy 学习笔记 2 ----- 爬取Mininova网站种子文件信息...

1. 任务描述

目标网站截图：

-------------------------------------

可以看到种子文件的列表，这些链接的url可以用正则表达式表示为： /tor/\d+

详情页截图

截图中的：资源名称, 资源大小，和资源描述就是我们要抓取的信息。

2. 定义scrapy item.

mininova.pyimport scrapy

class TorrentItem(scrapy.Item):

url = scrapy.Field()

name = scrapy.Field()

description = scrapy.Field()

size = scrapy.Field()

3. 查看网页源文件，确定我们要抓取的内容的XPath表达式。对XML Path Language不熟悉的话可以参考：http://www.w3.org/TR/xpath/ItemXPath

name//hi/text()

size//div[@id='specifications']/p[2]/text()[2]

descripthin//div[@id='description']

4. 最终的代码为：

mininova.pyimport scrapy

from scrapy.contrib.spiders import CrawlSpider, Rule

from scrapy.contrib.linkextractors import LinkExtractor

class TorrentItem(scrapy.Item):

url = scrapy.Field()

name = scrapy.Field()

description = scrapy.Field()

size = scrapy.Field()

class MininovaSpider(CrawlSpider):

name = 'mininova'

allowed_domains = ['mininova.org']

start_urls = ['http://www.mininova.org/yesterday']

rules = [Rule(LinkExtractor(allow=['/tor/\d+']), 'parse_torrent')]

def parse_torrent(self, response):

torrent = TorrentItem()

torrent['url'] = response.url

torrent['name'] = response.xpath("//h1/text()").extract()

torrent['description'] = response.xpath("//div[@id='description']").extract()

torrent['size'] = response.xpath("//div[@id='specifications']/p[2]/text()[2]").extract()

return torrent

5. 运行代码

将mininova.py复制到C:\

打开cmd，输入命令： scrapy runspider --output=spider_out.json mininova.py

文件的后缀名很重要，scrapy会根据后缀名确定输出格式

6. 查看输出