Scrapy框架爬取dmoztools网站两个网址信息——基于Xpath内置选择器

问题背景

目的是将该网站关于Python的书籍信息和资源信息爬取并导出。

实现过程

一、调出cmd窗口

cd /d  所要保存的路径
#t2为新建的文件名
scrapy startproject t2
cd q2
scrapy genspider t2t dmoztools.net
#接下来可直接在cmd窗口中进行测试，也可以到Python编辑器里面进行测试
cd t2
scrapy shell http://dmoztools.net/Computers/Programming/Languages/Python/Books/
response.body

二、在PyCharm中打开q2文件
在Pycharm中进行测试

#在Pycharm下方的Terminal框中
cd /d 所要保存的路径\t2\t2
scrapy shell http://dmoztools.net/Computers/Programming/Languages/Python/Books/
response.body

在t2的spiders文件下打开q2q文件，补充相关代码：

# -*- coding: utf-8 -*-
import scrapy

from t2.items import T2Item

class T2tSpider(scrapy.Spider):
    name = "t2t"
    allowed_domains = ["dmoztools.net"]
    start_urls = ['http://dmoztools.net/Computers/Programming/Languages/Python/Books/',
                  'http://dmoztools.net/Computers/Programming/Languages/Python/Resources/']
    def parse(self, response):
        # sel = scrapy.selector.Selector(response)
        # sites=sel.xpath('//*[@id="site-list-content"]/div')
        sites=response.xpath('//*[@id="site-list-content"]/div')
        items = []
        for site in sites:
            item = T2Item()
            item['title']=site.xpath('div/a/div/text()').extract()[0]
            item['link']=site.xpath('div/a/@href').extract()[1]
            item['desc']=site.xpath('div/div/text()').extract()[0]
            items.append(item)
        return items

打开t2下的items文件，补充相关代码：

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class T2Item(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()