python与spider的区别_python – Scraw spider与Scraped items之间的区别

我正在编写一个Scrapy CrawlSpider,它在第一页上读取AD列表,获取一些信息,如列表和AD网址,然后向每个AD网址发出请求以获取其详细信息.

它在测试环境中工作和分页显然很好,但今天试图完成一次运行我意识到在日志中：

抓取3852页(228页/分),刮掉256件(每件15件)

我不理解抓取页面和Scraped项目之间存在巨大差异的原因.任何人都可以帮助我意识到这些物品丢失的地方？

我的蜘蛛代码：

class MySpider(CrawlSpider):

name = "myspider"

allowed_domains = ["myspider.com", "myspider.co"]

start_urls = [

"http://www.myspider.com/offers/myCity/typeOfAd/?search=fast",

]

#Pagination

rules = (

Rule (

SgmlLinkExtractor()

, callback='parse_start_url', follow= True),

)

#1st page

def parse_start_url(self, response):

hxs = HtmlXPathSelector(response)

next_page = hxs.select("//a[@class='pagNext']/@href").extract()

offers = hxs.select("//div[@class='hlist']")

for offer in offers:

myItem = myItem()

myItem['url'] = offer.select('.//span[@class="location"]/a/@href').extract()[0]

myItem['thumb'] = oferta.select('.//div[@class="itemFoto"]/div/a/img/@src').extract()[0]

request = Request(myItem['url'], callback = self.second_page)

request.meta['myItem'] = myItem

yield request

if next_page:

yield Request(next_page[0], callback=self.parse_start_url)

def second_page(self,response):

myItem = response.meta['myItem']

loader = myItemLoader(item=myItem, response=response)

loader.add_xpath('address', '//span[@itemprop="streetAddress"]/text()')

return loader.load_item()