scrapy爬虫框架 (2. logging模块的使用、yield scrapy.Request()函数间传参)

1.logging模块的使用

1.1scrapy项目中的使用

1.settings.py中设置LOG_LEVEL=“WARNING”
2.settings.py中设置LOG_FILE="./log.log" #这是日志保存的位置，设置后终端就不会显示日志内容
3.程序里进行日志输出：

import logging
logger=logging.getLogger(__name__) #实例化logger
class spider(scrapy.Spider):
	....
	...
	def parse(self,response):
		...
		...
		logger.warning(item)

1.2普通项目

import logging
logging.basicConfig(…) #设置日志格式
logger=logging.getLogger(name) #实例化logger
在任何py文件中调用logger即可

2.同一函数间传数据（常用于翻页请求）

翻页请求：
1.找到下一页地址
2.组装成一个Request对象交给引擎

2.1首先爬取腾讯课堂第一页的代码如下

import scrapy


class NextpageSpider(scrapy.Spider):
    name = 'NextPage'
    allowed_domains = ['qq.com']
    start_urls = ['https://ke.qq.com/course/list?mt=1001&st=2002&tt=3019&price_min=1&page=1']

    def parse(self, response):
        datalist=response.xpath("//div[@class='market-bd market-bd-6 course-list course-card-list-multi-wrap js-course-list']//li")
        for data in datalist:
            item={}
            if data.xpath(".//h4//a/text()").extract_first() !=None:
                item["name"] = data.xpath(".//h4//a/text()").extract_first()
                item["price"] = data.xpath(".//span[@class='line-cell item-price  custom-string']/text()").extract_first()
                item["number"] = data.xpath("normalize-space(.//span[@class='line-cell item-user custom-string']/text())").extract_first()
                print(item)

2.2找到下页的路径，并且注意最后一页的下页路径和第一页的下页路径不一样，根据这个来决定停止

第一页时下页的url地址：

最后一页时的下页地址：

根据这个url地址就可以通过yield构造Request对象传给引擎

# -*- coding: utf-8 -*-
import scrapy


class NextpageSpider(scrapy.Spider):
    name = 'NextPage'
    allowed_domains = ['qq.com']
    start_urls = ['https://ke.qq.com/course/list?mt=1001&st=2002&tt=3019&price_min=1&page=1']

    def parse(self, response):
        datalist=response.xpath("//div[@class='market-bd market-bd-6 course-list course-card-list-multi-wrap js-course-list']//li")
        for data in datalist:
            item={}
            if data.xpath(".//h4//a/text()").extract_first() !=None:
                item["name"] = data.xpath(".//h4//a/text()").extract_first()
                item["price"] = data.xpath(".//span[@class='line-cell item-price  custom-string']/text()").extract_first()
                item["number"] = data.xpath("normalize-space(.//span[@class='line-cell item-user custom-string']/text())").extract_first()
                print(item)
        #找到下一页的url地址
        next_url=response.xpath("//a[@class='page-next-btn icon-font i-v-right']/@href").extract_first()
        if next_url != "javascript:void(0);" and next_url != None:
            #通过yield把地址包装成Request请求传向引擎
            yield scrapy.Request(next_url,callback=self.parse)
            #也可以是return scrapy.Request(next_url,callback=self.parse);因为return是将请求传回引擎，引擎会自动判断，如果是请求会再次爬取，是数据就会进行处理。
        else:
            print("end")

3.不同函数间传参

# -*- coding: utf-8 -*-
import scrapy


class NextpageSpider(scrapy.Spider):
    name = 'NextPage'
    allowed_domains = ['qq.com']
    start_urls = ['https://ke.qq.com/course/list?mt=1001&st=2002&tt=3019&price_min=1&page=1']

    def parse(self, response):
        datalist=response.xpath("//div[@class='market-bd market-bd-6 course-list course-card-list-multi-wrap js-course-list']//li")
        for data in datalist:
            item={}
            if data.xpath(".//h4//a/text()").extract_first() !=None:
                item["name"] = data.xpath(".//h4//a/text()").extract_first()
                item["price"] = data.xpath(".//span[@class='line-cell item-price  custom-string']/text()").extract_first()
                item["number"] = data.xpath("normalize-space(.//span[@class='line-cell item-user custom-string']/text())").extract_first()
            	yield scrapy.Request(next_url,callback=self.parse2,meta={"item":item}) ##把数据以字典形式通过meta参数传递给parse2函数
            	
     def parse2(self,response):
     	item=response.meta["item"]   ##取出数据

4.scrapy.Request知识点：