python中scrapy shell的使用和自定义配置文件

scrapy shell

Scrapy shell是⼀个交互终端，我们可以在未启动spider的情况下,尝试及调试代
码，也可以⽤来测试XPath表达式

使⽤⽅法：

scrapy shell 需要调试的网站

>>> shelp()
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x00000187FCC2A1C0>
[s]   item       {}
[s]   request    <GET http://www.baidu.com>
[s]   response   <200 http://www.baidu.com>
[s]   settings   <scrapy.settings.Settings object at 0x00000187FCC28D90>
[s]   spider     <DefaultSpider 'default' at 0x187fcf6bbe0>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser
>>>

自定义配置文件

settings.py

CUSTOM_VARIABLE = 'I am Custom variable'

管道文件:pipelines.py

 #方式1
from tencent.settings import CUSTOM_VARIABLE

class TencentPipeline:
    def process_item(self, item, spider):
        #print('pipelines TencentPipeline process_item')
        #print(item)
        #print(CUSTOM_VARIABLE) #方式1
		print(spider.settings.get('CUSTOM_VARIABLE')) #方式2
        return item

结果:

I am Custom variable
I am Custom variable
I am Custom variable
I am Custom variable
I am Custom variable