scrapy的name变量_python-将file_name参数传递给管道以在scrapy中...

我需要从命令行中获取一个自变量(-a FILE_NAME =“ stuff”),并将其应用于在pipeline.py文件中由我的CSVWriterPipeLine创建的文件. (我之所以使用pipeline.py是因为内置的导出器正在重复数据并在输出文件中重复标题.相同的代码,但是在管道中进行写入修复了它.)

我尝试从scrapy.utils.project导入get_project_settings中看到

但是我无法从命令行更改文件名.

我还尝试实现页面上的@avaleske解决方案,因为它专门解决了这个问题,但是我不知道他谈论的代码在我的scrapy文件夹中的位置.

救命？

settings.py：

BOT_NAME = 'internal_links'

SPIDER_MODULES = ['internal_links.spiders']

NEWSPIDER_MODULE = 'internal_links.spiders'

CLOSESPIDER_PAGECOUNT = 100

ITEM_PIPELINES = ['internal_links.pipelines.CsvWriterPipeline']

# Crawl responsibly by identifying yourself (and your website) on the user-agent

USER_AGENT = 'internal_links (+http://www.mycompany.com)'

FILE_NAME = "mytestfilename"

pipelines.py：

import csv

class CsvWriterPipeline(object):

def __init__(self, file_name):

header = ["URL"]

self.file_name = file_name

self.csvwriter = csv.writer(open(self.file_name, 'wb'))

self.csvwriter.writerow(header)

def process_item(self, item, internallinkspider):

# build your row to export, then export the row

row = [item['url']]

self.csvwriter.writerow(row)

return item

spider.py：

from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

from scrapy.contrib.spiders import CrawlSpider, Rule

from internal_links.items import MyItem

class MySpider(CrawlSpider):

name = 'internallinkspider'

allowed_domains = ['angieslist.com']

start_urls = ['http://www.angieslist.com']

rules = (Rule(SgmlLinkExtractor(), callback='parse_url', follow=True), )

def parse_url(self, response):

item = MyItem()

item['url'] = response.url

return item