Scrapy小案例

Scrapy小案例

创建项目：

scrapy startproject TXmovies

cd TXmovies

scrapy genspider txms v.qq.com

修改setting:

ROBOTSTXT_OBEY = False
 
DOWNLOAD_DELAY = 1
 
DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en',
  'User-Agent':'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36'
}
 
ITEM_PIPELINES = {
   'TXmovies.pipelines.TxmoviesPipeline': 300,
}

修改items:

import scrapy


class TxmoviesItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    name = scrapy.Field()
    description=scrapy.Field()
    pass