Scrapy学习总结(一)——基本命令及使用基础

1. scrapy --help

Scrapy 1.5.0 - project: mingyan

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  check         Check spider contracts
  crawl         Run a spider
  edit          Edit spider
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  list          List available spiders
  parse         Parse URL (using its spider) and print the results
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

Use "scrapy <command> -h" to see more info about a command

2. scrapy startproject xxx 创建一个xxx项目

3. scrapy genspider xxx example.com

会在spider目录下创建一个‘xxx.py’文件，其中的spider的name=‘xxx’

1. name是指spider的名字
2. example.com 是指要爬取的网站的域名
   有两种情况：
   1、在工程中产生一个spider
   2、在同一个工程产生多个spider，不同的spider要求name不同

4. scrapy crawl name 运行工程里的爬虫

5. scrapy runspider xxx.py 运行单个爬虫