1、创建scrapy项目
$ scrapy startproject ArticleSpider
New Scrapy project 'ArticleSpider' , using template directory '/Users/chao/.pyenv/versions/2.7.13/lib/python2.7/site-packages/scrapy/templates/project' , created in : /Users/... /ArticleSpider
You can start your first spider with:
cd ArticleSpider
scrapy genspider example example.com
2、默认模板创建
$ cd ArticleSpider
$ scrapy genspider jobbole blog.jobbole.com
即在spider目录下创建了jobbole.py文件,终端显示:
Created spider 'jobbole' using template 'basic' in module : ArticleSpider .spiders .jobbole
3、修改setting.py
ROBOTSTXT_OBEY = False
4、建立main.py(方便在pycharm上断点调试)
# coding:utf-8
from scrapy.cmdline import execute
import sys
import os
sys.path .append (os.path .dirname (os.path .abspath (__file__)))
#print(os.path.abspath(__file__)) # 自身路径
#print(os.path.dirname(os.path.abspath(__file__))) # 父目录路径
execute(["scrapy" , "crawl" , "jobbole" ])
$ scrapy crawl jobbole
5、调试xpath
$ scrapy shell http: //blog.jobbole.com/ 112558 /
即进入python shell的调试环境,避免每次启动main.py都会解析url而浪费时间。
>>> title = response.xpath("//*[@id='post-112558']/div[1]/h1/text()" ) >>> title [] >>> title.extract()
>>> title.extract()[0 ]
>>> print(title.extract())
6、调试正确的内容写进jobbole.py
import scrapy
import re
import sys
reload(sys)
sys.setdefaultencoding('utf-8' )
class JobboleSpider (scrapy.Spider) :
name = 'jobbole'
allowed_domains = ['blog.jobbole.com' ]
start_urls = ['http://blog.jobbole.com/112558/' ]
def parse (self, response) :
title = response.xpath("//*[@id='post-112558']/div[1]/h1/text()" ).extract()[0 ]
create_data = response.xpath('//*[@id="post-112558"]/div[2]/p/text()' ).extract()[0 ].strip().replace('·' , '' )
praise_nums = int(response.xpath('//span[contains(@class, "href-style vote-post-up")]/h10/text()' ).extract()[0 ])
favor_nums = response.xpath('//span[contains(@class, "bookmark-btn")]/text()' ).extract_first("" )
match_re = re.match(".*?(\d+).*" , favor_nums)
if match_re:
favor_nums = int(match_re.group(1 ))
else :
favor_nums = 0
comment_nums = response.xpath('//*[@id="post-112558"]/div[3]/div[4]/a/span/text()' ).extract()[0 ]
match_re2 = re.match(".*?(\d+).*" , comment_nums)
if match_re2:
comment_nums = int(match_re2.group(1 ))
else :
comment_nums = 0
content = response.xpath('//div[@class="entry"]' ).extract()[0 ]
create_data = response.xpath('//*[@id="post-112558"]/div[2]/p/text()' ).extract()[0 ].strip().replace('·' , '' )
tag_list = response.xpath('//p[@class="entry-meta-hide-on-mobile"]/a/text()' ).extract()
[element for element in tag_list if not element.strip().endswith("评论" )]
tags = "," .join(tag_list)