嵩天python测验4_0.4嵩天老师python爬虫笔记整理week4

这里对北理爬虫课程第四周内容回顾,本周主要介绍了Scrapy框架。

1.如果官方例子无法运行，可以查看下twisted的版本，如果为17.0以上，那么先删除scrapy,twisted，运行以下两步

conda uninstall scrapy

conda uninstall twisted

接着重新安装scrapy以及twisted 16.6.0版本，运行以下两步

conda install twisted==16.6.0

conda install scrapy --no-deps

conda install scrapy —no-deps的意思是安装scrapy的时候不要安装依赖的twisted，因为conda安装scrapy的时候会让我们将twisted一起安装了。

2.安装完成之后官方的例子可以运行了，但是老师给的例子依旧不行，这里尝试了一下猜测是名字的问题，做了以下修改:自己命名fname

# -*- coding: utf-8 -*-

import scrapy

class DemoSpider(scrapy.Spider):

name = "demo"

#allowed_domains = ["python123.io"]

start_urls = ['http://python123.io/']

def parse(self, response):

#fname=response.url.split('/')[-1]

fname="demo.html"

with open(fname,'wb') as f:

f.write(response.body)

self.log('Saved file %s.' % fname)

完成这步之后老师的第一个例子就可以运行了。

3.第一个例子成功之后尝试了第二个例子，但依旧无法运行，这里参考了一位同学的发帖，对stocks.py文件进行如下修改

# -*- coding: utf-8 -*-

import scrapy

import re

import random

class StocksSpider(scrapy.Spider):

name = "stocks"

start_urls = ['http://quote.eastmoney.com/stocklist.html']

def __init__(self):

self.user_agent_list = [\

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"\

"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",\

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",\

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",\

"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",\

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",\

"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",\

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\

"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",\

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",\

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\

"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",\

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",\

"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"

]

def parse(self, response):

ua = random.choice(self.user_agent_list)#随机抽取User-Agent

headers = {

'Accept-Encoding':'gzip, deflate, sdch, br',

'Accept-Language':'zh-CN,zh;q=0.8',

'Connection':'keep-alive',

'Referer':'https://gupiao.baidu.com/',

'User-Agent':ua

}#构造请求头

for href in response.css('a::attr(href)').extract():

try:

stock = re.findall(r"[s][hz]\d{6}", href)[0]

url = 'https://gupiao.baidu.com/stock/' + stock + '.html'

yield scrapy.Request(url, callback=self.parse_stock,headers=headers)

except:

continue

def parse_stock(self, response):

infoDict = {}

stockInfo = response.css('.stock-bets')

name = stockInfo.css('.bets-name').extract()[0]

keyList = stockInfo.css('dt').extract()

valueList = stockInfo.css('dd').extract()

for i in range(len(keyList)):

key = re.findall(r'>.*', keyList[i])[0][1:-5]

try:

val = re.findall(r'\d+\.?.*', valueList[i])[0][0:-5]

except:

val = '--'

infoDict[key]=val

infoDict.update(

{'股票名称': re.findall('\s.*\(',name)[0].split()[0] + \

re.findall('\>.*\<', name)[0][1:-1]})

yield infoDict

注意修改完成之后settings.py文件一定要修改，我一开始忘记修改了最后就没有文件产生。

参考资料：