1. 定义一个抽象基类Catcher用来指定抽象行为:
#!/usr/bin/env python
# coding = utf-8
import urllib
import urllib2
import csv
import time
from abc import ABCMeta, abstractmethod
class Catcher(object):
__metaclass__ = ABCMeta
def __init__(self):
print "I'm catcher."
#@abstractmethod
#def setMetaData(self): pass
@abstractmethod
def setDateDict(self): pass
@abstractmethod
def setCinemaDict(self): pass
@abstractmethod
def dateDictFilter(self): pass
@abstractmethod
def cinemaDictFilter(self): pass
@abstractmethod
def getMovieList(self): pass
@abstractmethod
def saveMovieList(self): pass
@abstractmethod
def execute(self): pass
#dateDict = getDateDict()
#dateDict = dateDictFilter()
#cinemaDict = getCinemaDict()
#cinemaDict = cinemaDictFilter()
#getMovieList()
#saveMovieList()
@staticmethod
def getCurrentDate():
return time.strftime('%Y-%m-%d',time.localtime(time.time()))
@staticmethod
def post(url, data):
req = urllib2.Request(url)
data = urllib.urlencode(data)
#enable cookie
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
response = opener.open(req, data)
return response.read()
@staticmethod
def createCSVFile(fileName, fileContent):
csvfile = file(fileName, 'wb')
writer = csv.writer(csvfile)
writer.writerows(fileContent)
csvfile.close()
方法名 | 方法类型 | 功能 |
---|---|---|
getCurrentDate | static | 根据格式获取当前系统日期 |
post | static | 使用urllib2,提交data参数指定的表单数据 到参数url指定的地址 |
createCSVFile | static | 创建csv文件,各个字段用逗号分隔 |
setDateDict | instance | 设置所有需要提取影讯的日期,存放在成员变量,dataDict中 |
setCinemaDict | instance | 设置所有需要提取影讯的影院,存放在成员变量,cinemaDict中 |
dateDictFilter | instance | 过滤影讯日期,有些不需要的,可以剔除 |
cinemaDictFilter | instance | 过滤影院,有些不需要的,可以剔除 |
getMovieList | instance | 获取影片列表 |
saveMovieList | instance | 存储影片列表 |
execute | instance | 执行抓取过程 |
2. 具体负责抓取太平洋影城影讯的类PacificCatcher,比基类多定义了2个方法:setMetaData, getOptionDictBySelectClass;
setMetaData负责提取页面内容:http://www.sztpy.com.cn/zh-CN/products009.html;
getOptionDictBySelectClass负责根据setMetaData获取的dateDict和cinemaDict;
getMovieList和saveMovieList也实现了具体的业务逻辑。
在该Python文件末尾定义了PacificCatcher类实例,并执行了execute方法:
pacificCatcher = PacificCatcher("D:\\pacific\\", "http://www.sztpy.com.cn/zh-CN/products009.html")
pacificCatcher.execute()
#!/usr/bin/env python
# coding = utf-8
from pyquery import PyQuery as pyq
from Catcher import Catcher
class PacificCatcher(Catcher):
def __init__(self, savePath, targetUrl):
self.savePath = savePath
self.targetUrl = targetUrl
self.currentDate = Catcher.getCurrentDate('%Y-%m-%d')
self.metaData = ''
def setMetaData(self):
self.metaData = pyq(url=self.targetUrl)
def getOptionDictBySelectClass(self, className):
resultDict = {}
for select in self.metaData('select'):
tip = select.values()[0]
if tip == className:
options = select.findall('option')
if len(options) < 1:
continue;
else:
for i in range(0, len(options)):
if len(options[i].values()) == 1 and len(options[i].values()[0]) > 1:
resultDict[options[i].values()[0]] = options[i].text
break
return resultDict
def setDateDict(self):
self.dateDict = self.getOptionDictBySelectClass('search_secle')
def setCinemaDict(self):
self.cinemaDict = self.getOptionDictBySelectClass('search_secle5')
def dateDictFilter(self):
count = 0
keys = self.dateDict.keys()
keys.sort()
for k in keys:
if(self.dateDict[k] >= self.currentDate and count < 3):
count += 1
else:
del self.dateDict[k]
def cinemaDictFilter(self): pass
def getMovieList(self, movieHtml, date, cinema):
fileContent = []
for table in movieHtml('table'):
trs = table.findall('tr')
count = len(trs)
if(count > 1):
for i in range(1, count):
tds = trs[i].findall('td')
tdLength = len(tds)
if tdLength == 7:
room = tds[0].text
movieTime = tds[1].findall('span')[0].text
name = tds[2].text
country = tds[3].text
language = tds[4].text
price = tds[5].findall('span')[0].text
wmv = tds[6].text
content = [date, cinema.encode('utf-8'), room.encode('utf-8'), movieTime, name.encode('utf-8'), country.encode('utf-8'), language.encode('utf-8'), price.encode('utf-8'), wmv.encode('utf-8')]
fileContent.append(content)
return fileContent
def saveMovieList(self):
contents = {}
postDataDict = {'searchType':'0'}
for dateId,date in self.dateDict.items():
fileContent = []
for cinemaId, cinema in self.cinemaDict.items():
postDataDict['typeid1'] = dateId
postDataDict['proTypeID'] = cinemaId
content = self.getMovieList(pyq(Catcher.post(self.targetUrl, postDataDict)), date, cinema)
if(len(content) > 1):
fileContent.extend(content)
if(len(fileContent) > 1):
contents[date] = fileContent
for date, fileContent in contents.items():
Catcher.createCSVFile(self.savePath + "\\" + date + ".csv", fileContent)
def execute(self):
self.setMetaData()
self.setDateDict()
self.dateDictFilter()
self.setCinemaDict()
self.saveMovieList()
pacificCatcher = PacificCatcher("D:\\pacific\\csv\\", "http://www.sztpy.com.cn/zh-CN/products009.html")
pacificCatcher.execute()
3. 结果展示:
打开dos,执行:tree /F d:\pacific
返回结果:
D:\PACIFIC
└─csv
2016-05-21.csv
2016-05-22.csv
打开2016-05-21.csv,可以看到csv文件内容如下(截取片段):
2016-05-21,深圳天利名城店,激光1号厅,11:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光1号厅,11:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光1号厅,11:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光1号厅,11:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光1号厅,11:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光2号厅,10:30,愤怒的小鸟(3D),芬兰,英语,80元,97分钟
2016-05-21,深圳天利名城店,激光2号厅,10:30,愤怒的小鸟(3D),芬兰,英语,80元,97分钟
2016-05-21,深圳天利名城店,激光2号厅,10:30,愤怒的小鸟(3D),芬兰,英语,80元,97分钟
2016-05-21,深圳天利名城店,激光2号厅,10:30,愤怒的小鸟(3D),芬兰,英语,80元,97分钟
2016-05-21,深圳天利名城店,激光2号厅,10:30,愤怒的小鸟(3D),芬兰,英语,80元,97分钟
2016-05-21,深圳天利名城店,激光2号厅,10:30,愤怒的小鸟(3D),芬兰,英语,80元,97分钟
2016-05-21,深圳天利名城店,激光2号厅,10:30,愤怒的小鸟(3D),芬兰,英语,80元,97分钟
2016-05-21,深圳天利名城店,激光3号厅,11:20,愤怒的小鸟(3D),芬兰,国语,80元,97分钟
2016-05-21,深圳天利名城店,激光3号厅,11:20,愤怒的小鸟(3D),芬兰,国语,80元,97分钟
2016-05-21,深圳天利名城店,激光3号厅,11:20,愤怒的小鸟(3D),芬兰,国语,80元,97分钟
2016-05-21,深圳天利名城店,激光3号厅,11:20,愤怒的小鸟(3D),芬兰,国语,80元,97分钟
2016-05-21,深圳天利名城店,激光3号厅,11:20,愤怒的小鸟(3D),芬兰,国语,80元,97分钟
2016-05-21,深圳天利名城店,激光3号厅,11:20,愤怒的小鸟(3D),芬兰,国语,80元,97分钟
2016-05-21,深圳天利名城店,激光5号厅,12:00,愤怒的小鸟(3D),芬兰,国语,100元,97分钟
2016-05-21,深圳天利名城店,激光5号厅,12:00,愤怒的小鸟(3D),芬兰,国语,100元,97分钟
2016-05-21,深圳天利名城店,激光5号厅,12:00,愤怒的小鸟(3D),芬兰,国语,100元,97分钟
2016-05-21,深圳天利名城店,激光5号厅,12:00,愤怒的小鸟(3D),芬兰,国语,100元,97分钟
2016-05-21,深圳天利名城店,激光5号厅,12:00,愤怒的小鸟(3D),芬兰,国语,100元,97分钟
2016-05-21,深圳天利名城店,激光5号厅,12:00,愤怒的小鸟(3D),芬兰,国语,100元,97分钟
2016-05-21,深圳天利名城店,激光6号厅,12:10,分歧者3:忠诚世界,美国,英语,80元,120分钟
2016-05-21,深圳天利名城店,激光6号厅,12:10,分歧者3:忠诚世界,美国,英语,80元,120分钟
2016-05-21,深圳天利名城店,激光6号厅,12:10,分歧者3:忠诚世界,美国,英语,80元,120分钟
2016-05-21,深圳天利名城店,激光6号厅,12:10,分歧者3:忠诚世界,美国,英语,80元,120分钟
2016-05-21,深圳天利名城店,激光6号厅,12:10,分歧者3:忠诚世界,美国,英语,80元,120分钟
2016-05-21,深圳天利名城店,激光7号厅,10:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光7号厅,10:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光7号厅,10:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光7号厅,10:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光7号厅,10:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
2016-05-21,深圳天利名城店,激光7号厅,10:40,分歧者3:忠诚世界,美国,英语,40元,120分钟
4. 代码需要优化的还有很多,比如:
需要添加各种异常的处理,
http请求需要添加各种返回码的处理