创新项目实训(九)
前言
我们组打算搭建一个国内旅游比价网站,
而我负责的部份是各大订酒店网站的数据获取及整理
主要参考版上的经验分享+自己的修改理解
小白0经验入门记录、边爬边学习ing
有错误或更好的建议都可以指教讨论
马蜂窝网站分析
马蜂窝是动态网站
- 首先ajax呈现页面(ajax.php?iMddId=10099&iAreaId…)
- 各网站的价格再调用poi接口获取(price?poi_id…)
可以看到有6个price接口,分别对应马蜂窝、booking、携程、艺龙、agoda、有鱼,
而我们目标获取携程跟艺龙
接下来分析网页,先获取ajax页面,里面比较重要的参数是_ts(时间戳)跟_sn
_sn参数
搜全局搜不到,应该就是js加密处理了,点all找script看看,
前两个都是关于页面布局,所以我们点index.js?1552035728
恩…看起来就是这个了
随意找几个地方打上断点,这就看到_ts参数了!
363这行会重复几次,这时跳回network点xhr看,
发现新产生的页面_ts参数跟刚刚断点产生的一样
仔细看一下364行return的部份
首先生成字符串(1)
(JSON.stringify(_0xe7fex39) + _0xe7fex34))
再通过函数_0xe7fex2[__Ox2133f[40]]生成字符串(2),
最后取其位置[2:12]就是_sn参数
研究一下_0xe7fex2[__Ox2133f[40]]这个函数,
_0xe7fex28就是最开始生成的字符串(1)
_0xe7fexc会透过字符串(1)生成长度为4的数组
最后返回_0xe7fex10(_0xe7fex2d)的结果
记录一下这个判断语句
!!_0xe7fex2a ? _0xe7fex2d : _0xe7fex10(_0xe7fex2d)
- !!(a) 当a是实际的变量时为True;否则null、undefined、’ '都为False
- a?b:c 当a为True时执行b;否则执行c
而(_0xe7fex2a)为undefined,所以是return(_0xe7fex10)函数的结果
上面这段操作其实就是Md5加密
Md5加密:
输出:固定长度128-bits。由四个32位分组组成,
将这四个32位分组级联后将生成一个128位散列值。
基本方式为:求余、取余、调整长度、与链接变量进行循环运算。得出结果。
所以我们直接引用python里的hashlib库
def par(self,a):
hl = hashlib.md5()
hl.update(a)
return hl.hexdigest()[2:12]
这样就得到_sn的值了
但具体的字符串组成到底需要哪些参数呢,
一样打断点来找
各网站的price poi也是一样道理,直接打断点复制字符串
总结一下怎麽获取价格
1.先得到页面(获得酒店名跟马蜂窝的酒店id)
2.透过酒店id 对应各网站的接口获取价格
def getHotel(n):
t = str(int(time.time() * 1000))
room = '{"_ts":"' + t + '","has_booking_rooms":"0","has_faved":"0","iAdultsNum":"2","iAreaId":"-1","iChildrenNum":"0","iDistance":"10000","iMddId":"10575","iPage":"'+str(n)+'","iPoiId":"","iPriceMax":"","iPriceMin":"","iRegionId":"-1","nLat":"0","nLng":"0","position_name":"","sAction":"getPoiList5","sCheckIn":"2021-07-09","sCheckOut":"2021-07-10","sChildrenAge":"","sKeyWord":"","sSortFlag":"DESC","sSortType":"hot","sTags":""}c9d6618dbc657b41a66eb0af952906f1'
sn = par(room.encode('utf-8'))
url = 'https://www.mafengwo.cn/hotel/ajax.php?iMddId=10575&iAreaId=-1&iRegionId=-1&iPoiId=&position_name=&nLat=0&nLng=0&iDistance=10000&sCheckIn=2021-07-09&sCheckOut=2021-07-10&iAdultsNum=2&iChildrenNum=0&sChildrenAge=&iPriceMin=&iPriceMax=&sTags=&sSortType=hot&sSortFlag=DESC&has_booking_rooms=0&has_faved=0&sKeyWord=&iPage={}&sAction=getPoiList5&_ts={}&_sn={}'.format(
n,t, sn)
headers = {
'Cookie': 自己的cookie,
'Host': 'www.mafengwo.cn',
'Accept-Encoding': "gzip, deflate, br",
'Accept-Language': "zh-CN,zh;q=0.9",
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66",
'Accept': "application/json, text/java, */*; q=0.01",
'Referer': "http://www.mafengwo.cn/hotel/10099/",
'X-Requested-With': "XMLHttpRequest",
'Connection': "keep-alive",
}
res = requests.get(url=url,headers=headers)
data = json.loads(res.text)
page = jsonpath(data,'$.html')[0]
#用xpath提取
selector = Selector(page)
name = selector.xpath('//*[@class="hotel-item clearfix _j_hotel_item"]/@data-name').extract()
hotel_id = selector.xpath('//*[@class="hotel-item clearfix _j_hotel_item"]/@data-id').extract()
def elong(id,len):
id = ",".join(id)
elong = '{"_ts":"' + t + '","adults_num":"2","check_in":"2021-07-09","check_out":"2021-07-10","children_age":"","children_num":"0","mddid":"10575","ota":"elong","platform":"www","poi_ids":"'+id+'"}c9d6618dbc657b41a66eb0af952906f1'
sn = par(elong.encode('utf-8'))
url = 'https://www.mafengwo.cn/hotel/ajax_remote/price?poi_ids=' + p.quote(id) + '&mddid=10575&ota=elong&check_in=2021-07-09&check_out=2021-07-10&adults_num=2&children_num=0&children_age=&platform=www&_ts={}&_sn={}'.format(t, sn)
headers = {
'Cookie': 自己的cookie,
'Host': 'www.mafengwo.cn',
'Accept-Encoding': "gzip, deflate, br",
'Accept-Language': "zh-CN,zh;q=0.9",
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66",
'Accept': "application/json, text/java, */*; q=0.01",
'Referer': "http://www.mafengwo.cn/hotel/10575/",
'X-Requested-With': "XMLHttpRequest",
'Connection': "keep-alive",
}
res = requests.get(url=url,headers=headers)
#用正则提取
price = re.findall(r'\"list\"\:\{\"elong\"\:(.*?)\}', res.text)