python php 抓包,python - 抓包只抓到json，真实的地址却找不到

巴扎黑2017-04-18 10:24:532楼

说一下我的方法，已经爬取到数据。我用的是firebug,打开后找到如下路径：https://www.yilan.io/article/recommended

看了一下要post的内容，需要这组数据{"skip":0,"limit":20}。下面开始写代码：

import urllib2

import urllib

import gzip

from StringIO import StringIO

import json

api = 'https://www.yilan.io/article/recommended'

data = {"skip":0,"limit":20}

headers = { 'Accept': 'application/json, text/plain, */*',

'Accept-Encoding': 'gzip, deflate',

'Accept-Language': 'zh-CN,zh',

'Connection': 'keep-alive',

'Cookie': 'XSRF-TOKEN=APc3KgEq-6wavGArI6rLf6tPW69j7H_Qm2s0; user=%7B%22_id%22%3A%22%22%2C%22role%22%3A%7B%22title%22%3A%22anon%22%2C%22bitMask%22%3A1610612736%7D%7D; Metrix-sid=s%3AjDAFvFGo3C0BJzR7cTXBXHl6VM493Gp0.C1svjUqfnY3NhUluURMDdaL3HEpUX8rpSj9%2F9yhKnEI',

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:51.0) Gecko/20100101 Firefox/51.0',

'X-XSRF-TOKEN': 'APc3KgEq-6wavGArI6rLf6tPW69j7H_Qm2s0'

}

url_data = urllib.urlencode(data)

request = urllib2.Request(api, data=url_data,headers=headers)

content = urllib2.urlopen(request).read()

contents = StringIO(content)

f = gzip.GzipFile(mode='rb', fileobj=contents).read()

b = json.loads(f)

print b

运行结果如下：

[{u'readCount': 12, u'siteTitle': u'\u5de5\u5177\u7656', u'siteUrl': u'http://jianshu.milkythinking.com/feeds/collections/2mvgxp', u'siteFavicon': u'http://7xiyp4.com2.z0.glb.qiniucdn.com/site-5627773e8b3ac7e104c6280f-favicon', u'feature_image': u'http://7x2wdd.com2.z0.glb.qiniucdn.com/54e7178471ab07ea378e0d254a57b3cc', u'author': u'', u'url': ...]

然后提取想要的内容就行，可以更改limit的值来改变一次所要获取的内容数量。

该网站可能会在后台检查你所post的数据，如果错误会导致404，这也就是直接打开该路径不能访问的原因。