python爬取js里的数据_python – Scrapy,在Javascript中抓取数据

(我把它发布到了scrapy-users邮件列表,但保罗建议我在这里发布它,因为它补充了shell命令交互的答案.)

通常,使用第三方服务呈现某些数据可视化(地图,表格等)的网站必须以某种方式发送数据,并且在大多数情况下,可以从浏览器访问这些数据.

所以,基本上你可以在一个漂亮的json格式中准备好消费所需的所有数据.

Scrapy在写蜘蛛之前提供了shell命令,这对于思想者来说非常方便：

$scrapy shell https://www.mcdonalds.com.sg/locate-us/

2013-09-27 00:44:14-0400 [scrapy] INFO: Scrapy 0.16.5 started (bot: scrapybot)

...

In [1]: from scrapy.http import FormRequest

In [2]: url = 'https://www.mcdonalds.com.sg/wp-admin/admin-ajax.php'

In [3]: payload = {'action': 'ws_search_store_location', 'store_name':'0', 'store_area':'0', 'store_type':'0'}

In [4]: req = FormRequest(url, formdata=payload)

In [5]: fetch(req)

2013-09-27 00:45:13-0400 [default] DEBUG: Crawled (200) (referer: None)

...

In [6]: import json

In [7]: data = json.loads(response.body)

In [8]: len(data['stores']['listing'])

Out[8]: 127

In [9]: data['stores']['listing'][0]

Out[9]:

{u'address': u'678A Woodlands Avenue 6
#01-05
Singapore 731678',

u'city': u'Singapore',

u'id': 78,

u'lat': u'1.440409',

u'lon': u'103.801489',

u'name': u"McDonald's Admiralty",

u'op_hours': u'24 hours

Dessert Kiosk: 0900-0100',

u'phone': u'68940513',

u'region': u'north',

u'type': [u'24hrs', u'dessert_kiosk'],

u'zip': u'731678'}

简而言之：在你的蜘蛛中,你必须返回上面的FormRequest(…),然后在回调中加载来自response.body的json对象,最后是列表数据中每个商店的数据[‘stores’] [‘listing’ ]创建一个包含所需值的项目.

像这样的东西：

class McDonaldSpider(BaseSpider):

name = "mcdonalds"

allowed_domains = ["mcdonalds.com.sg"]

start_urls = ["https://www.mcdonalds.com.sg/locate-us/"]

def parse(self, response):

# This receives the response from the start url. But we don't do anything with it.

url = 'https://www.mcdonalds.com.sg/wp-admin/admin-ajax.php'

payload = {'action': 'ws_search_store_location', 'store_name':'0', 'store_area':'0', 'store_type':'0'}

return FormRequest(url, formdata=payload, callback=self.parse_stores)

def parse_stores(self, response):

data = json.loads(response.body)

for store in data['stores']['listing']:

yield McDonaldsItem(name=store['name'], address=store['address'])