scrapy Ajax request,scrapy爬虫:scrapy.FormRequest中formdata参数详解

1. 背景

在网页爬取的时候，有时候会使用scrapy.FormRequest向目标网站提交数据(表单提交)。参照scrapy官方文档的标准写法是：

# header信息

unicornHeader = {

'Host': 'www.example.com',

'Referer': 'http://www.example.com/',

}

# 表单需要提交的数据

myFormData = {'name': 'John Doe', 'age': '27'}

# 自定义信息，向下层响应(response)传递下去

customerData = {'key1': 'value1', 'key2': 'value2'}

yield scrapy.FormRequest(url = "http://www.example.com/post/action",

headers = unicornHeader,

method = 'POST', # GET or POST

formdata = myFormData, # 表单提交的数据

meta = customerData, # 自定义，向response传递数据

callback = self.after_post,

errback = self.error_handle,

# 如果需要多次提交表单，且url一样，那么就必须加此参数dont_filter，防止被当成重复网页过滤掉了

dont_filter = True

)

但是，当表单提交数据myFormData 是形如字典内嵌字典的形式，又该如何写？

2. 案例 — 参数为字典

在做亚马逊网站爬取时，当进入商家店铺，爬取店铺内商品列表时，发现采取的方式是ajax请求，返回的是json数据。

请求信息如下：

响应信息如下：

如上图所示，From Data中的数据包含一个字典：

marketplaceID:ATVPDKIKX0DER

seller:A2FE6D62A4WM6Q

productSearchRequestData:{"marketplace":"ATVPDKIKX0DER","seller":"A2FE6D62A4WM6Q","url":"/sp/ajax/products","pageSize":12,"searchKeyword":"","extraRestrictions":{},"pageNumber":"1"}

# formDate 必须构造如下：

myFormData = {

'marketplaceID' : 'ATVPDKIKX0DER',

'seller' : 'A2FE6D62A4WM6Q',

# 注意下面这一行，内部字典是作为一个字符串的形式

'productSearchRequestData' :'{"marketplace":"ATVPDKIKX0DER","seller":"A2FE6D62A4WM6Q","url":"/sp/ajax/products","pageSize":12,"searchKeyword":"","extraRestrictions":{},"pageNumber":"1"}'

}

在amazon中实际使用的构造方法如下：

def sendRequestForProducts(response):

ajaxParam = response.meta

for pageIdx in range(1, ajaxParam['totalPageNum']+1):

ajaxParam['isFirstAjax'] = False

ajaxParam['pageNumber'] = pageIdx

unicornHeader = {

'Host': 'www.amazon.com',

'Origin': 'https://www.amazon.com',

'Referer': ajaxParam['referUrl'],

}

'''

marketplaceID:ATVPDKIKX0DER

seller:AYZQAQRQKEXRP

productSearchRequestData:{"marketplace":"ATVPDKIKX0DER","seller":"AYZQAQRQKEXRP","url":"/sp/ajax/products","pageSize":12,"searchKeyword":"","extraRestrictions":{},"pageNumber":1}

'''

productSearchRequestData = '{"marketplace": "ATVPDKIKX0DER", "seller": "' + f'{ajaxParam["sellerID"]}' + '","url": "/sp/ajax/products", "pageSize": 12, "searchKeyword": "","extraRestrictions": {}, "pageNumber": "' + str(pageIdx) + '"}'

formdataProduct = {

'marketplaceID': ajaxParam['marketplaceID'],

'seller': ajaxParam['sellerID'],

'productSearchRequestData': productSearchRequestData

}

productAjaxMeta = ajaxParam

# 请求店铺商品列表

yield scrapy.FormRequest(

url = 'https://www.amazon.com/sp/ajax/products',

headers = unicornHeader,

formdata = formdataProduct,

func = 'POST',

meta = productAjaxMeta,

callback = self.solderProductAjax,

errback = self.error, # 处理http error

dont_filter = True, # 需要加此参数的

)