scrapy post提交json内容

编写测试接口 post_api.py

# -*- coding:utf-8 -*-
# @FileName  :post_api.py
# @Time      :2020/8/7 9:25
# @Author    :pylemon


from flask import Flask, request

app = Flask(__name__)


@app.route('/', methods=['POST'])
def insert():
    r = request
    info = request.data.decode('utf-8')

    return {'success': True, 'msg': info}


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=80)

首先使用 requests 编写测试看看

# -*- coding:utf-8 -*-
# @FileName  :requests_test.py
# @Time      :2020/8/7 9:29
# @Author    :pylemon
import json

import requests

# json 格式
data = {"name": 'pylemon'}
response = requests.post(url='http://127.0.0.1', data=json.dumps(data))
print(response.text)
# 结果: {"msg":"{\"name\": \"pylemon\"}","success":true}
# dict 格式
data = {"name": 'pylemon'}
response = requests.post(url='http://127.0.0.1', data=data)
print(response.text)
# 结果：{"msg":"","success":true}

运行结果 发现 requests post 请求时，data值设置为dict类型 api 接口就无法获取，需要json格式化

那么scrapy 如何post 提交 json 呢

错误示例

import json

import scrapy


class ATSpider(scrapy.Spider):
    name = '_t'
    allowed_domains = []
    start_urls = ['http://127.0.0.1']

    def start_requests(self):
        data = {"ww": 'aaa'}

        yield scrapy.FormRequest(url="http://127.0.0.1", formdata=json.dumps(data))

    def parse(self, response):
        pass

scrapy.FormRequest的 方法不支持json 的格式，必须时dict。

scrapy 错误提示：
ValueError: not enough values to unpack (expected 2, got 1)

但是 改成 yield scrapy.FormRequest(url="http://127.0.0.1", formdata=data) 接口又获取不到，所以只能用scrapy.Request 编写post 的请求

正确示例

import json

import scrapy


class ATSpider(scrapy.Spider):
    name = '_t'
    allowed_domains = []
    start_urls = ['http://127.0.0.1']

    def start_requests(self):
        data = {"name": 'pylemon'}
        request = scrapy.Request('http://127.0.0.1', method='POST',
                          body=json.dumps(data),
                          headers={'Content-Type': 'application/json'})
        yield request

    def parse(self, response):
        pass