之前在抓一个网站Danaos的时候,发现能用requests和postman发送同样的header和payload的时候都能得到正确的结果,但是scrapy就会返回400错误,后来发现这是Twisted本身存在的问题,看了官网也没找到解决方法
spider文件在这里:
class DanspiderSpider(scrapy.Spider):
name = 'danspider'
allowed_domains = ['www.danaos.com']
def start_requests(self):
payload = '{"serviceDto":{"ViewType":"2","ViewDate":"","RevisionNumber":"1","LanguageId":"1","ItemCount":-1,"StartIndex":0,"Signature":"","TagList":[]},"year":-1,"assetType":"Fleet_Details"}'
url = "https://www.danaos.com/Services/ContentAssetService.svc/GetContentAssetList"
headers = {
"Accept"