scrapy各种请求参数

最新推荐文章于 2024-07-17 15:15:27 发布

__IProgrammer

最新推荐文章于 2024-07-17 15:15:27 发布

阅读量3.3k

点赞数

分类专栏：爬虫

本文链接：https://blog.csdn.net/wang785994599/article/details/108997215

版权

爬虫专栏收录该内容

20 篇文章 13 订阅

订阅专栏

get请求查询字符串

1.使用FormRequest

    start_urls = ['https://careers.tencent.com/tencentcareer/api/post/Query?']

    def start_requests(self):
        for url in self.start_urls:
            params = {
                "timestamp": str(int(time.time() * 1000)),
                "countryId": "",
                "cityId": "",
                "bgIds": "",
                "productId": "",
                "categoryId": "",
                "parentCategoryId": "",
                "attrId": "",
                "keyword": "",
                "pageIndex": "1",
                "pageSize": "10",
                "language": "zh-cn",
                "area": "cn"
            }
            yield scrapy.FormRequest(url, formdata=params,method="get")

2.urlencode

	from urllib.parse import urlencode
    start_urls = ['https://careers.tencent.com/tencentcareer/api/post/Query?']

    def start_requests(self):
        for url in self.start_urls:
            params = {
                "timestamp": str(int(time.time() * 1000)),
                "countryId": "",
                "cityId": "",
                "bgIds": "",
                "productId": "",
                "categoryId": "",
                "parentCategoryId": "",
                "attrId": "",
                "keyword": "",
                "pageIndex": "1",
                "pageSize": "10",
                "language": "zh-cn",
                "area": "cn"
            }
            url = url + urlencode(params)
            yield scrapy.Request(url,method="get"

get请求 payload

payload= {"page": 1, "size": 15, "searchKeys": ["companyNameCN", "companyNameEN"], "isSearch": False,
                    "selectAndMap": {"categoryId": [id], "boothAreaSearch": [], "boothNumber": []},
                    "orderModel": {"order": "asc", "properties": ["companyNamePinyin"]},
                    "selectOrMap": {"isPovertyAlleviation": [], "isFirstJoin": [], "isContinuousJoin": [],
                                    "isBrand": [],
                                    "exhibitorType": [], "productTrait": [], "isCfWinner": [], "tradeTypes": [],
                                    "isGreenAward": [],
                                    "isInvitationAward": []}, "searchValue": ""}
            yield scrapy.Request(url,
                                 method="get",
                                 body=json.dumps(data),
                                 )