scrapy的一些事

保存html源码方法：

        with open("a,html",'w',encoding='utf-8') as f:
            f.write(response.body.decode())

下载网页图片：

from urllib import request
request.urlretrieve('网址','xxx.jpg')

对内容进行去除空格：

def process_content(self,content):
        content = [re.sub(r'\s|\xa0','',i) for i in content]#将换行什么的符号替换成空格
        content = [i for i in content if len(i) > 0]#将空格号去掉
        return content

若爬取的url地址不完整：

import urllib
item["href"] is not None:
	item["href"] = urllib.parse.urljoin(response.url,item["href"])

登录可以有以下操作：

1. cookies:`

    def start_requests(self):
        cookies = "_s_tentry=-; Apache=1769401044437.5996.1588400454551; SINAGLOBAL=1769401044437.5996.1588400454551; ULV=1588400454765:1:1:1:1769401044437.5996.1588400454551:; login_sid_t=8111d554c0a2d6dcf9228ebc58708bdf; cross_origin_proto=SSL; Ugrow-G0=9ec894e3c5cc0435786b4ee8ec8a55cc; YF-V5-G0=7a7738669dbd9095bf06898e71d6256d; UOR=,,www.baidu.com; wb_view_log=1366*7681; SUB=_2A25zutvVDeRhGeRK7lIS8CfJwj6IHXVQzkodrDV8PUNbmtAKLVjNkW9NU2bIGGUcalCxA0p7h9RIP8zDZ5jkAOnZ; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9W5U.g5UpcseWH5HDGKF_78r5JpX5KzhUgL.FozXSK50eh.f1Kz2dJLoI7yWwJyadJMXSBtt; SUHB=0P19BEYcGR28ZI; ALF=1621090026; SSOLoginState=1589554053; wvr=6; wb_view_log_2450309592=1366*7681; YF-Page-G0=e44a6a701dd9c412116754ca0e3c82c3|1589556569|1589556569; webim_unReadCount=%7B%22time%22%3A1589556679580%2C%22dm_pub_total%22%3A0%2C%22chat_group_client%22%3A0%2C%22chat_group_notice%22%3A0%2C%22allcountNum%22%3A0%2C%22msgbox%22%3A0%7D"
        cookies = {i.split("=")[0]:i.split("=")[1] for i in cookies.split(";")}
        yield scrapy.Request(
            self.start_urls[0],
            callback = self.parse,
            cookies = cookies
        )

2. post登录

        authenticity_token = response.xpath('//div[@class = "auth-form px-3"]/form/input[1]/@value').extract_first()
        ga_id = response.xpath('//div[@class = "auth-form px-3"]/form/input[2]/@value').extract_first()
        commit = response.xpath('//input[@name = "commit"]/@value').extract_first()
        post_data = dict(
            login='Azhong-github',
            password= 'WUzhong961028',
            authenticity_token = authenticity_token,
            ga_id = str(ga_id),
            commit = commit
       

3. scrapy模拟登录之自动登录

def parse(self, response):
        yield scrapy.FormRequest.from_response(
            response,#自动的从respon中寻找form表单
            formdata={"login":"Azhong-github","password":"WUzhong961028"},
            callback = self.after_login
        )

get请求：模拟输入（如百度搜索框）：
要善于发现get请求的规律，下面？是分隔符

kw={"kw":"电子科技大学"}       
        kw = parse.urlencode(kw,encoding = "utf-8"("gb2312")) #kw：wd=%e7%94%b5%e5%ad%90%e7%a7%91%e6%8a%80%e5%a4%a7%e5%ad%a6
        url = response.url + "?" + kw
        print(type(url))
        yield scrapy.Request(
            url,
            callback = self.parse_detail
        )