Python解决下载pdf问题bug

最新推荐文章于 2021-03-26 22:27:47 发布

十一姐

最新推荐文章于 2021-03-26 22:27:47 发布

阅读量2.4k

点赞数

分类专栏：爬虫相关学习

本文链接：https://blog.csdn.net/weixin_43411585/article/details/102953108

版权

爬虫相关学习专栏收录该内容

148 篇文章

订阅专栏

代码如下：

1、BUG问题：requests返回的二进制结果resp.content=b’ '为空，无法下载pdf
2、产生原因： response.close()方法会调用HttpWorkerRequest.CloseConnection()方法。终止(Terminate)与客户端的套接字连接，并使得服务器，客户端以及之间设施上的缓存（buffer）失效。导致发送到客户端的数据丢失。如果还未存数据就关闭连接，易造成数据丢失。
3、解决方法：response.close()关闭提后或者不使用

def get_proxies():
    proxy = {}
    return proxy

def pdf_url_requests(pdf_url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
        "Accept-Encoding": "gzip, deflate, br",
        "Accept-Language": "zh-CN,zh;q=0.9",
    }
    for i in range(4):
        try:
            resp = requests.get(pdf_url, headers=headers, proxies=proxy, stream=True, timeout=20)
            resp.encoding = "utf-8"
        except socket.error as err:
            logging.warning(f"{pdf_url} : this website may socket timeout, sleep 1s and try again : {err} ")
            time.sleep(random.uniform(0.5, 1.5))
            proxy = get_proxies()
        except Exception as _e:
            time.sleep(random.uniform(0.5, 1.5))
            logging.exception(f'{pdf_url} ：Exception error happened in methods_get_requests :{_e}')
            proxy = get_proxies()
        else:
            if not resp:
                continue
            return resp

def download_pdf(pdf_url, pdf_name):
    r = pdf_url_requests(pdf_url)
    pdf_path = f"{E:/pdf/AuditReport/{pdf_name}"
    with open(pdf_path, 'wb') as f:
        for content in r.iter_content(chunk_size=512):
            if content:
                f.write(content)
    r.close()