Python 文件下载工具方法

2019独角兽企业重金招聘Python工程师标准>>>

在使用Requests进行爬虫学习的时候封装的文件下载的工具方法

import requests
from contextlib import closing
import os
HEADERS={
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.37 Safari/537.36 ",
    "Referer": "http://www.51voa.com/"}
def download(download_url, save_path, file_name, suffix):
    """
    下载工具方法
    :param download_url:
    :param save_path:
    :param file_name:
    :param suffix:
    :return:
    """
    counter = 0
    if not save_path.endswith("\\"):
        save_path += "\\"
        if not os.path.exists(save_path):
            os.makedirs(save_path)
        # 读取完网络请求流之后需要关闭
        with closing(requests.get(download_url, headers=HEADERS, stream=True)) as resp:
            # 打开文件准备写入
            with open(''.join([save_path, file_name, suffix]), 'wb') as f:
                # 写入我文件
                for chunk in resp.iter_content(chunk_size=1024):
                    if chunk:
                        f.write(chunk)
                        counter += 1
                        if counter == 100:
                            f.flush()
                            counter = 0

 

转载于:https://my.oschina.net/liuyuantao/blog/842625