建标库(JianBiaoKu.com)图片数据转PDF爬虫案例

原创文章，码字不易，未经博主允许请勿转载。博主QQ：859699677

本文链接：https://blog.csdn.net/joson1234567890/article/details/105138707

本文介绍了一种使用Python爬虫获取项目文件中所有页码的方法，并通过异步请求获取图片URL，最后将图片转换为PDF文件。文章详细展示了如何定义函数获取所有请求URL、异步请求图片、下载图片并生成PDF。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1、获取单个项目文件所有页码

def get_all_url(self):
    """获取所有请求URL"""
    res = self.s.get(self.startUrl, headers=headers)
    res.encoding = 'utf-8'
    selector = etree.HTML(res.text)
    urls = selector.xpath('//ul[@class="book_catalog"]//@href')
    params = '/webarbs/book/{}/'.format(self.projectId)
    urls = [self.domain + params + x for x in urls]
    title = selector.xpath('//div[@class="location"]/span/a[last()]/text()')[0]
    title = re.sub(r'\||\<|\>|\\|\/|\:|\*|\"|\?','-',str(title))
    return urls, title

2、异步请求获取所有图片URL

async def getImageUrls(self, url):
	"""获取图片URL"""
    conn = TCPConnector(limit=10)
    async with ClientSession(connector=conn) as session:
        async with session.get(url, headers=self.headers) as response:
            html = await response.text()
            img = re.findall('http://www.zzguifan.com.+?\.jpg', html)[0]
            return img
            
def download(self, page, task):
    """下载图片"""
    res = self.s.get(task.result(), headers=self.headers)
    imgName = page.split('/')[-1][:-6]
    savePath = sys.path[0] + '\\' + self.projectId + '\\'
    self.mkdir(savePath)
    fullPath = savePath + imgName + '.jpg'
    self.imageIds[int(imgName)] = fullPath  # 保存路径信息方便后面生成PDF
    with open(fullPath, 'wb') as f:
        f.write(res.content)

async def taskManger(self):
    """异步任务管理"""
    tasks = []
    for url in self.urls:
        task = asyncio.create_task(self.getImageUrls(url))
        task.add_done_callback(partial(self.download, url))  # 设置回调函数参数
        tasks.append(task)
    await asyncio.gather(*tasks)

3、生成PDF文件

def makePdf(pdfFileName, listPages):
    """图片转PDF"""
    cover = Image.open(listPages[0])
    width, height = cover.size

    pdf = FPDF(unit="pt", format=[width, height])

    for page in listPages:
        pdf.add_page()
        pdf.image(page, 0, 0)

    pdf.output(pdfFileName, "F")