queue.Queue、生产者消费者架构

1.多组件的Pipeline技术：

2.生产者消费者爬虫架构
3.多线程数据通信的queue.Queue

import requests
from bs4 import BeautifulSoup

urls = [
    f"http://www.cnblogs.com/#p{page}"
    for page in range(1, 50 + 1)
]


def craw(url):
    r = requests.get(url)
    return r.text


def parse(html):
    # class="post-item-title"
    soup = BeautifulSoup(html, "html.parser")
    links = soup.find_all("a",

    class_="post-item-title")

    return [(link["href"], link.get_text()) for link in links]


if __name__=="__main__":
    for result in parse(craw(urls[2])):
        print(result)

输出结果：
('https://www.cnblogs.com/imyalost/p/14966389.html', '性能测试基础知识体系')
('https://www.cnblogs.com/it-rabbit-cyj/p/14966325.html', '海量数据Excel报表利器——EasyExcel（开场篇）')
('https://www.cnblogs.com/henjay724/p/14966318.html', '痞子衡嵌入式：以i.MXRT1xxx的GPIO模块为例谈谈中断处理函数(IRQHandler)的标准流程')
('https://www.cnblogs.com/zero-zyq/p/14907381.html', 'Salesforce LWC学习(三十五) 使用 REST API实现不写Apex的批量创建/更新数据')
('https://www.cnblogs.com/wangpenghui522/p/14965971.html', '一篇技术博文引发的stylelint项目实践')
('https://www.cnblogs.com/Wendy-r/p/14961692.html', '用户RFM模型及应用')
('https://www.cnblogs.com/three-fighter/p/14965974.html', '假如我是超级躺王')
('https://www.cnblogs.com/xuliuzai/p/14956248.html', '数据库服务器资源使用情况周报')
('https://www.cnblogs.com/bolingcavalry/p/14965877.html', 'hive学习笔记之五：分桶')
('https://www.cnblogs.com/lianngkyle/p/14965824.html', 'external-attacher源码分析(1)-main方法与启动参数分析')
('https://www.cnblogs.com/niejunlei/p/14965620.html', '解决缓存穿透的几种应用思考')
('https://www.cnblogs.com/nhyq-wyj/p/14965740.html', '简易的服务器接口开发入门教程')
('https://www.cnblogs.com/Yee-Q/p/14965722.html', 'Netty 框架学习 —— 预置的 ChannelHandler 和编解码器')
('https://www.cnblogs.com/Ahoo-Wang/p/cosid-1-1-0.html', 'CosId 1.1.0 发布，通用、灵活、高性能的分布式 ID 生成器')
('https://www.cnblogs.com/hont/p/14965382.html', 'CapsuleAO的实现与学习')
('https://www.cnblogs.com/wsjhk/p/14965651.html', '【原创】Ingress-Nginx-Controller的Metrics监控源码改造简析')
('https://www.cnblogs.com/rossiXYZ/p/14965391.html', '[源码解析] 深度学习分布式训练框架 horovod (9) --- 启动 on spark')
('https://www.cnblogs.com/onehm/p/14965379.html', 'mycat高可用-安全管理-监控   看这一篇就够了')
('https://www.cnblogs.com/guying-blog/p/14965291.html', '23天设计模式之迭代器模式')
('https://www.cnblogs.com/larrydpk/p/14965169.html', 'Terraform状态State管理，让变更有记录')

Process finished with exit code 0

consumer.py

import queue
import blog_spider
import time
import random
import threading
def do_craw(url_queue:queue.Queue,html_queue:queue.Queue):
    while True:
        url = url_queue.get()
        html=blog_spider.craw(url)
        html_queue.put(html)
def do_parse(html_queue:queue.Queue,fout):
    while True:
        html=html_queue.get()
        results=blog_spider.parse(html)
        for result in results:
            fout.writr(str(result)+"\n")