基于Scrapy框架的Weibo爬虫

大名鼎鼎的Scrapy，用了以后感觉并发爬取速度真的是秒杀一切，仅做学习交流。

第一，pip安装Scrapy。

第二，cmd命令行输入scrapy startproject weibo

第三，进入weibo文件夹，SHIFT加右键进入powershell输入scrapy genspider wb web

第四，打开item.py编辑，因为我是测试，只抓取微博用户id账号、微博用户名和微博内容，其实通过m.weibo.cn的api可以获取很多字段，这里道理是一样的。

import scrapy


class WeiboItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    _id = scrapy.Field()
    _name = scrapy.Field()
    _text = scrapy.Field()

第五，新建get_cookie.py，用来获取cookie。

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import json


browser_options = Options()
browser = webdriver.Chrome(options=browser_options)
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) '
                         'AppleWebKit/537.36 (KHTML, like Gecko) Chrome'
                         '/103.0.0.0 Safari/537.36'}
print("浏览器已成功创建。")


def get_cookie(url='https://passport.weibo.cn/signin/login?'):
    url = url
    browser.get(url)
    print('现在请在Chrome浏览器中登录你的账号。')
    waiting = True
    while waiting:
        ok = input('登录成功后，输入任意内容并按回车键继续。')
        if ok:
            waiting = False
    with open('cookies.txt', 'w') as f:
        f.write(json.dumps(browser.get_cookies()))
        f.close()
    print('已成功保存cookie信息。')


if __name__ == '__main__':
    get_cookie()

第六，打开middlewares.py编辑，读取cookies.txt，别忘记import json。

    def process_request(self, request, spider):
        request.headers = {'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                           'Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67'}
        with open('cookies.txt', 'r') as f:
            cookies = json.load(f)
            request.cookies = cookies

第七，打开settings.py编辑，把以下三个的注释取掉，值改掉，其他的不用管：

ROBOTSTXT_OBEY = False

COOKIES_ENABLED = True

ITEM_PIPELINES = {
    "weibo.pipelines.WeiboPipeline": 300,}

第八，打开wb.py编辑，设置spider的逻辑，注意为什么使用deepcopy，因为如果不使用deepcopy复制当前微博item的其他数据，长微博的内容就会与微博id和用户名不匹配，可能是并发造成的，修改priority并没用。

from weibo.items import WeiboItem
from copy import deepcopy
import scrapy
import json

class WbSpider(scrapy.Spider):
    name = "wb"
    allowed_domains = ["m.weibo.cn"]
    Q = input('请输入搜索关键词：')
    page = 1
    start_urls = [f"https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D61%26q%3D{Q}"
                  f"%26t%3D%26page_type=searchall&page={page}"]

    def parse(self, response):
        result = json.loads(response.text)
        result = result['data']['cards']
        item = WeiboItem()
        for detail in result:
            item['_id'] = detail['mblog']['user']['id']
            item['_name'] = detail['mblog']['user']['screen_name']
            print('++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++', detail['mblog']['isLongText'])

            if detail['mblog']['isLongText'] is True:
                blog_id = detail['mblog']['id']
                blog_id = ''.join(blog_id)
                full_text_url = 'https://m.weibo.cn/statuses/extend?id=' + blog_id
                yield scrapy.Request(full_text_url, callback=self.long_text_parse, meta={'item': deepcopy(item)},
                                     priority=1)
            else:
                _text = detail['mblog']['text']
                _text = _text.replace('\n', '')
                _text = ''.join([x.strip() for x in _text])
                item['_text'] = _text
                yield item

        self.page += 1
        next_page = f"https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D61%26q%3D{self.Q}" \
                    f"%26t%3D%26page_type=searchall&page={self.page}"
        yield scrapy.Request(next_page, callback=self.parse, meta={'item': item})

    def long_text_parse(self, response):
        result = json.loads(response.text)
        item = response.meta['item']
        _text = result['data']['longTextContent']
        _text = _text.replace('\n', '')
        _text = ''.join([x.strip() for x in _text])
        item['_text'] = _text
        yield item

第九，打开pipelines.py编辑，用来设置将抓取的数据导出为CSV文件。

from weibo.spiders.wb import WbSpider
import csv


class WeiboPipeline:
    def __init__(self):
        self.f = open(f'{WbSpider.Q}.csv', 'w', encoding='utf8', newline='')
        self.file_name = ['_id', '_name', '_text']
        self.writer = csv.DictWriter(self.f, fieldnames=self.file_name)
        self.writer.writeheader()

    def process_item(self, item, spider):
        self.writer.writerow(dict(item))
        return item

    def close_spider(self, spider):
        self.f.close()

第十，新建start.py，避免每次必须在cmd中启动爬虫。

from scrapy import cmdline

cmd = 'scrapy crawl wb'.split()
cmdline.execute(cmd)

最后，直接运行start.py即可。

总结：Scrapy的难点核心在于spider文件的爬取逻辑，其他的都很easy。