scrapy爬取GitHub爬虫实战记录

最新推荐文章于 2024-04-08 10:58:04 发布

雨天雨天氵

最新推荐文章于 2024-04-08 10:58:04 发布

阅读量1.4k

点赞数

分类专栏：爬虫文章标签： github 大数据数据挖掘

本文链接：https://blog.csdn.net/qq_44795237/article/details/109157478

版权

本文详细记录了使用Scrapy爬虫抓取GitHub数据的实战过程，包括项目打包成.exe遇到的问题及解决方案。在打包过程中，通过修改settings.py、middlewares.py、items.py、pipelines.py等文件，实现了数据的高效抓取和存储。通过设置配置参数，如ROBOTSTXT_OBEY、CONCURRENT_REQUESTS和DOWNLOAD_DELAY，优化了爬虫行为。此外，还介绍了如何在spiders文件夹下创建爬虫，并通过命令行运行爬虫，实现数据的导出。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

scrapy爬虫实战记录（从实现到打包.exe）

项目爬虫过程记录。

项目打包中遇到的问题：

项目打包方式在网上基本都有，有关scrapy打包的介绍不多介绍，将别人的打包必要文件拿来，放在于.cfg文件同级目录下，命名随意。

在打包过程中，自己最开始是同网上一样，使用crawl.来启动爬虫，但发现没办法输入-o 后面的参数，虽然url能够导入，我们也需要url从命令行执行exe时传入。第二个问题很好解决，调用sys就可以。另一个问题在网上找了很久，才看到别人一段没有说的小串代码，觉得就是这样的就试了试，才成功的。

对于第一个启动FEED功能，帮助我们把数据全部写入到本地时，自己先找到的是调用cmdline.excute，但是打包后，会提示scrapy没有crawl命令，就僵住了。之后发现可以修改get_project_settings()中的参数，修改s[‘FEED_URI’] = 'output.csv’即可成功完成打包，并直接运行。

# -*- coding: utf-8 -*-
# import scrapy
# from scrapy import cmdline
import sys
import xlwt
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

# 这里是必须引入的
# import robotparser

import scrapy.spiderloader
import scrapy.statscollectors
import scrapy.logformatter
import scrapy.dupefilters
import scrapy.squeues

import scrapy.extensions.spiderstate
import scrapy.extensions.corestats
import scrapy.extensions.telnet
import scrapy.extensions.logstats
import scrapy.extensions.memusage
import scrapy.extensions.memdebug
import scrapy.extensions.feedexport
import scrapy.extensions.closespider
import scrapy.extensions.debug
import scrapy.extensions.httpcache
import scrapy.extensions.statsmailer
import scrapy.extensions.throttle

import scrapy.core.scheduler
import scrapy.core.engine
import scrapy.core.scraper
import scrapy.core.spidermw
import scrapy.core.downloader

import scrapy.downloadermiddlewares.stats
import scrapy.downloadermiddlewares.httpcache
import scrapy.downloadermiddlewares.cookies
import scrapy.downloadermiddlewares.useragent
import scrapy.downloadermiddlewares.httpproxy
import scrapy.downloadermiddlewares.ajaxcrawl
import scrapy.downloadermiddlewares.decompression
import scrapy.downloadermiddlewares.defaultheaders
import scrapy.downloadermiddlewares.downloadtimeout
import scrapy.downloadermiddlewares.httpauth
import scrapy.downloadermiddlewares.httpcompression
import scrapy.downloadermiddlewares.redirect
import scrapy.downloadermiddlewares.retry
import scrapy.downloadermiddlewares.robotstxt

import scrapy.spidermiddlewares.depth
import scrapy.spidermiddlewares.httperror
import scrapy.spidermiddlewares.offsite
import scrapy.spidermiddlewares.referer
import scrapy.spidermiddlewares.urllength

import scrapy.pipelines

import scrapy.core.downloader.handlers.http
import scrapy.core.downloader.contextfactory

# 自己项目用到的
# import scrapy.pipelines.images  # 用到图片管道
import re  # 用到openpyxl库

#调用FEED功能
s = get_project_settings()  #获取到配置项
s['FEED_FORMAT'] = 'csv'
# s['LOG_LEVEL'] = 'INFO'
s['FEED_URI'] = 'output.csv'
# s['LOG_FILE'] = 'Q1.log'
# process = CrawlerProcess({
   
#     'FEED_URI':'output.csv'
# })

# process = get_project_settings()
process = CrawlerProcess(s)
# cmdline.execute(("scrapy crawl spider -a url="+str(sys.argv[1])+" -o output.csv").split())
# cmdline.execute(("scrapy runspider /gitIssue/spiders/spider.py -a url="+str(sys.argv[1])+" -o output.csv").split())
process.crawl('spider', url=str(sys.argv[1]))

process.start()  # the script will block here until the crawling is finished

开始修改配置参数

1、修改settings.py文件

settings文件是scrapy的主要配置文件，修改项：
1、ROBOTSTXT_OBEY = False：ROBOTSTXT是网站上的爬虫协议，即在ROBOTS.txt文件中的网站数据不允许爬取，在此我们直接不遵守协议。
2、CONCURRENT_REQUESTS = 1：同时发起的请求数量，影响到爬虫的速度，爬取速度越快（也受其它很多参数影响），所需IP代理池越大，不然容易导致IP被封。
3、DOWNLOAD_DELAY = 2：每次请求后，停留的时间，如果不需要，直接注释掉即可。
4、DOWNLOADER_MIDDLEWARES：启用下载中间件，如果没有自己实现的中间件，可以直接注释掉。
5、ITEM_PIPELINES：管道配置，同4/
6、DOWNLOAD_TIMEOUT：可选参数，设置网页请求的超时时间/
7、RETRY_ENABLED = True：可选启用超时链接重试。
8、RETRY_TIMES = 4：重试次数。

BOT_NAME = 'gitIssue'

SPIDER_MODULES = ['gitIssue.spiders']
NEWSPIDER_MODULE = 'gitIssue.spiders'
ROBOTSTXT_OBEY = False
CONCURRENT_REQUESTS = 1
DOWNLOAD_DELAY = 2

RETRY_ENABLED = True
DOWNLOAD_TIMEOUT = 18
RETRY_TIMES = 4

DOWNLOADER_MIDDLEWARES = {
   
   'gitIssue.middlewares.RandomUA': 543,
    # 'gitIssue.middlewares.ProxyMiddleware': 542,
}

ITEM_PIPELINES = {
   
   'gitIssue.pipelines.GitissuePipeline': 300,
}

2、middlewares.py文件

在文件中，可以自定义user_agent池和IP_proxy池，也可以在其它地方加入。代码直接加在文件最末尾即可，同时，注意如果要启用功能，必须在settings文件中开启DOWNLOADER_MIDDLEWARES。

RandomUA()是自定义的User-agent代理，ProxyMiddleware为自定义的IP代理池，IP代理只需要模仿格式填充到list中就可以。
管道中间件代码:

import random
class RandomUA():
    def __init__(self):
        self.user_agent = [
             "Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; Nexus S Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)",
            "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.0 Safari/532.5",
            "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/532.9 (KHTML, like Gecko) Chrome/5.0.310.0 Safari/532.9",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7",
            "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.601.0 Safari/534.14",
            "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.601.0 Safari/534.14",
            "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.27 (KHTML, like Gecko) Chrome/12.0.712.0 Safari/534.27",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.24 Safari/535.1",
            "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7",
            "Mozilla/5.0 (Windows; U; Windows NT 6.0 x64; en-US; rv:1.9pre) Gecko/2008072421 Minefield/3.0.2pre",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10",
            "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)",
            "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB5",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0E)",
            "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
            "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
            "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0",
            "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2) Gecko/20110622 Firefox/6.0a2",
            "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
            "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b4pre) Gecko/20100815 Minefield/4.0b4pre",
            "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0 )",
            "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)",
            "Mozilla/5.0 (Windows; U; Windows XP) Gecko MultiZilla/1.6.1.0a",
            "Mozilla/2.02E (Win95; U)",
            "Mozilla/3.01Gold (Win95; I)",
            "Mozilla/4.8 [en] (Windows NT 5.1; U)",
            "Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko Netscape/7.1 (ax)",
            "HTC_Dream Mozilla/5.0 (Linux; U; Android 1.5; en-ca; Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.2; U; de-DE) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/234.40.1 Safari/534.6 TouchPad/1.0",
            "Mozilla/5.0 (Linux; U; Android 1.5; en-us; sdk Build/CUPCAKE) AppleWebkit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 1.5; en-us; htc_bahamas Build/CRB17) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 2.1-update1; de-de; HTC Desire 1.19.161.5 Build/ERE27) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 1.5; de-ch; HTC Hero Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 2.1; en-us; HTC Legend Build/cupcake) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 1.5; de-de; HTC Magic Build/PLAT-RC33) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 FirePHP/0.3",
            "Mozilla/5.0 (Linux; U; Android 1.6; en-us; HTC_TATTOO_A3288 Build/DRC79) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 1.0; en-us; dream) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
            "Mozilla/5.0 (Linux; U; Android 1.5; en-us; T-Mobile G1 Build/CRB43) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari 525.20.1",
            "Mozilla/5.0 (Linux; U; Android 1.5; en-gb; T-Mobile_G2_Touch Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Droid Build/FRG22D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Milestone Build/ SHOLS_U2_01.03.1) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.0.1; de-de; Milestone Build/SHOLS_U2_01.14.0) AppleWebKi