Scrapy
pillowss
这个作者很懒,什么都没留下…
展开
-
scrapy运行多个爬虫
from scrapy.utils.project import get_project_settingsfrom scrapy.crawler import CrawlerProcessdef main(): setting = get_project_settings() process = CrawlerProcess(setting) didntWorkSpider = ['sample'] for spider_name in process.spiders原创 2022-03-14 18:20:39 · 1177 阅读 · 0 评论 -
修改gerapy_selenium 添加虚拟显示浏览器
import timefrom scrapy.http import HtmlResponsefrom scrapy.utils.python import global_object_namefrom selenium.common.exceptions import TimeoutExceptionfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.wait import WebDriverWai原创 2022-01-21 18:45:58 · 7923 阅读 · 0 评论 -
docker 构建scrapyd
docker 构建scrapydDockerfile`FROM selenium/standalone-chrome:85.0-chromedriver-85.0-20200907USER rootRUN apt-get update && \ apt-get install -y xvfb && \ apt-get install -y python3-distutils && \ curl https://bootstrap.pyp原创 2022-01-21 18:37:54 · 699 阅读 · 0 评论 -
scrapyd设置访问密码
https://www.cnblogs.com/angdh/p/12298000.html原创 2021-08-04 18:24:39 · 350 阅读 · 0 评论 -
scrapy-redis发送post请求
def make_request_from_data(self, data): """ 重写make_request_from_data方法,data是scrapy-redis读取redis中的[url,form_data,meta],然后发送post请求 :param data: redis中都去的请求数据,是一个list :return: 一个FormRequest对象 """ request_type ...原创 2021-06-23 17:08:00 · 308 阅读 · 0 评论 -
gerapy自动批量部署脚本
'''这是一个自动部署zip包的脚本,遍历当前文件夹内所有zip文件,并部署到服务器ip:服务器ipAuthorization:认证的token'''import requestsimport osimport jsonimport timeip = 'http://127.0.0.1:8000'headers = { 'Authorization': "Token 123456789123456789", 'Proxy-Connection': 'keep-alive原创 2021-05-21 10:35:26 · 147 阅读 · 0 评论 -
scrapyd官方API
文章目录daemonstatus.jsonaddversion.jsonschedule.jsoncancel.jsonlistprojects.jsonlistversions.jsonlistspiders.jsonlistjobs.jsondelversion.jsondelproject.jsondaemonstatus.json检查服务的加载状态GETcurl http://localhost:6800/daemonstatus.json{ “status”: “o原创 2020-12-28 17:47:59 · 192 阅读 · 0 评论 -
Scrapy的安装
1、首先安装wheel# 安装依赖包pip install wheel2、然后下载驱动https://www.lfd.uci.edu/~gohlke/pythonlibs/下载网址(在最后面)下载好后放到桌面,Shift cmd命令符窗口pip install Twisted-17.9.0-cp36-cp36m-win32.whl安装3、安装pywin32pip install pywin324、安装pip install Scrapypip install Scra原创 2020-11-05 12:18:18 · 79 阅读 · 0 评论