redhat6.5部署python爬虫项目（三）

在项目同步到服务器后开始准备安装项目需要的各种python包。

第一种方式（最普通）：

在scrapy项目中的Terminal下生成requirements.txt文件

pip freeze > requirements.txt

然后更新同步到服务器。

创建本次项目的虚拟环境

which virtualenv

//如果没有就安装

pip3 install virtualenv

//没有venvs文件就创建个
// 关于最后面这个python，如果你把python3设置为默认python，就按照下面代码。如果没有记得加上python3或者你自己设置的名称。（cnki是我的项目，需按照自己的修改）
virtualenv /root/venvs/cnki -p python

//创建环境
source /root/venvs/cnki/bin/activate

如果前面有（cnki）字样说明创建成功

(cnki) [root@oracle11g cnki_spider]#

然后执行安装requirements.txt内的包

pip3 install -r requirements.txt

安装完毕后执行项目启动代码。

 

补充：

运行后遇到报错：

(cnki) [root@oracle11g cnki_spider]# scrapy crawl cnki
Traceback (most recent call last):
  File "/root/venvs/cnki/bin/scrapy", line 8, in <module>
    sys.exit(execute())
  File "/root/venvs/cnki/lib/python3.7/site-packages/scrapy/cmdline.py", line 144, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/root/venvs/cnki/lib/python3.7/site-packages/scrapy/crawler.py", line 280, in __init__
    super().__init__(settings)
  File "/root/venvs/cnki/lib/python3.7/site-packages/scrapy/crawler.py", line 152, in __init__
    self.spider_loader = self._get_spider_loader(settings)
  File "/root/venvs/cnki/lib/python3.7/site-packages/scrapy/crawler.py", line 146, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/root/venvs/cnki/lib/python3.7/site-packages/scrapy/spiderloader.py", line 67, in from_settings
    return cls(settings)
  File "/root/venvs/cnki/lib/python3.7/site-packages/scrapy/spiderloader.py", line 24, in __init__
    self._load_all_spiders()
  File "/root/venvs/cnki/lib/python3.7/site-packages/scrapy/spiderloader.py", line 51, in _load_all_spiders
    for module in walk_modules(name):
  File "/root/venvs/cnki/lib/python3.7/site-packages/scrapy/utils/misc.py", line 89, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/python3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/cnki_spider/cnki_spider_master/spiders/cnki_journal_cn.py", line 11, in <module>
    from cnki_spider_master.items import *
  File "/usr/local/cnki_spider/cnki_spider_master/items.py", line 7, in <module>
    from models import *
ModuleNotFoundError: No module named 'models'

解决办法：

在models前面加上一个点即可