Install and cofigure Scrapy under Centos7

How to set up the environment for Scrapy on Centos 7 in generic way.

For a new centos, you need to install pip tool first with yum.

yum install python-pip

install virtualenv with pip

pip install virtualenv

Create a new vitualenv for python with no site packages, then activate it.

virtualenv –no-site-packages {{v1 dir name}}
source bin/activate

Intall all packages with several pip command

pip install scrapy
pip install requests
pip install scrapy-splash
pip install selenium

or

pip install -r requirements.txt

requirements.txt:

Scrapy==1.3.2
requests==2.13.0
scrapy-splash==0.7.1
selenium==3.0.2

Webdriver needed for some spiders

1. 
   
2. Download the webdriver
3. Make sure it in the right place.
   

on windows
put the webdriver.exe to Path environment
on linux
mv /webdriver /usr/bin