Python 3 爬虫 笔记:
安装环境(基于centos7.x)
安装Python 3.5 和 pip3
1. sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpm
2. yum update
3. yum install -y python35u python35u-libs ptyhon35u-devel python35u-pip
4. 验证是否安装成功,输入 python3.5 .....
安装 Anaconda
1. wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh
2. chmod +x Anaconda3-2019.10-Linux-x86_64.sh
3. bash Anaconda3-2019.10-Linux-x86_64.sh
4. 按照提示,按回车,输入 yes 等等
5. vi /root/.bashrc
6. export PATH="/root/anaconda3/bin:$PATH"
7. source /root/.bashrc
8. 验证是否安装成功,输入:conda list 列出一堆信息 安装成功
9. 输入 python 进入python界面会提示一些信息,看否是有 Anaconda字样
以上是python3 爬虫基本环境配置
-------------------------------------------------------------------------------------------
开发环境配置
1. 安装requests : pip3 install requests
2. 如果提示你的pip版本低,则需要输入命令来升级pip : pip install --upgrade pip
3. 再次安装requests库,执行第一条
4. 安装wheel: pip3 install wheel
5. wget https://files.pythonhosted.org/packages/29/b9/d26a6ab2ee178415ab8c0c591d2a1eb782a50c42a417ae390055f86a63c1/requests-2.17.3-py2.py3-none-any.whl
6. pip3 install requests-2.17.3-py2.py3-none-any.whl
7. 安装selenium: pip3 install selenium
8. 安装selenium的wheel: wget https://files.pythonhosted.org/packages/80/d6/4294f0b4bce4de0abf13e17190289f9d0613b0a44e5dd6a7f5ca98459853/selenium-3.141.0-py2.py3-none-any.whl
9. pip3 install selenium-3.141.0-py2.py3-none-any.whl
10. 验证是否OK,进入python3 命令行输入 import selenium 无输出基本OK
11. 安装ChromeDriver : wget https://chromedriver.storage.googleapis.com/76.0.3809.126/chromedriver_linux64.zip
12. mv chromedriver /usr/bin/
13. 验证是否OK: /usr/bin/chromedriver 如果监听了9515端口就代表正常
14. PhantomJS安装: wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
15. tar xvf phantomjs-2.1.1-linux-x86_64.tar.bz2
16. tar xvf phantomjs-2.1.1-linux-x86_64.tar.bz2
17. /home/phantomjs/bin/phantomjs
18. 安装AIOHTTP:pip3 install aiohttp
...........未完待续
这么多库,按的我想吐