Selenium+ChromeDriver:可以更方便的爬取网站主页,而不用考虑加密抓包之类的内容。但也有其缺点,工作效率不如requests快。而且在Linux无面板主机部署起来略麻烦。
部分操作摘于
先切换阿里源
cd /etc/yum.repos.d/
yum -y install wget
wget http://mirrors.aliyun.com/repo/Centos-7.repo
mv CentOS-Base.repo CentOS-Base.repo.bak
mv Centos-7.repo CentOS-Base.repo
yum clean all
yum makecache
yum update
安装Python3
cd /usr/src
mkdir python3
cd python3/
yum -y install wget
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gcc make
yum -y install libffi-devel
wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz
tar -zxvf Python-3.7.3.tgz
rm Python-3.7.3.tgz -rf
cd Python-3.7.3
./configure
make && make install
安装selenium+chromedirvier的环境
mkdir sel
cd sel/
#下载Chrome
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
#安装Chrome
yum install ./google-chrome-stable_current_x86_64.rpm
#查看chrome的版本
google-chrome --version
#安装Chromedirver,需要找到和你chrome相匹配的chromedriver ,下载地址:http://chromedriver.chromium.org/downloads
wget http://npm.taobao.org/mirrors/chromedriver/75.0.3770.90/chromedriver_linux64.zip
#解压chromedirver
yum install -y unzip
unzip chromedriver_linux64.zip
#为chromedriver授权
chmod 755 chromedriver
#测试代码
yum -y install vim
vim test.py
#其中excutable_path是chromedriver的路径
```python
from selenium import webdriver
option = webdriver.ChromeOptions()
option.add_argument('--no-sandbox')
option.add_argument('--headless')
driver = webdriver.Chrome(executable_path='/root/sel/chromedriver', chrome_options=option)
driver.get('http://www.baidu.com')
print(driver.page_source)
```
#代码测试
pip3 install requests selenium -i https://pypi.tuna.tsinghua.edu.cn/simple/
pip3 install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple/
python3 test.py
#上传下载
yum -y install lrzsz unzip
代码:爬取东方财富网的评论,默认为
‘’’
hk:00700 , 02331 , 01810 腾讯、李宁、小米
us:baba , BIDU , NKE , googl 阿里、百度、耐克、谷歌
‘’’
其中 ‘hk’ 表示港股,00700 表示腾讯
import re
import requests
from selenium import webdriver
import os
exc_path='/root/sel/chromedriver'
def get_html(url):
global exc_path
option = webdriver.ChromeOptions()
option.add_argument('--no-sandbox')
option.add_argument('--headless')
driver = webdriver.Chrome(executable_path=exc_path