selenium+chromedriver部署到云主机并爬取东方财富网的股票评论（一）

Selenium+ChromeDriver:可以更方便的爬取网站主页，而不用考虑加密抓包之类的内容。但也有其缺点，工作效率不如requests快。而且在Linux无面板主机部署起来略麻烦。

部分操作摘于

https://blog.csdn.net/qq_32502511/article/details/85245455

先切换阿里源

  cd  /etc/yum.repos.d/
  yum -y install wget
  wget  http://mirrors.aliyun.com/repo/Centos-7.repo
  mv  CentOS-Base.repo CentOS-Base.repo.bak
  mv Centos-7.repo CentOS-Base.repo
  yum clean all
  yum makecache
  yum update

---

安装Python3

   cd /usr/src
   mkdir python3
   cd python3/
   yum -y install wget
   yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gcc make
   yum -y install libffi-devel 
   wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz
   tar -zxvf Python-3.7.3.tgz 
   rm Python-3.7.3.tgz -rf
   cd Python-3.7.3
   ./configure 
   make && make install

---

安装selenium+chromedirvier的环境

mkdir sel 
cd sel/
#下载Chrome
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

#安装Chrome
yum install ./google-chrome-stable_current_x86_64.rpm

#查看chrome的版本
google-chrome --version

#安装Chromedirver，需要找到和你chrome相匹配的chromedriver ，下载地址：http://chromedriver.chromium.org/downloads
wget http://npm.taobao.org/mirrors/chromedriver/75.0.3770.90/chromedriver_linux64.zip

#解压chromedirver
yum install -y unzip
unzip chromedriver_linux64.zip

#为chromedriver授权
chmod 755 chromedriver

#测试代码
yum -y install vim
vim test.py

#其中excutable_path是chromedriver的路径
```python
from selenium import webdriver
option = webdriver.ChromeOptions()
option.add_argument('--no-sandbox')  
option.add_argument('--headless')  
driver = webdriver.Chrome(executable_path='/root/sel/chromedriver', chrome_options=option)
driver.get('http://www.baidu.com')
print(driver.page_source)
```

#代码测试
pip3 install requests selenium -i  https://pypi.tuna.tsinghua.edu.cn/simple/
pip3 install --upgrade pip -i  https://pypi.tuna.tsinghua.edu.cn/simple/
python3 test.py 


#上传下载
yum -y install lrzsz unzip 

代码：爬取东方财富网的评论，默认为

‘’’
hk:00700 , 02331 , 01810 腾讯、李宁、小米
us:baba , BIDU , NKE , googl 阿里、百度、耐克、谷歌
‘’’

其中 ‘hk’ 表示港股，00700 表示腾讯

import re
import requests
from selenium import webdriver
import os

exc_path='/root/sel/chromedriver'

def get_html(url):
    global exc_path
    option = webdriver.ChromeOptions()
    option.add_argument('--no-sandbox')
    option.add_argument('--headless')
    driver = webdriver.Chrome(executable_path=exc_path