selenium爬虫(chrome)

 

Ubuntu版本

安装google-chrome-stable过程

sudo wget http://www.linuxidc.com/files/repo/google-chrome.list -P /etc/apt/sources.list.d/

wget -q -O - https://dl.google.com/linux/linux_signing_key.pub  | sudo apt-key add -

apt-get update

apt-get install google-chrome-stable

如果报错：/etc/apt/sources.list.d/google-chrome.list文件

只留一个deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main

安装chromedriver(linux)

wget http://chromedriver.storage.googleapis.com/78.0.3904.70/chromedriver_linux64.zip

apt-get install unzip -y

unzip chromedriver_linux64.zip

sudo mv -f chromedriver /usr/local/share/chromedriver

sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver

安装selenium和数据库插件

apt-get install python-pip

pip install selenium

pip install MySQL-python

 

 import MySQLdb 和from selenium.webdriver.chrome.options import Options 导入无报错说明数据库和chrome可用

test.pt测试代码

# -*- coding:utf-8 -*-
import sys   
reload(sys)
sys.setdefaultencoding('utf8') 

from selenium import webdriver
from time import sleep
from selenium.webdriver import ActionChains

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from datetime import datetime
from datetime import timedelta
from time import sleep
# import xlwt
import logging
import os,shutil
import time
import datetime
from selenium.webdriver.chrome.options import Options

if __name__ == '__main__':
    # 初始化webdriver实例
    time_start=time.time()
    logging.getLogger().setLevel(logging.INFO)
    options = Options()
    options.set_headless()
    options.add_argument('--no-sandbox')
    options.add_argument('start-maximized')
    options.add_argument('disable-infobars')
    options.add_argument('--disable-extensions')
    driver = webdriver.Chrome(chrome_options=options)

    driver.get("https://baidu.com")
    print("-------------")
    driver.close()

python test.py运行不报错就说明安装成功

win版本

win安装chrome浏览器

http://chromedriver.storage.googleapis.com/78.0.3904.70/chromedriver_win32.zip

解压压缩包，找到chromedriver.exe复制到chrome的安装目录

如果要看有界面的需要去掉

options.set_headless()

pip install selenium

pip install MySQLdb 或pip install mysql-python

如果报错自行处理即可

可用test.py测试

 

后台运行

/root/start.sh

chmod a+x  /root/start.sh

nohup python /root/test/test.py > /root/test/test.log 2>&1 &

如果需要定时任务crontab-e

python /root/test/test.py需要python全路径

nohup /usr/bin/python /root/test/test.py > /root/test/test.log 2>&1 &

以上python环境是：python2.7

升级到python3.5

ll /usr/bin/python*

如果Ubuntu服务器中包含python3.5 直接把/usr/bin/python指向连接就可以了

rm -rf  /usr/bin/python

ln -s /usr/bin/python3.5  /usr/bin/python

python

安装pip3：

wget https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py

pip3 install selenium