云主机下搭建网页爬虫

1. 相关组件

OS：腾讯云Ubuntu

浏览器：chrome + chromedriver

python爬虫框架： selenium

2.安装部署

2.1 安装chrome

sudo apt-get install libxss1 libappindicator1 libindicator7

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

sudo apt install ./google-chrome*.deb

google-chrome-stable --version

google-chrome  --headless --disable-gpu  https://www.baidu.com/

2.2 安装chromedriver

wget https://storage.googleapis.com/chrome-for-testing-public/123.0.6312.58/linux64/chromedriver-linux64.zip

unzip chromedriver-linux64.zip

cp chromedriver /usr/local/bin/

2.3 安装python3

略

2.4 安装selenium

pip install selenium

2.5 编写爬虫程序

from pyvirtualdisplay import Display
from selenium import webdriver

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')#无头模式，服务器没有图形界面这个必须
chrome_options.add_argument('--disable-gpu')#不需要gpu加速
chrome_options.add_argument('--no-sandbox') # 这个配置很重要
#s = Service('/usr/local/bin/chromedriver')
client = webdriver.Chrome(options=chrome_options)    # 如果没有把chromedriver加入到PATH中，就需要指明路径

#client.get("https://ec.chng.com.cn/channel/home/?SlJfApAfmEBp=1704119704734#/")
#client.get("https://ec.chng.com.cn/channel/home/?SlJfApAfmEBp=1704119704734#/detail?id=12542364")
client.get("https://www.zhihu.com/question/553813879/answer/3440060321")
print (client.page_source.encode('utf-8'))
 
client.quit()