webmagic 在Linux的使用

1.安装Chrom和配置chromDriverPath

1、安装chrome

用下面的命令安装Google Chrome

yum install https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

也可以先下载至本地，然后安装

wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

yum install ./google-chrome-stable_current_x86_64.rpm

安装必要的库

yum install mesa-libOSMesa-devel gnu-free-sans-fonts wqy-zenhei-fonts

2、安装 chromedriver（末尾附chrome和chromedriver的对应版本）

chrome官网

wget https://chromedriver.storage.googleapis.com/2.38/chromedriver_linux64.zip

淘宝源（推荐）

wget http://npm.taobao.org/mirrors/chromedriver/2.41/chromedriver_linux64.zip

将下载的文件解压，放在如下位置

unzip chromedriver_linux64.zip

mv chromedriver /usr/bin/

给予执行权限

chmod +x /usr/bin/chromedriver

2.webmagic运行报错

  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 3.10.0-957.27.2.el7.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 200 milliseconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'instance-2uqp40yz', ip: '192.168.0.6', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-957.27.2.el7.x86_64', java.version: '1.8.0_221'
Driver info: driver.version: ChromeDriver

执行google_master报错

[32753:32753:0722/142930.026945:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.

提示要增加 --no-sandbox

//webmagic默认会打开浏览器 关闭浏览器
ChromeOptions chromeOptions=new ChromeOptions();
chromeOptions.addArguments("--no-sandbox");