6.爬虫必备功能整理

设置proxy和headers

1. 抓取动态页面所需的selenium+PhantomJS

desired_capabilities = DesiredCapabilities.PHANTOMJS.copy()
desired_capabilities["phantomjs.page.settings.userAgent"] = r'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.3408.400 QQBrowser/9.6.12028.400'  
desired_capabilities["phantomjs.page.settings.loadImages"] = False #不加载图片
Proxy = webdriver.Proxy()
Proxy.proxy_type = ProxyType.MANUAL
Proxy.http_proxy = '219.130.39.55:53281' # 代理IP和端口号
Proxy.add_to_capabilities(desired_capabilities)  
browser = webdriver.PhantomJS()
browser.start_session(desired_capabilities)

2. 正在学习的request

head = {'User-Agent' : r'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.3408.400 QQBrowser/9.6.12028.400'}
    head['Referer'] = 'http://www.toutiao.com/ch/news_tech/'
    head['Connection'] = 'keep-alive'
    head['host'] = 'http://www.toutiao.com'

    proxie = {
        'http' : 'http://120.15.31.67:8118'
    }
    response = RS.get(url, headers = head, proxies = proxie)

3. 避免爬取过快的休眠时间

time.sleep(8)

待续