Python获取网页动态内容 js渲染后的内容

2019独角兽企业重金招聘Python工程师标准>>>

import time
from selenium import webdriver


def getHtml(url, loadmore = False, waittime = 2):
    browser = webdriver.Chrome('chromedriver')
    browser.get(url)
    time.sleep(waittime)
    if loadmore:
        while True:
            try:
                next_button = browser.find_element_by_class_name("more")
                next_button.click()
                time.sleep(waittime)
            except:
                break
    html = browser.page_source
    browser.quit()
    return html

print(getHtml('https://www.oschina.net/',True,20))

 

此处说明下：cromedriver需要下载，然后放到Python的根目录，才能使用

下载地址： https://pan.baidu.com/s/1QHgqaNddumna-2UVMJagRg

 

转载于:https://my.oschina.net/xsh1208/blog/1837260