Python爬虫进阶——爬取数据js动态加载的网页

上一篇章，讲解了python简单爬取网页静态数据的方法。
python爬虫小白
今天，来讲讲如何用python爬取——数据用js动态加载的网页。所用到的库selenium以及与它搭配的webdriver。这两者用来打开浏览器，模拟浏览器从网页中读取数据的过程，这样才能找到数据。

代码如下：

from selenium import webdriver
import time
import xlwt
import datetime

'''get the url of the aim'''
url = '初始网址’

browser = webdriver.Chrome('/Users/kakufumisakai/Downloads/chromedriver')
browser.get(url)
time.sleep(10)

html_text = browser.page_source

time.sleep(5)

#get all elements that their css style contian class = 'extra-header-right'
elements = browser.find_elements_by_css_selector("[class = 'extra-header-right']")
#get the aim
element = elements[1]
browser.execute_script("arguments[0].click()", element)
#get the url of the aim
aimurl = browser.current_url

'''drop the scroll bar'''
js = "return action = document.body.scrollHeight"
height = browser.execute_script(js)

browser.execute_script('window.scrollTo(0, document.body.scrollHeight)')
time.sleep(5)

dates = browser.find_elements_by_xpath("//uni-view[@class = 'item item']/uni-view[@class = 'time']")

status = True

#Number
num = 0
#the position of starting index
n = 0

#judge the last date in one page is more than 5, or less than 5. former, flag = 1, else flag = 0
flag = 0

while status:
    for date in