python2.7.x+requests+正则表达式爬取《糗事百科》段子

糗事百科url为https://www.qiushibaike.com/text/page/1/
其中后面的/1/表示第几页，这里只爬取前面7页，具体看下面的代码。
（在chrome浏览器中，利用开发者工具，查看到的元素和在pycharm中requests.get请求下来的网页不一样，标签变了，导致按照视频教程的正则匹配不到。于是，自己在pycharm中查看标签，重新写正则匹配式。）

#!/usr/bin/env python 
# -*- coding:utf-8 -*-

import requests
import re

num = 0
def parse_page(url):
    headers = {
        'User-Agent': "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) "
                      "Chrome/72.0.3610.2 Mobile Safari/537.36",
        'Referer': "https://www.qiushibaike.com/text/"
    }
    response = requests.get(url, headers=headers)
    text = response.text
    #print(text)

    # contents = re.findall(r'<div\sclass="content">.*?<span>\s*(.*?)\s*</span>', text, re.DOTALL)
    contents = re.findall(r'<a href="/article/.*?>\s*(.*?)\s*</a>', text, re.DOTALL)

    #print(len(contents))
    duanzi = []
    for content in contents:
        content = content.replace('<br/>', '')
        content = re.sub(r'<span class="fullArtitle" >.*?</span>', '', content) #去掉“查看全文”
        content = re.sub(r'<div([\s\S]*)</div>', '', content)  # 去掉多余的div
        content = content.strip()
        print(content)
        duanzi.append(content)

        global num
        num = num+1

def main():
    base_url = 'https://www.qiushibaike.com/text/page/{}/'
    for x in range(1, 8):
        url = base_url.format(x)
        parse_page(url)
    print(num)

if __name__ == '__main__':
    main()

使用正则表达式匹配任意字符包括空格和换行符，参考：
https://my.oschina.net/zchuanzhao/blog/849948