天涯python_用python做了个简易的天涯脱水功能

大学时，喜欢上课猫在后排看天涯煮酒版块连载的历史文章，那时还在用nokia的功能机，只能把天涯上一页一页地把作者发表内容抠下来，保存为txt，导到手机中在里面看。后来知道了天涯脱水版，就是免费提供“只看楼主”功能，抠起来就方便多了。两三年下来也看了几十篇。。。

前两天在b站看到python的爬虫教程，照着做了一个，缅怀当时那个上了大学不认真听课也没心思泡妞的少年

源码如下：用python3.7.3 导入了requests模块，如果没有需要安装一下：pip install requests

运行后输入天涯论坛主贴的网址，比如：http://bbs.tianya.cn/post-no05-471093-1.shtml，就会在当前文件夹下建立新的文件夹，然后把每一页的楼主回复保存下来。没回复就不保存。

# -*- coding: UTF-8 -*-

import requests

import re

import os

# http://bbs.tianya.cn/post-no05-471093-1.shtml

paper_url = str(input('请输入天涯社区主贴网址：'))

def get_page_count(paper_url):  # 获取页面数量

response = requests.get(paper_url)

response.encoding = 'utf-8'

result = response.text

reg = r'pageCount : (.*?),'

page_count = re.findall(reg, result)

# print(page_count)

return int((page_count[0]))

def get_paper_name(paper_url):  # 获取文章名称+版块名称，建立文件夹用

response = requests.get(paper_url)

response.encoding = 'utf-8'

result = response.text

reg = r'

(.*?)_论坛_天涯社区'

paper_name = re.findall(reg, result)[0]

paper_name = paper_name.replace(' ', '_') #  文件夹名有空格情况下mkdir会不正常，用下划线替代

# print(paper_name)

return paper_name

def get_author_id(paper_url):  # 获取作者ID，提取作者回复的内容用

response = requests.get(paper_url)

response.encoding = 'utf-8'

result = response.text

reg = r'authorId : (.*?),'

author_id = re.findall(reg, result)

return author_id[0]

def get_author_content(paper_url):

page_count = get_page_count(paper_url)

author_id = get_author_id(paper_url)

paper_name = get_paper_name(paper_url)

os.system(r'mkdir' + r' ' + paper_name)

page_url = paper_url[0:-7]  # http://bbs.tianya.cn/post-no05-471093- 截到这里

for i in range(1, page_count + 1):  # 遍历所有页面

page_num = str(i)

page_url_i = page_url + page_num + '.shtml'  # 拼凑为第i个页面

response = requests.get(page_url_i)

response.encoding = 'utf-8'

result = response.text

reg = r'hostid=' + author_id + r'.*?

(.*?)

author_bbs_content = re.findall(reg, result, re.S)

if author_bbs_content != []:  # 本页作者有回复则保存

print('正在保存第%s页数据' % page_num)

# 保存第i个页面为html

fn = open('%s\\%s.html' % (paper_name, page_num), 'w', encoding='utf-8')

for j in author_bbs_content:

fn.write(j)

fn.close

get_author_content(paper_url)