这个寒假闲来没事开始爬虫,学了几天了,期间自己爬了一些网站,现在大致整理一下:
import requests
import re
response = requests.get("http://www.shuquge.com/txt/96636/index.html")
# 万能设置编码
response.encoding = response.apparent_encoding
html = response.text
#正则表达式获取所有的章节目录和url网址
result = re.findall('<dd><a href="(.*?)">(.*?)</a></dd>',html)
print(result)
for url,name in result:
new_url = "http://www.shuquge.com/txt/96636/" + url
response1 = requests.get(new_url)
response1.encoding = response1.apparent_encoding
html = response1.text
result1 = re.findall('<div id="content" class="showtxt">(.*?)</div>',html,re.S)
# re.S是自动换行
with open(str(name) + ".txt", mode="w", encoding="utf-8") as f:
f.write(str(result1[0]).replace(" ","").replace("<br/>",""))
"""
print("http://www.shuquge.com/txt/96636/" + url)
print(name)
response1 = requests.get("http://www.shuquge.com/txt/96636/28767693.html")
response1.encoding = response1.apparent_encoding
html = response1.text
with open("1.html",mode="w",encoding="utf-8") as f:
f.write(html)
"""
爬取的是小说《抢救大明朝》,Pycharm截止到我现在运行这些代码还是可以的,大家可以看截图:
爬取最后的小说界面是:
这是爬虫第一天敲的案例!