1.使用requests,和BeautifulSoup解析
import requests from bs4 import BeautifulSoup HEADERS = { "Referer": "https://movie.douban.com/top250", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/65.0.3325.181 Safari/537.36" } html = requests.get("https://movie.douban.com/top250?start=0&filter=") bsobj = BeautifulSoup(html.content, 'lxml') with open(r"C:\Users\jone\Desktop\douban.txt", 'w') as fs: fs.write(str(bsobj.contents).encode('gb18030').decode('gbk', 'ignore')) print("done")
2.问题解析:
1.模拟浏览器登陆,添加头信息
2.BeautifulSoup无法解析Response响应,创建BeautifulSoup时,应该解析html.content
3.将网页源码写入txt文件时候,将网页内容转化为字符串后,先encode成gb18030,再解码为gbk,选择忽略不能识别的字符。