电影页面:
https://movie.douban.com/subject/26363254
热评:
第一页:https://movie.douban.com/subject/26363254/comments?start=0&limit=20&sort=new_score&status=P
第n页:https://movie.douban.com/subject/26363254/comments?start=20*(n-1)&limit=20&sort=new_score&status=P
中文乱码了,查看当前网页的编码格式:
print requests.get(url).encoding
然后手动设置网页编码格式:
r.encoding = 'utf-8'
获取全部短评数量:
total_num = soup.find('li', class_="is-active").span.text.strip()
total_num = int(total_num[3:-1])
page_num = total_num / 20
last_num = total_num % 20
多线程
pool = ThreadPool(20)
#pool = multiprocessing.Pool(multiprocessing.cpu_count())#import multiprocessing
pool.map(get_content,(url for url in urls))#映射到主函数中进行循环
pool.close()
pool.join()
# 设置等待时间,避免爬取太快
import time
# 用于在超时的时候抛出异常,便于捕获重连
import socket
timeout = 3
socket.setdefaulttimeout(timeout)
# 超时重连
state = False
while not state:
try:
html = get_html(url)
state = True
except socket.timeout:
state = False
print('访问过快!超时重连')
#!/usr/bin/env python
# -*- coding: utf-