在实现第一页内容和头像的获取后,想实现代码自动翻页获取后面内容。其原理就是找到下一页的链接,并且访问即可。主要是在第二个实例中加入部分代码
第二个实例
- 获取html内容
def get_info( url, data = None):
header = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, sdch',
'Accept-Language': 'zh-CN,zh;q=0.8',
'Connection': 'keep-alive',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36'
}
timeout = random.choice(range(60,180))
while True :
try:
rep = requests.get(url, headers = header, timeout = timeout)
rep.encoding = 'utf-8'
break
except socket.timeout as e:
print('3:',e)
time.sleep(random.choice(range(8,15)))
except socket.error as e:
print('4:'<