爬虫实战

一:选择网站

https://www.xiaohua.com/duanzi?page=1

二:发现规律:

第一页:https://www.xiaohua.com/duanzi?page=1

第二页:https://www.xiaohua.com/duanzi?page=2

第三页:https://www.xiaohua.com/duanzi?page=3

三:编写代码

1,修改请求头        headers = {"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"}

2,代码实现:

import urllib.request
import urllib.parse
from lxml import etree
import ssl
import os

def urls(url,p):
	for x in range(1,int(p)):
		print(x)
		urls = url+str(x)
		print(urls)
		headers = {"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"}
		request = urllib.request.Request(urls, headers = headers)
		html = urllib.request.urlopen(request).read()
		content = etree.HTML(html)
		res = content.xpath('//div/p/a/text()')
		for y in res:
			with open("D:/xiaohua.txt", "a+",encoding = 'utf-8')as f:
				f.write(y)
				f.write('\n-----------------------------------------------\n')
				f.close
		print("第"+str(x)+"页话爬取完成")

def main():
	p = input("爬取到第几页:")
	url = 'https://www.xiaohua.com/duanzi?page='
	urls(url,p)



if __name__ == '__main__':
	main()

3,附图片