python爬取糗事百科的标题和内容

这篇文章基于python3来编写，这里使用来xpath来解析数据，由于糗事百科的反爬机制，这里我们需要加入header信息，我认为最主要的就是解析数据这块，我推荐https://www.cnblogs.com/gaochsh/p/6757475.html这个博客，博主由浅入深的解释来如何来使用xpath来获取我们需要的节点，在xpath中，返回的是一个元素，我们可以继续对这个元素进行xpath解析，这里主要是方法的简要介绍，就只提取来一个页面的内容

import requests
from lxml import etree
from bs4 import BeautifulSoup
import re
page=1
url = 'http://www.qiushibaike.com/hot/page/' + str(page)
headers={
'Cookie':'BIDUPSID=944A139885EC4A8CDAC4B9278AAA9E23; PSTM=1514534756; BAIDUID=9A3C4DD0DB98A4A48B7720E055B378E6:FG=1; BDUSS=XpsRERJeUJCaFVWQWZxTXZ2c3lRMTQ3RlVwa3h4UHV6TG14Y3RKd3QzM0M4SmhhQUFBQUFBJCQAAAAAAAAAAAEAAAAiyDo1aG9tZdDEu6jFrbfFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMJjcVrCY3Fac; MCITY=-75%3A; H_PS_645EC=290aIE41J1QwHtZSUPAQjr16T%2Bz3Un20Ght9uT4Ww7CXkDceENpmFfbfob0ApxI0oZbX; BD_CK_SAM=1; PSINO=2; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; BD_HOME=1; H_PS_PSSID=25314_1443_24565_21097_17001_20928; BD_UPN=1a314353; sugstore=1; H_WISE_SIDS=121076_110316_108267_122159_102431_100098_120139_110772_120009_118888_118869_118854_118837_118805_122187_107314_121255_121533_121924_121215_117331_121862_117437_121666_121561_120943_121042_122169_122138_121617_120852_121465_121307_120262_116407_110085_122021; bd_traffictrace=031431; BDSVRTM=402; plus_lsv=90c10fea240be0ef; plus_cv=1::m:11f40515; Hm_lvt_12423ecbc0e2ca965d84259063d35238=1517623328,1517636910,1517639481; Hm_lpvt_12423ecbc0e2ca965d84259063d35238=1517639481; SE_LAUNCH=5%3A25293991_0%3A25293991_3%3A25293991',
	'User - Agent': 'Mozilla / 5.0(Macintosh;IntelMacOSX10_13_1) AppleWebKit / 537.36(KHTML, likeGecko) Chrome / 63.0.3239.132Safari / 537.36,'
	}
html = requests.get(url=url,headers=headers).text
selector = etree.HTML(html)
list=selector.xpath('//div[@id="content-left"]/div')
for l in list:
	print("name"+str(l.xpath('div/a/h2/text()')).replace("\\n",""))
	print("content"+str(l.xpath('a/div/span