爬虫学习（二）

任务：
提取 丁香园论坛的回复内容

1. beautifulsoup

基础语法：
参考文档
代码：

import urllib.request
from bs4 import BeautifulSoup as bs
def main():
  headers = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.2)Gecko/2008070208 Firefox/3.0.1',  'Accept': '*/*', 'Connection': 'keep-alive'}
  url = 'http://www.dxy.cn/bbs/thread/626626'
  request = urllib.request.Request(url, headers=headers)
  response = urllib.request.urlopen(request).read().decode("utf-8","ignore")
  html = bs(response, 'lxml')
  for data in html.find_all("tbody"):
      try:
          userid = data.find("div", class_="auth").get_text(strip=True)
          content = data.find("td", class_="postbody").get_text(strip=True)
          print(userid)
          print(content)
      except:
          pass
if __name__ == '__main__':
  main()

结果：

2. xpath

基础语法：
参考文档
代码：

import urllib.request
from lxml import etree
headers = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.2)Gecko/2008070208 Firefox/3.0.1',  'Accept': '*/*', 'Connection': 'keep-alive'}
url = 'http://www.dxy.cn/bbs/thread/626626'
request = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(request).read().decode("utf-8","ignore")
page = etree.HTML(response)
user = page.xpath("//div[@class='auth']/a/text()")
content=page.xpath('//td[@class="postbody"]')
for i in range(0,len(user)):
   	 print(user[i].strip()+":"+content[i].xpath('string(.)').strip())

结果：