使用etree与xpath爬取Discuz论坛

引入模块

在pycharm中下载lxml库
通过from lxml import etree引入模块

测试

import requests
from lxml import etree
url = "https://www.discuz.net/forum-developer-1.html"
text = requests.get(url).text
html = etree.HTML(text)
context = etree.tostring(html).decode()
print(html.xpath('//div[@id="threadlist"]/div[2]/form/table/*'))
print(html.xpath('//*[@id="threadlisttableid"]/*'))

这里是通过xpath语法将论坛的所有tbody便签

print(html.xpath('//tbody/tr/th/a[@class="s xst"]/text()'))