最近一直想学网页爬虫,相信很多人跟我一样选择了北京理工大学的教程,个人感觉老师讲的还不错,毕竟每个视频的时间不过6分钟左右吧。在单元6中,老师详解了实例《中国大学排名爬虫》,我按照老师的代码,执行时出现typeerror,现将修改后的代码附上:
import requests from bs4 import BeautifulSoup import bs4 def getHTMLText(url): try: r = requests.get(url, timeout = 30) r.raise_for_status() r.encoding = r.apparent_encoding return r.text except: return "" def fillUnivList(ulist, html): soup = BeautifulSoup(html,"html.parser") for tr in soup.find('tbody').children: if isinstance(tr, bs4.element.Tag): tds = tr('td') ulist.append([tds[0].contents[0], tds[1].string, tds[2].string, tds[3].string]) def printUnivList(ulist, num): print("{:^10}\t{:^6}\t{:^10}\t{:^10}".format("排名","学校名称","省份","总分")) for i in range(num): u = ulist[i] print("{:^10}\t{:^6}\t{:^10}\t{:^10}".format(u[0], u[1], u[2], u[3])) def main(): uinfo = [] url = 'http://www.zuihaodaxue.com/zuihaodaxuepaiming2017.html' html = getHTMLText(url) fillUnivList(uinfo, html) printUnivList(uinfo, 30) main() |
代码执行结果如下所示:
(顺便吐槽一下,学校排名又下降了55555)