学习资料参考:Python网络爬虫实战
爬虫对象:新浪新闻
主要练习简单的爬新闻标题,时间,内容,评论数等,并将其制作成Excel表格。
源程序如下:
import requests
import json
from bs4 import BeautifulSoup
import pandas
results = []
zturl='http://api.roll.news.sina.com.cn/zt_listchannel=new&cat_1=gnxw&cat_2==gdxw1||=gatxw||=zspl||=mtjj&level==1||=2&show_ext=1&show_all=1&show_num=22&tag=1&format=json&page=1&callback=newsloadercallback&_=1504319383856'
res = requests.get(zturl)
jd = json.loads(res.text.lstrip(' newsloadercallback(').rstrip(');'))
def getComment(news_url):
newsid = news_url.split('/')[-1].lstrip('doc-i').rstrip('.shtml')
commentURL = 'http://comment5.news.sina.com.cn/page/info?version=1&\
format=js&channel=gn&newsid=comos-{}&group=&compress=0&ie=utf-8&oe=utf-8&\
page=1&page_size=20'.format(newsid)
res2 = requests.get(commentURL)
jd1 = json.loads(res2.text.lstrip('var data='))
return jd1['result']['count']['total']
for allnews in jd['result']['data']:
newsSum = {}
res1 = requests.get(allnews['url'])
res1.encoding = 'utf-8'
soup = BeautifulSoup(res1.text,'html.parser')
newstitle = allnews['title']
newsurl = allnews['url']
newsSum['time'] = soup.select('.timesource'[0].contents[0].strip()
newsSum['title'] = allnews['title']
newsSum['url'] = allnews['url']
#newsSum['article'] = soup.select('.article')[0].text
newsSum['editor'] = soup.select('.article-editor')[0].text.lstrip('责任编辑:')
newsSum['comment'] = getComment(allnews['url'])
newsSum['source'] = soup.select('.time-source a')[0]['href']
results.append(newsSum)
df = pandas.DataFrame(results)
df.to_excel('news.xlsx')
通过该程序,可以得到新浪新闻中新闻的标题,时间,作者,等讯息。并导出到‘news.xlsx’表格中。
当然,也会遇到一些问题,比如弹出: IndexError: list index out of range.则是因为原先的新闻链接被删除了,导致列表为空。等等