爬虫学习——爬虫之新浪新闻

学习资料参考：Python网络爬虫实战

爬虫对象：新浪新闻

主要练习简单的爬新闻标题，时间，内容，评论数等，并将其制作成Excel表格。

源程序如下：

import requests
import json
from bs4 import BeautifulSoup
import pandas

results = []

zturl='http://api.roll.news.sina.com.cn/zt_listchannel=new&cat_1=gnxw&cat_2==gdxw1||=gatxw||=zspl||=mtjj&level==1||=2&show_ext=1&show_all=1&show_num=22&tag=1&format=json&page=1&callback=newsloadercallback&_=1504319383856'

res = requests.get(zturl)
jd = json.loads(res.text.lstrip('  newsloadercallback(').rstrip(');'))

def getComment(news_url):
    newsid = news_url.split('/')[-1].lstrip('doc-i').rstrip('.shtml')
    commentURL = 'http://comment5.news.sina.com.cn/page/info?version=1&\
format=js&channel=gn&newsid=comos-{}&group=&compress=0&ie=utf-8&oe=utf-8&\
page=1&page_size=20'.format(newsid)

    res2 = requests.get(commentURL)

    jd1 = json.loads(res2.text.lstrip('var data='))
    return jd1['result']['count']['total']

for allnews in jd['result']['data']:
    newsSum = {}

    res1 = requests.get(allnews['url'])
    res1.encoding = 'utf-8'
    soup = BeautifulSoup(res1.text,'html.parser')

    newstitle = allnews['title']
    newsurl = allnews['url']                       
    newsSum['time'] = soup.select('.timesource'[0].contents[0].strip()  
    newsSum['title'] = allnews['title']
    newsSum['url'] = allnews['url']
    #newsSum['article'] = soup.select('.article')[0].text
    newsSum['editor'] = soup.select('.article-editor')[0].text.lstrip('责任编辑：')    
    newsSum['comment'] = getComment(allnews['url'])
    newsSum['source'] = soup.select('.time-source a')[0]['href']

    results.append(newsSum)

df = pandas.DataFrame(results)

df.to_excel('news.xlsx')

通过该程序，可以得到新浪新闻中新闻的标题，时间，作者，等讯息。并导出到‘news.xlsx’表格中。

当然，也会遇到一些问题，比如弹出: IndexError: list index out of range.则是因为原先的新闻链接被删除了，导致列表为空。等等