python爬虫（二）爬取知乎问答

都说知乎上问答的质量挺高，刚学爬虫没几天，现在对其问答内容进行爬虫实验。

在知乎首页，通过输入关键词，搜索问题，之后点击问题找到该问题对应的网友回答。

根据该过程，爬虫过程需要分为两步：

1、通过关键词（Java）搜索问题，得到url=https://www.zhihu.com/search?type=content&q=java，根据该url爬取该页面下所有的问题及其对应的问题id；

2、根据第一步得到的问题及其id，得到url=https://www.zhihu.com/question/31437847，爬取该url页面下所有的网友回答。

具体代码如下（https://github.com/tianyunzqs/crawler/tree/master/zhihu）

#!usr/bin/env python
# -*- coding: utf-8 -*-

import re
from urllib import request, parse
from bs4 import BeautifulSoup

keyword_list = ['svm', '支持向量机', 'libsvm']
fout = open("E:/python_file/zhihu.txt", "w", encoding="utf-8")
for keyword in keyword_list:
    print(keyword)
    url = 'https://www.zhihu.com/search?type=content&q=' + parse.quote(keyword)
    user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) ' \
                 'Chrome/39.0.2171.95 Safari/537.36'
    headers = {'User-Agent': user_agent}
    keyword_question_url_list = {}
    try:
        req = request.Request(url, headers=headers)
        response = request.urlopen(req, timeout=5)
        content = response.read().decode('utf-8')
        soup = BeautifulSoup(content, 'html.parser')
        all_div = soup.find_all('li', attrs={'class': re.compile('item clearfix.*?')})
        question_url_list = {}
        for e_div in all_div:
            title = e_div.find_all('a', attrs={'class': 'js-title-link',
                                               'target': '_blank',
                                               'href': re.compile('/question/[0-9]+')})
            if title:
                title = title[0].text
                _id = e_div.find_all('link', attrs={'itemprop': 'url',
                                                    'href': re.compile('/question/[0-9]+/answer/[0-9]+')})
                href = _id[0].attrs.get('href')
                pattern = re.compile('/question/(.*?)/answer/(.*?)$', re.S)
                items = re.findall(pattern, href)
                question_id = items[0][0]
                question_url_list[title] = 'https://www.zhihu.com/question/' + question_id
            else:
                title_id = e_div.find_all('a', attrs={'class': 'js-title-link',
                                                   'target': '_blank',
                                                   'href': re.compile('https://zhuanlan.zhihu.com/p/[0-9]+')})
                if title_id:
                    title = title_id[0].text
                    href = title_id[0].attrs.get('href')
                    question_url_list[title] = href
                else:
                    continue
        keyword_question_url_list[keyword] = question_url_list
        # for q, d in question_url_list.items():
        #     print(q, d)
    except:
        continue

    for keyword, question_url_list in keyword_question_url_list.items():
        for question, url in question_url_list.items():
            fout.write(question + "\n")
            try:
                req = request.Request(url, headers=headers)
                with request.urlopen(req, timeout=5) as response:
                    content = response.read().decode('utf-8')
                    soup = BeautifulSoup(content, 'html.parser')
                    all_div = soup.find_all('div', attrs={'class': 'List-item'})
                    for e_div in all_div:
                        answer = e_div.find_all('span', attrs={'class': 'RichText CopyrightRichText-richText',
                                                               'itemprop': 'text'})
                        answer = answer[0].text
                        fout.write(answer + "\n")
            except request.URLError as e:
                if hasattr(e, "code"):
                    print(e.code)
                if hasattr(e, "reason"):
                    print(e.reason)

存在的问题：

以上程序可以很好完成第一步，但是第二步只能取到问题的前2个回答。

根据http://www.cnblogs.com/buzhizhitong/p/5697526.html的介绍，应该可以用Selenium+Phantomjs来解决，以后再尝试。