根据关键词爬取豆瓣相关书籍信息

根据书的类型爬取相关信息并且保存至csv，这里以‘python’为例。
结果可以看到除了书名外，其他都有混杂，因为研究html不太彻底。得到的数据有点错误。

import sys
import time
import requests
import pandas as pd
import re
import csv
import numpy as np
from bs4 import BeautifulSoup
book_tag="python"
pagenum=1
header={'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'}
book_list=[]
while True:
    url="https://book.douban.com/tag/"+book_tag+"?start="+str(pagenum*20)
    try:
        response=requests.get(url,headers=header)
        response.raise_for_status()
        response.encoding=response.apparent_encoding
        responsetext=response.text
    except:
        print("爬取数据失败")
    # with open('douban.txt','wb+') as f:
    #     f.write(responsetext.encode())
    #     f.close()

    soup=BeautifulSoup(responsetext,'html.parser')
    list_soup=soup.find_all('li',class_='subject-item')
    if list_soup==[]:
        break
    def has_title(tag):
        return tag.has_attr('title')
    for item in list_soup:
        title_info=item.find(has_title).get_text().replace("\n","")
        title_info=title_info.replace(" ","")
        print(title_info)
        book_info=item.find('div',class_="pub").string
        book_info=book_info.strip().split("/")
        try:
            orgin_author=book_info[0]
        except:
            orgin_author="暂无"
        try:
            year_info=book_info[-2]
        except:
            year_info="暂无"
        try:
            public_info=book_info[-3]
        except:
            public_info="暂无"
        if item.find('span',class_="rating_nums")!=None:
            rank_point=item.find('span',class_="rating_nums").string
        else:
            rank_point="暂无"
        book_list.append([''.join(title_info.split()),orgin_author,year_info,public_info,rank_point])
    print("目前爬取至%d页"%pagenum)
    pagenum+=1
book_df=pd.DataFrame(book_list,columns=["书名","作者","出版年份","出版社","评分"])
book_df.to_csv('book.csv')

这是我做的结果文档
第一次发csdn，小白还有很多不懂，希望自己能坚持呀！