python代码解析

最新推荐文章于 2024-07-05 05:25:41 发布

R先森

最新推荐文章于 2024-07-05 05:25:41 发布

阅读量5k

点赞数 4

分类专栏：网络爬虫编程语言学习

本文链接：https://blog.csdn.net/beagreatprogrammer/article/details/50217107

版权

这篇博客主要解析了Python的基础语法，包括import、reload()、def、变量定义、数组、循环、time模块、异常处理、格式化输出、strip()、sorted()、open()、__main__与__init__、文件头声明、openpyxl库、range()与len()函数、数据类型转换、None、raw_input()与input()、with...as...结构以及yield关键字，适合初学者了解Python编程。

摘要由CSDN通过智能技术生成

最近准备好好学学python爬虫，从网上找了个例子，爬取豆瓣读书的信息，原文地址：http://blog.csdn.net/lanbing510/article/details/45887075

代码地址：

https://raw.githubusercontent.com/lanbing510/DouBanSpider/master/doubanSpider.py

因为以前基本没接触过python，所以对代码做个简单的解析，先贴下代码如下：

#-*- coding: UTF-8 -*-

import sys
import time
import urllib
import urllib2
import requests
import numpy as np
from bs4 import BeautifulSoup
from openpyxl import Workbook

reload(sys)
sys.setdefaultencoding('utf8')


def book_spider(book_tag):
    page_num=0;
    count=1
    book_list=[]
    try_times=0
    
    #Some User Agents
    hds=[{'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'},\
         {'User-Agent':'Mozilla/5.0 (Windows NT 6.2) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.12 Safari/535.11'},\
         {'User-Agent': 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0)'}]
         
    while(1):
        url="http://www.douban.com/tag/"+urllib.quote(book_tag)+"/book?start="+str(page_num*15)
        time.sleep(np.random.rand()*2)
        
        #Last Version
        try:
            req = urllib2.Request(url, headers=hds[page_num%len(hds)])
            source_code = urllib2.urlopen(req).read()
            plain_text=str(source_code)   
        except (urllib2.HTTPError, urllib2.URLError), e:
            print e
            continue
  
        ##Previous Version, IP is easy to be Forbidden
        #source_code = requests.get(url) 
        #plain_text = source_code.text  
        
        soup = BeautifulSoup(plain_text)
        list_soup = soup.find('div', {'class': 'mod book-list'})
        
        try_times+=1;
        if list_soup==None and try_times<200:
            continue
        elif list_soup==None or len(list_soup)<=1:
            break # Break when no informatoin got after 200 times requesting
        
        for book_info in list_soup.findAll('dd'):
            title = book_info.find('a', {'class':'title'}).string.strip()
            desc = book_info.find('div', {'class':'desc'}).string.strip()
            desc_list = desc.split('/')
            
            try:
                author_info = '作者/译者： ' + '/'.join(desc_list[0:-3])
            except:
                author_info='作者/译者： 暂无'
            try:
                pub_info = '出版信息： ' + '/'.join(desc_list[-3:])
            except:
                pub_info='出版信息： 暂无'
            try:
                rating = book_info.find('span', {'class':'rating_nums'}).string.strip()
            except:
                rating='0.0'
            try:
                people_num = book_info.findAll('span')[2].string.strip()
                people_num=people_num.strip('人评价')
            except:
                people_num='0'
            
            book_list.append([title,rating,people_num,author_info,pub_info])
            try_times=0 #set 0 when got v