Python3--翻页

在实现第一页内容和头像的获取后，想实现代码自动翻页获取后面内容。其原理就是找到下一页的链接，并且访问即可。主要是在第二个实例中加入部分代码
第二个实例

• 获取html内容

def get_info( url, data = None):
    header = {
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        'Accept-Encoding': 'gzip, deflate, sdch',
        'Accept-Language': 'zh-CN,zh;q=0.8',
        'Connection': 'keep-alive',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36'
    }
    timeout = random.choice(range(60,180))
    while True :
        try:
            rep = requests.get(url, headers = header, timeout = timeout)
            rep.encoding = 'utf-8'
            break
        except socket.timeout as e:
            print('3:',e)
            time.sleep(random.choice(range(8,15)))
        except socket.error as e:
            print('4:', e)
            time.sleep(random.choice(range(20, 60)))

        except http.client.BadStatusLine as e:
            print('5:', e)
            time.sleep(random.choice(range(30, 80)))

        except http.client.IncompleteRead as e:
            print('6:', e)
            time.sleep(random.choice(range(5, 15)))
    return rep.text

这个部分和前面一样，不需要改变。

• 相关信息的获取

def get_data(html ):
    final = []
    pictures = []
    bs = BeautifulSoup(html, "html.parser")
    body = bs.body
    content_left = body.find(id = 'content-left') #找到该页总框
    contents = content_left.find_all('div',class_ = 'article block untagged mb15')#找到所有内容框
    pages = content_left.find('ul',class_ = 'pagination')#找到翻页框
    next_page = pages.find('span',class_= 'next')#找到下一页
    nextUrl = next_page.find_parent('a').get('href')#获取下一页的url


    for content in contents: #对每个故事进行遍历
        temp = []
        author = content.find('div',class_='author clearfix')
        picture = author.find('img')
        picture_src = picture.get('src')
        pictures.append(picture_src)
        user_name = content.find("h2").string
        temp.append(user_name)
        data = content.find(class_ = 'content')
        story = data.find('span').get_text()
        temp.append(story)
        numbers = content.find_all('i', class_ = 'number')
        good = numbers[0].string + '好笑'
        temp.append(good)
        comment = numbers[1].string + '评论'
        temp.append(comment)
        temp.append(picture_src)
        final.append(temp)

    return final,pictures,nextUrl#返回当页数据，当页头像url和下一页的url

这里主要是添加了下一页url的返回

• 翻页的实现

def page_turn( url):
    results_list = [] #百科信息
    pictures_list = [] #头像
    currentUrl = url#当前url的信息
    count = 1
    while count <=3:#这里设定只收集前3页的信息
        html = get_info(currentUrl)#调用get_info()函数
        results,pictures,nextUrl = get_data(html)#调用get_data()函数
        results_list.append(results)#将获取的当前页内容的list集加入总list中
        pictures_list.append(pictures)#将当前页的头像url集加入总头像list中
        currentUrl = url + nextUrl[1:]#下一页url处理
        print(currentUrl)
        count = count +1
    return results_list,pictures_list#返回总结果

这里是一个翻页函数。需要注意的是results_list和pictures_list的元素也是list!

• 写入CSV

def write_data(datas, name):
    file_name = name
    for data in datas:
        with open(file_name, 'a', errors='ignore', newline='') as f:
            f_csv = csv.writer(f)
            f_csv.writerows(data)

由于之前的实例都是对单页的内容进行摘取，所以只有传入的参数是格式是[[list1],[list2]…[listn]]。而这里是三个嵌套[[[list1]..[listn]],[[list1]…[listn]]…[[list1]…[listn]]]这种样式。因此需要一个循环去掉一层。

• 下载头像

def download_pic(imgs):
    count = 1
    if not os.path.exists('pic'):
        os.makedirs('pic')
    for pictures in imgs:
        for picture in pictures:
            if picture == '/static/images/thumb/anony.png?v=b61e7f5162d14b7c0d5f419cd6649c87':
                print("静态图片")
                continue
            else:
                try:
                    r = requests.get(picture)
                except BaseException as e:
                    print("图片下载失败", e)
                    time.sleep(random.choice(range(30, 80)))
                else:
                    filename = str(count) + '.jpg'
                    path = "pic/" + filename
                    f = open(path, 'wb')
                    f.write(r.content)
                    print(count)
                    count = count + 1

原理同上

• 主函数

if __name__ == '__main__':
    url ='http://www.qiushibaike.com/'
    result,picture = page_turn(url)
    write_data(result, 'qiubai.csv')
    download_pic(picture)

源代码