【python学习笔记】爬虫总遇到的：'latin-1' codec can't encode character...

小练习爬取豆瓣电影热度排行榜

原因：

　　从火狐浏览器中复制header内容是复制出来的带省略号，这个需要注意（这类问题似乎就是url或者header中有类似的错误）

 还会遇到这样的错误：ssl-warnings InsecureRequestWarning，但似乎并不影响最终的结果，但似乎慢一点

网上有许多解决方案，比如加入如下代码：

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

练习代码：

#coding=utf-8

import requests
import json
import urllib3
from easygui import msgbox

def getUrls(total):
    n = total / 20 
    if int(n) != n: n = int(n) + 1
    url_list = []
    for i in range(int(n)):
        page_start = str(i * 20)
        url_list.append('https://movie.douban.com/j/search_subjects?type=movie&tag=热门&sort=recommend&page_limit=20&page_start='+ page_start)
    return url_list

def getContent(total):
    title = "警告："
    msg = "查询条目数应该为整数！！！"
    if not isinstance(total, int): 
        msgbox(msg=msg, title=title)
        return
    headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0'}
    url_list = getUrls(total)
    content_list = []
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
    for url in url_list:
        response = requests.get(url, headers, verify=False)
        content = response.content.decode() #这里得到一个json字符串
        content_list.extend(json.loads(content)['subjects'])
    if len(content_list) < total:
        return content_list
    else:
        return content_list[:total]
if __name__ == "__main__":
    total = int(input("请输入要查看的条目数："))
    for i, item in enumerate(getContent(total)):
        print(i + 1, "、", item['title'], item['rate'], item['url'])
    
    

 

转载于:https://www.cnblogs.com/smoggy/p/10664983.html