Python 学习爬虫

最新推荐文章于 2022-07-18 18:00:51 发布

Saidywin

最新推荐文章于 2022-07-18 18:00:51 发布

阅读量357

点赞数

文章标签： python 爬虫

本文链接：https://blog.csdn.net/weixin_41447442/article/details/124591680

版权

刚开始学习爬虫，用了以下代码

from urllib import request
from http import cookiejar  

cookie_support = request.HTTPCookieProcessor(cookiejar.CookieJar())  
opener = request.build_opener(cookie_support, request.HTTPHandler)  
request.install_opener(opener)  

content = request.urlopen('https://movie.douban.com/').read().decode('utf-8')  
print(content)

发现报错了

Traceback (most recent call last):
  File "E:\Program\python3.7sourcecode-master\python3.7sourcecode-master\chapter18\cookie_request.py", line 11, in <module>
    content = request.urlopen('https://movie.douban.com/').read().decode('utf-8')
  File "C:\Users\MAIBENBEN\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\MAIBENBEN\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "C:\Users\MAIBENBEN\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "C:\Users\MAIBENBEN\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 561, in error
    return self._call_chain(*args)
  File "C:\Users\MAIBENBEN\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Users\MAIBENBEN\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 418: 

Process finished with exit code 1

然后发现说是Http error 418 找了下原因是在做request 的时候没有加入User-agent，改成如下

from urllib.request import urlopen, Request
url = 'https://movie.douban.com'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}
ret = Request(url, headers=headers)
res = urlopen(ret)
content = res.read().decode('utf-8')
print(content)

然后又出错了

python 写数据入文件碰见的bug：UnicodeEncodeError: ‘gbk‘ codec can‘t encode character ‘\xee‘ in position 21865:

解决方案：

在setting里边指定编码为utf-8，就可以解决问题了，如下图所示。原因就是lang在豆瓣网是

html lang="zh-CN"
在这里插入图片描述

最后跑以下成功：

<!DOCTYPE html>
<html lang="zh-CN" class="ua-windows ua-webkit">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="renderer" content="webkit">
<meta name="referrer" content="always">
<meta name="google-site-verification" content="ok0wCgT20tBBgo9_zat2iAcimtN4Ftf5ccsh092Xeyw" />
<title>
豆瓣电影 Top 250
</title>

Saidywin

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
打赏
0
评论
Python 学习爬虫

刚开始学习爬虫，用了以下代码from urllib import requestfrom http import cookiejar cookie_support = request.HTTPCookieProcessor(cookiejar.CookieJar()) opener = request.build_opener(cookie_support, request.HTTPHandler) request.install_opener(opener) content =
复制链接

扫一扫