爬取需要登录的网页内容，获取cookie

最新推荐文章于 2024-08-08 18:34:38 发布

回身挽流光

最新推荐文章于 2024-08-08 18:34:38 发布

阅读量3k

点赞数 2

本文链接：https://blog.csdn.net/AloesZhang/article/details/95001695

版权

首先需要获取到网页的cookie,例如爬取人人网登录后的网址为http://www.renren.com/971405629/profile，找到cookie，如下图
在这里插入图片描述
最后代码如下就可以模拟网页登录了

import urllib.request
import urllib.parse
url = 'http://www.renren.com/971405629/profile'
headers = {'User-Agent':' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
'Cookie':' anonymid=jxsntfqs-ofu0t6; depovince=ZGQT; _r01_=1; jebe_key=37a5e620-420b-42e4-8361-ab88024b3324%7Cfe293cb2ffcb374252a27291355dc10f%7C1562486156712%7C1%7C1562486158570; jebe_key=37a5e620-420b-42e4-8361-ab88024b3324%7Cfe293cb2ffcb374252a27291355dc10f%7C1562486156712%7C1%7C1562486158576; wp=0; ick_login=7a1fbbfe-cf38-498b-b031-18cd95c838e0; jebecookies=d3a55224-0887-4972-b301-fde9306964d5|||||; JSESSIONID=abcC8N0ZWGKTJ9eUEHlVw; _de=C5E0E40596487205091539360EA5D908; p=f90a637b5d70675e5226acc6ef2d753a9; first_login_flag=1; ln_uact=18701902391; ln_hurl=http://head.xiaonei.com/photos/0/0/men_main.gif; t=e69c70422cb6a8c8f0a57170308239c19; societyguester=e69c70422cb6a8c8f0a57170308239c19; id=971405629; xnsid=3479f0f9; ver=7.0; loginfrom=null; wp_fold=0',
           }
request = urllib.request.Request(url=url, headers=headers)
response = urllib.request.urlopen(request)
with open('renren.html', 'wb') as fp:
    fp.write(response.read())