cookie模拟登录
抓取需要登录才能访问的页面
1、先登录成功1次,获取到携带登陆信息的Cookie
F12打开控制台,在页面输入用户名、密码,登录成功,找到/home(一般在抓到地址的上面)
2、携带着cookie发请求
** Cookie
** Referer(源,代表你从哪里转过来的)
** User-Agent
import requests
from lxml import etree
url = 'http://www.renren.com/969255813/profile'
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "zh-CN,zh;q=0.9",
"Connection": "keep-alive",
"Cookie": "",
"Host": "www.renren.com",
"Referer": "http://www.renren.com/SysHome.do",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
}
html = requests.get(url,headers=headers).text
parse_html = etree.HTML(html)
result = parse_html.xpath('//*[@id="operate_area"]/div[1]/ul/li[1]/span/text()')[0].strip()
print(result)
- 知识点
利用requests模块中的session会话保持功能
- session会话使用流程
1、实例化session对象
session = requests.session()
2、让session对象发送get或者post请求
res = session.get(url,headers=headers)
- 具体步骤
1、寻找登录时POST的地址
查看网页源码,查看form,找action对应的地址: http://www.renren.com/PLogin.do
2、发送用户名和密码信息到POST的地址
* 用户名和密码信息以什么方式发送? -- 字典
键 :<input>标签中name的值(email,password)
值 :真实的用户名和密码
post_data = {'email':'','password':''}
- 程序实现
整体思路
1、先POST: 把用户名和密码信息POST到某个地址中
2、再GET: 正常请求去获取页面信息
import requests
from lxml import etree
post_url = 'http://www.renren.com/PLogin.do'
post_data = {
'email' : 'xxxxxx',
'password' : 'xxxxxx'
}
headers = {
'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
'Referer' : 'http://www.renren.com/SysHome.do'
}
session = requests.session()
session.post(post_url,data=post_data,headers=headers)
url = 'http://www.renren.com/970294164/profile'
html = session.get(url,headers=headers).text
parse_html = etree.HTML(html)
result = parse_html.xpath('//*[@id="operate_area"]/div[1]/ul/li[1]/span/text()')[0].strip()
print(result)