python_spider 03 requsets 模块的使用

get请求和post请求初识别,代理,session 和ssl证书

一:reuqests 库的get 和post请求
知识点:
　　>:1 想要发送什么请求就调用什么请求的方法
　　>:2
　　　　response 的属性
　　　　response.text() # 获取文本
　　　　response.content() #以2进制的方式获取,需要docode()成对应编码
　　　　response.url() #返回url
　　　　response.encoding() #返回编码方式
　　　　response.status_code() #返回状态

二:get请求的例子

get请求
import requests
# url =‘https://www.baidu.com/’
# response =requests.get(url)
params ={
“wd”:‘中国’
}
headers ={“User-Agent”:“Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36”}
response=requests.get('http://www.baidu.com/s?’,params=params,headers=headers)
print(response.url) #返回拼接后的url
print(response.content.decode(‘utf-8’)) #content #uncode解码
print(response.status_code)# 返回链接状态码
print(response.encoding) #返回编码方式
#print(response.text) #返回默认按照系统编码的

具体步骤:

导包
分析网页,传参get请求,200响应
输入中国查看网页入参
params={
‘wd’:‘中国’
}
params 是传递参数用的
这样就获取到了最基础的get请求内容

三:post请求例子:

post 请求:是输入内容不随这html显示在标题上.是一种安全的请求
例子是拉勾网,…这个网站不把 内容写在html里面,写在post里面…

1 post请求
 2 '''
 3 import requests
 4 
 5 data ={
 6     "first":"true",
 7     "pn":"1",
 8     "kd":"python"
 9 
10 }
11 proxy={
12     "https":"221.218.102.146:58208"
13 }
14 headers = {
15     'Origin': 'https://www.lagou.com',
16     'X-Anit-Forge-Code': '0',
17     'Accept-Encoding': 'gzip, deflate, br',
18     'Accept-Language': 'zh-CN,zh;q=0.9',
19     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',
20     'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
21     'Accept': 'application/json, text/javascript, */*; q=0.01',
22     'Referer': 'https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput=',
23     'X-Requested-With': 'XMLHttpRequest',
24     'Connection': 'keep-alive',
25     'X-Anit-Forge-Token': 'None',
26     'Cookie':'_ga=GA1.2.182345315.1550592539; _gid=GA1.2.2105546638.1550592539; user_trace_token=20190220000855-a848d5ea-3460-11e9-95fb-525400f775ce; LGUID=20190220000855-a848ddc6-3460-11e9-95fb-525400f775ce; index_location_city=%E6%B7%B1%E5%9C%B3; JSESSIONID=ABAAABAAAIAACBI9C3F23A9363576FBCAEBE7AC328116CF; WEBTJ-ID=20190220232250-1690b80958d21f-0bc575533ce91d-3c604504-1049088-1690b80958e1ac; _gat=1; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1550592539,1550632242,1550674096,1550676170; LGSID=20190220232250-62a45f5d-3523-11e9-8292-5254005c3644; PRE_UTM=m_cf_cpc_360_pc; PRE_HOST=www.so.com; PRE_SITE=https%3A%2F%2Fwww.so.com%2Fs%3Fie%3Dutf-8%26src%3D360chrome_toolbar_search%26q%3D%25E6%258B%2589%25E9%2592%25A9; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Flp%2Fhtml%2Fcommon.html%3Futm_source%3Dm_cf_cpc_360_pc%26m_kw%3D360_cpc_sz_e110f9_265e1f_%25E6%258B%2589%25E9%2592%25A9; TG-TRACK-CODE=index_search; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1550676179; LGRID=20190220232259-67a01432-3523-11e9-8292-5254005c3644; SEARCH_ID=97c647fb59dd46e7a1e492cb426da9cc'
27 }
28 response=requests.post('https://www.lagou.com/jobs/positionAjax.json?city=%E6%B7%B1%E5%9C%B3&need',data=data,headers=headers,proxies=proxy)
29 print(response.content.decode('utf-8'))
30 # print(response.text)

from data是post 请求输入的内容
data ={
“first”:“true”,
“pn”:“1”,
“kd”:“python”}

---

四:requests的代理问题

---

知识点:requsests的proxies = 可以输入ip要注意请求方式是http 还是https

import requests
url ='https://httpbin.org/ip'
proxy ={
    "https":'61.164.39.66:53281'
}
response =requests.get(url,proxies=proxy)
print(response.text)



https://httpbin.org/ip 这个网站可以返回请求者的ip
用于代理ip的网站很多.个人喜欢xici ,kaidaili等,都很多

五:session 的使用

---

如果想要在多次请求中共享cookies,那么使用session
import requests
# session 是保留登录状态,方便下次直接登录
session= requests.session()
url =你需要登录的网站
data =输入内容
headers =请求头
session.post(url,data=data,headers=headers)
response =session.get(url)

登录后用session保持登录请求和原来的requests 一样

六 :处理ssl不认证网站
有些网站不没有合法的ssl 证书
只需要requests 请求中将 verify =False就可以了

import requests

response = requests.get("https://www.baidu.com/", verify=True)

# 也可以省略不写
# response = requests.get("https://www.baidu.com/")
print(response.text)