01 更改user-agent
>>> import requests
>>> r=requests.get("https://www.amazon.cn/dp/B06XCX5FLV?ref_=Oct_ETopRankingC_desktop_NA")
>>> r.status_code
200
>>> r.request.headers
{'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
>>> kv ={'user-agent':'Mozilla/5.0'}
>>> r=requests.get("https://www.amazon.cn/dp/B06XCX5FLV?ref_=Oct_ETopRankingC_desktop_NA",headers=kv)
>>> r.status_code
200
>>> r.request.headers
{'user-agent': 'Mozilla/5.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
>>>
02 搜索引擎关键词提交
- 百度的关键词接口:
http://www.baidu.com/s?wd=keyword - 360关键词接口:
http://www.so.com/s?q=keyword
>>> kv={'wd':'python'}. #attention here!!!
>>> r=requests.get("http://www.baidu.com/s",params=kv)
>>> r.status_code
200
>>> r.request.url
'http://www.baidu.com/s?wd=python'
>>>
03 网络图片的爬取和存储
网络图片链接的格式:
http://www.example.com/picture.jpg
【例】
import requests
import os
url="http://b4-q.mafengwo.net/s14/M00/6A/D3/wKgE2lz5Id6AUvavACE7XExWyhE491.jpg"
root="//Users//qiuruipeng3//desktop//pics//"
path=root+url.split('/')[-1]
try:
if not os.path.exists(root):
os.mkdir(root)
if not os.path.exists(path):
r = requests.get(url)
with open (path,'wb') as f:
f.write(r.content)
f.close()
print("successfully done!!")
else:
print("error:the document has already existed!!")
except:
print("error:failed!!")
运行这段代码就会在响应位置产生一个名为pics的文件夹,并将图片存储到该文件夹中。
04 IP地址归属地的自动查询
>>> url="http://m.ip138.com/ip.asp?ip="
>>> r=requests.get(url+"202.204.80.112")
>>> r.status_code
200
>>> r.text[-500:]
'value="查询" class="form-btn" />\r\n\t\t\t\t\t</form>\r\n\t\t\t\t</div>\r\n\t\t\t\t<div class="query-hd">ip138.com IP查询(搜索IP地址的地理位置)</div>\r\n\t\t\t\t<h1 class="query">您查询的IP:202.204.80.112</h1><p class="result">本站主数据:北京市海淀区 北京理工大学 教育网</p><p class="result">参考数据一:北京市 北京理工大学</p>\r\n\r\n\t\t\t</div>\r\n\t\t</div>\r\n\r\n\t\t<div class="footer">\r\n\t\t\t<a href="http://www.miitbeian.gov.cn/" rel="nofollow" target="_blank">沪ICP备10013467号-1</a>\r\n\t\t</div>\r\n\t</div>\r\n\r\n\t<script type="text/javascript" src="/script/common.js"></script></body>\r\n</html>\r\n'
>>>