Requests库网络爬取实战
实例1:京东商品页面的爬取
import requests
url = "https://item.jd.com/100007136939.html"
try:
kv = {
'user-agent':'Mozilla/5.0'}
r = requests.get(url,headers = kv)
r.raise_for_status()
r.encoding = r.apparent_encoding
print(r.text[:1000])
except:
print("爬取失败")
京东也拒绝爬虫,需要将代理名改为服务器了
实例2:百度/360搜索关键词提交
#baidu
import requests
keyword = "Python"
try:
kv = {
'wd':keyword}
r = requests.get("http://www.baidu.com/s",params=kv