网络爬虫入门之规则

requests库的使用

• 爬取网页的通用代码框架

def getHTMLText(url)
	try:
		r = request.get(url,timeout = 30)
		r.raise_for_status()   #如果状态不是200，引发HTTPErrory异常
		r.encoding = r.apparent_encoding 
		return r.text
	except:
		return "产生异常“
if __name__ == "__main__":
	url = "http://www.baidu.com"
	print(getHTMLText(url))

• requests库的7个主要方法
  
• HTTP协议：请求与相应的协议
• HTTP协议对URL的操作方法
  

实例讲解

• 实例1：京东商品页面的爬取

import requests
url = "http://item.jd.com/2967929.html"
try:
	r = requests.get(url)
	r.raise_for_status()
	r.encoding = r.apparent_encoding
	print(r.text[:1000])
except:printf("爬取失败")

*实例2 亚马逊商品页面的爬取

import requests 
url = "https://www.amazon.cn/gp/product/B0M8L5Z3Y"
try:
	kv = {'user-agent':Mozilla/5.0}
	r = requests.get(url,headers = kv)
	r.raise_for_status()
	r.encoding = r.apparent_encoding
	print(r.text[1000:2000])
except"
	print("爬取失败”)

• 实例三：百度360关键词提交

import requests 
keyword = "Python"
try:
	kv = {'wd':keyword}
	r = requests.get("http://www.baidu.com/s",params = kv)
	print(r.requests.url)
	r.raise_for_status()
	print(len(r.text))
except:
	printf("爬取失败“）

• 实例5 ip地址查询库

import requests
url = "http://m.ip138.com/ip.asp?ip="
try:
	r = requests.get(url + '202.204.80.112')
	r = raise_for_status()
	r.encoding = r.apparent_encoding
	print(r.text[-500]
except:
	print("爬取失败”)