Python网络爬虫与信息提取——爬取实例

下面展示一些 内联代码片。

实例1

从京东https://item.jd.com/11198917385.html爬取信息

import requests
url="https://item.jd.com/11198917385.html"
try:
	kv = {'user-agent':'Mozilla/5.0'}
	r = requests.get(url,headers = kv)
	r.raise_for_status()
	r.encoding=r.apparent_encoding
	return r.text
except:
	return "产生异常"

这时候出现了问题：Non-UTF-8 code starting with ‘\xb2’ in file requests.py on line 9, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
解决办法：python默认的编码是UTF-8，汉字注释时需要在代码开头加上

#_*_ coding:gb2312_*_

结果还是出现问题SyntaxError: ‘return’ outside function
出现return函数只能用在函数里面

#_*_ coding:gb2312_*_

import requests

def GetHTMLText(url):
	try:
		kv = {'user-agent':'Mozilla/5.0'}
		r = requests.get(url,headers = kv)
		r.raise_for_status()
		r.encoding=r.apparent_encoding
		return r.text
	except:
		return "产生异常"
		
if __name__ =="__main__":
	url="https://item.jd.com/11198917385.html"
	print(GetHTMLText(url))

这时候还是产生异常
AttributeError: module ‘requests’ has no attribute ‘get’
因为我文件名取为requests，冲突了，改名之后就可以了
‘user-agent’:'Mozilla/5.0’将身份标志成浏览器，默认user-agent为爬虫

实例2

百度关键词接口：http://www.baidu.com/s?wd=keyword
360关键词接口：http://www.so.com/s？q=keyword

def GetHTMLText(url):
	try:
		kw={'wd':keyword}
		kv={'user-agent':'Mozilla/5.0'}
		r = requests.get(url,headers = kv,params=kw)
		print(r.request.url)
		r.raise_for_status()
		r.encoding=r.apparent_encoding
		print(len(r.text))
	except:
		return "产生异常"
		
if __name__ =="__main__":
	url="https://www.baidu.com"
	keyword="Python"

实例3

网络图片链接格式：http://www.example.com/picture.jpg

import requests
import os
url="http://www.example.com/picture.jpg"
root="D://pics//"
path=root+url.split('/')[-1]
try:
	if not os.path.exists(root):
		os.mkdir(root)
	if not os.path.exisrs(path):
		r=requests.get(url)
		with open(path,'wb') as f:
			f.write(r.content)
			f.close()
			print("文件保存成功")
	else:
		print("文件已存在")
except:
	print("爬取失败")

实例4：IP地址归属地查询

import requests
url="http://m.ip138.com/ip.asp?ip="
try:
	r=requests.get(url+'202.204.80.112')
	r.raise_for_status()
	r.encoding=r.apparent_encoding
	print(r.text[-500:])
except:
	print("爬取失败")