最近有个需求需要对指定一批URL获取其title,以便于筛选出所需要的URL,查了下网上工具挺多的。
我这里使用python中的urllib或urllib2包对URL 的内容提取.方法如下:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import urllib2
import re
url='http://segmentfault.com/q/1010000000124036'
html = urllib2.urlopen(url).read()
res_list = re.findall(r"<title>.*</title>", html)
for t in res_list:
print t
执行结果为:
python如何正确抓取网页标题 - SegmentFault
看这个帖子里说的BeautifulSoup也是很不错的, 没有试过,代码如下:
import urllib
from BeautifulSoup import BeautifulSoup
content = urllib.urlopen('http://segmentfault.com/q/1010000000124036').read()
soup = BeautifulSoup(content)
print soup.find('title')
用第一段代码在获取我的主页http://blog.csdn.net/lming_08?viewmode=list的title时,报错了:
Traceback (most recent call last):
File "get_title.py", line 9, in <module>
html = urllib2.urlopen(url).read()
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 438, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
是403错误,看来CSDN对本次请求是禁止的, 所以我们要模拟正常人访问浏览器的行为,加上headers
url='http://blog.csdn.net/lming_08?viewmode=list'
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'# 将user_agent写入头信息
headers = { 'User-Agent' : user_agent }
req = urllib2.Request(url, headers = headers)
html = urllib2.urlopen(req).read()
res_list = re.findall(r"<title>.*\s?.*</title>", html)
for t in res_list:
print t[7:-8]
执行没有问题,结果为:
lming_08技术博客
- 博客频道 - CSDN.NET
注意这里标题里面含有\n字符,而之前正则表达式r"<title>.*</title>"是不会包含\n的,所以改为 r"<title>.*\s?.*</title>" 或者 r"<title>.*\n?.*</title>"
最新发现这获取 http://www.smzdm.com/p/664187 title时仍然会报403错误, 查了下需要更新headers
user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'
headers = { 'User-Agent' : user_agent }
执行结果为:
lming_08@ubuntu:~/MyWorkSpace/Pycode/htmlparse$ python get_title.py
Mentholatum 曼秀雷敦 肌研极润保湿系列套装(洁面乳50g+化妆水100ml+乳液90ml+面膜18ml+眼膜2片+眼霜3g) 99元(199-100)_乐蜂网优惠_什么值得买
在实际操作中我们会对很多url进行解析, 中间免不了会出现服务器返回其他错误代码,因此我们要捕获异常继续执行:
try:
html = urllib2.urlopen(req).read()
res_list = re.findall(r"<title>.*\s?.*</title>", html)
for t in res_list:
print t[7:-8]
except urllib2.HTTPError:
print "failed parsing web url"
文章参考于:
http://www.cnblogs.com/txw1958/archive/2012/03/12/2392067.html
http://segmentfault.com/q/1010000000124036
http://blog.csdn.net/my2010sam/article/details/17398807