学习完上一篇文章,我们需要了解我们需要爬虫的网站的一些数据信息,也就是一些html标签。
bs4中最重要的就是数据的分类,解析,获取过程。
即:
response = urllib2.urlopen(res)
html = response.read()
soup = BeautifulSoup(html, "lxml")
someData = soup.select("div.content span")
这里的soup.select括号中的数据,是非常重要的。
如下图,我们如何定位select中的标签:
通过红线标记的地方,我们可以直接看到这里的div 的标签的数据,就是上面每一条段子的数据内容,所以,
我们先写一条
someData = soup.select("div.content")
进行一下测试:
调试推荐大家使用ipython工具,非常方便我们分析数据:
不截图了,直接复制文字:
suz@suz9527:~/Pytools$ ipython
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: from bs4 import BeautifulSoup
In [2]: import urllib2
In [3]: url = 'http://www.qiushibaike.com/text/page/1/'
In [4]: heads = {
...:
...: 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36',
...: 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
...: 'Connection': 'keep-alive',
...: 'Upgrade-Insecure-Requests': '1',
...: 'Referer': 'http://www.qiushibaike.com/',
...: 'Accept-Language': 'zh-CN,zh;q=0.8',
...: 'Cookie': '_xsrf=2|db27040e|6b4ed8d9536590d4ec5d2064cc2bef4f|1474364551; _qqq_uuid_="2|1:0|10:1474364551|10:_qqq_uuid_|56:MzBlNWFkOGE3MWEyMzc1MWIxMTE3MDBlZjM2M2RkZWQxYzU5YTg1Yw==|1dd2a4f4ce
...: acad26b5da9cc295d2965226ea25ee73289855cf032629c4992698"; Hm_lvt_2670efbdd59c7e3ed3749b458cafaa37=1474364592; Hm_lpvt_2670efbdd59c7e3ed3749b458cafaa37=1474364595; _ga=GA1.2.1125329542.1474364596'
...:
...: }
In [5]: res = urllib2.Request(url, headers=heads)
In [6]: response = urllib2.urlopen(res)
In [7]: html = response.read()
...:
In [8]: soup = BeautifulSoup(html, "lxml")
In [9]:
response = urllib2.urlopen(res)
html = response.read()
soup = BeautifulSoup(html, "lxml")
someData = soup.select("div.content span")
[<div class="content">\n<span>
In [10]: print someData
我们可以看到结果中
[<div class="content">\n<span>
有这么一行span属性,所以我们修改一下上面的select为:
someData = soup.select("div.content span")
查看刚才的输出结果:
In [11]: someData = soup.select("div.content span")
In [12]: print someData
这里的someData其实是个list,所以我们写个for循环输出一下可以看到:
好了,说明我们的想要得到的结果已经正常显示出来了。
下面是完整的代码,加上一些修饰处理:
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import urllib2
def getContent(n):
url = 'http://www.qiushibaike.com/text/page/' + str(n) + '/'
#url = 'http://www.qiushibaike.com/8hr/page/'+str(n)+'/'
print url
heads = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'Referer': 'http://www.qiushibaike.com/',
'Accept-Language': 'zh-CN,zh;q=0.8',
'Cookie': '_xsrf=2|db27040e|6b4ed8d9536590d4ec5d2064cc2bef4f|1474364551; _qqq_uuid_="2|1:0|10:1474364551|10:_qqq_uuid_|56:MzBlNWFkOGE3MWEyMzc1MWIxMTE3MDBlZjM2M2RkZWQxYzU5YTg1Yw==|1dd2a4f4ceacad26b5da9cc295d2965226ea25ee73289855cf032629c4992698"; Hm_lvt_2670efbdd59c7e3ed3749b458cafaa37=1474364592; Hm_lpvt_2670efbdd59c7e3ed3749b458cafaa37=1474364595; _ga=GA1.2.1125329542.1474364596'
}
res = urllib2.Request(url, headers=heads)
response = urllib2.urlopen(res)
html = response.read()
soup = BeautifulSoup(html, "lxml")
someData = soup.select("div.content span")
num = 0
for some in someData:
num = num + 1
print num
print some.text + '\n'
if __name__ == "__main__":
for i in range(1, 5):
getContent(i)
最终效果图: