简单爬虫----爬取斗罗大陆3的100个章节

搞了一晚上终于搞好了这个爬虫。。。话说获得下一章的标签真不容易，用到了select选择器，然后获得a标签里面的href属性费了好大劲，测试了下爬取100章差不多花了半分钟

代码：

 

#coding=utf-8
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
import urllib2
from bs4 import BeautifulSoup
import requests
f=open('text.txt','wb')
url="http://www.tycqxs.com/0_31/21142.html"
r=urllib2.urlopen(url).read()
soup=BeautifulSoup(r,"html.parser")
links=soup.find_all(id='content')
page=1
while page<100:
    for link in links:
        d=link.text
        f.write(d+'\n')
    temp=soup.select('div.bottem2 > a')
    lis=BeautifulSoup(str(temp),"html.parser").find_all('a')
    url="http://www.tycqxs.com"+lis[3]['href']
    r = urllib2.urlopen(url).read()
    soup = BeautifulSoup(r, "html.parser")
    links = soup.find_all(id='content')
    page=page+1
f.close()