一、实现抓取网页的去重深度遍历算法
若网页下链接复杂,建议抓取深度较小,图片内只是小部分输出结果
深度遍历算法的主要是采用栈(后进先出)来实现
import requests
import re
r = re.complie(r'<a.*href=\"(https?://.*?)[\"|\'].*')
import urllib3
urllib3.disable_warnings()
def geturlsdeep(seed,depth):
stack = [seed]
storage = {}
storage[seed] = 0
used=set()
while len(stack)>0:
try:
url = stack.pop(-1)
html=requests.get(url,timeout=5,verify=False).text
print("\t"*storage[url],"%d:%s"%(storage[url],url))
used.add(url)
new_urls = re.findall(r,html)
for new_url in new_urls:
if new_url not in used and new_url not in stack and storage[url]<depth:
storage[new_url] = storage[url]+1
stack.append(new_url)
except Exception as e:
pass
输出结果如下
![按层次输出](https://i-blog.csdnimg.cn/blog_migrate/2596e3c3937beea26b7b4c23e48bc341.png)
广度遍历去重算法
主要是利用队列(先进先出)
import requests
import re
r = re.compile(r'<a.*href=\"(https?://.*?)[\"|\'].*')
import urllib3
urllib3.disable_warnings()
tmplist = []
urlList= []
depthDict = {}
def geturls(depth):
while(len(tmplist)>0):
url = tmplist.pop(0)
urlList.append(url)
if depthDict[url] < depth:
html = requests.get(url).text
sublist = re.findall(r,html)
for u in sublist:
if u not in depthDict :
depthDict[u] = depthDict[url]+1
tmplist.append(u)
if __name__ == '__main__':
seed = 'http://httpbin.org'
depthDict[seed] = 0
tmplist.append(seed)
geturls(2)
for url in urlList:
print(depthDict[url],url)
输出结果如下:
![输出网页还有很多](https://i-blog.csdnimg.cn/blog_migrate/8250a5976a6b3eb6c1652e450c7aa990.png)