改进了一下,去掉哪些没用的。
留下的都是些有用的。接着要拔下来,然后放到TXT。。
#coding=utf-8
import urllib2
import re
from sgmllib import SGMLParser;
class CatCh(SGMLParser):
def reset(self):
self.url=[]
SGMLParser.reset(self)
def start_a(self,attrs):
href = [v for k, v in attrs if k == 'href']
if href:
self.url.extend(href)
url="http://wooyun.org"
content = urllib2.urlopen(url).read()
catch=CatCh()
catch.feed(content)
for item in catch.url:
if not re.match("#",item) :
if not re.match("^http://",item):
print url+item
运行结果:
http://wooyun.org/user.php?action=login
http://wooyun.org/user.php?action=register
http://wooyun.org/index.php
http://wooyun.org/corps/
http://wooyun.org/whitehats/
http://wooyun.org/teams/
http://wooyun.org/bugs/
http://wooyun.org/bug/submit
http://wooyun.org/corp_actions
http://wooyun.org/job.php?action=list
http://wooyun.org/notice/
http://wooyun.org/index.php
http://wooyun.org/notice.php?action=view&id=27
http://wooyun.org/notice.php?action=view&id=26
http://wooyun.org/notice.php?action=view&id=25
http://wooyun.org/notice.php?action=view&id=23
http://wooyun.org/notice.php?action=view&id=22
http://wooyun.org/notice.php?action=view&id=21
http://wooyun.org/whitehats/疯狗
http://wooyun.org/bugs/wooyun-2010-0570
http://wooyun.org/whitehats/mog
http://wooyun.org/bugs/wooyun-2011-03100
http://wooyun.org/whitehats/路人乙
http://wooyun.org/bugs/wooyun-2012-04466
http://wooyun.org/whitehats/Claud
http://wooyun.org/bugs/wooyun-2012-08222
http://wooyun.org/whitehats/blue
http://wooyun.org/bugs/wooyun-2012-09145
http://wooyun.org/whitehats/cssembly
http://wooyun.org/bugs/wooyun-2013-021657
http://wooyun.org/whitehats/xsser
http://wooyun.org/bugs/wooyun-2010-0248
http://wooyun.org/whitehats/爱梅小礼
http://wooyun.org/bugs/wooyun-2011-01413
http://wooyun.org/whitehats/蚊虫
http://wooyun.org/bugs/wooyun-2011-02395
http://wooyun.org/whitehats/GaRY
http://wooyun.org/bugs/wooyun-2012-06951
http://wooyun.org/whitehats/gainover
http://wooyun.org/bugs/wooyun-2012-07489
http://wooyun.org/whitehats/only_guest
http://wooyun.org/bugs/wooyun-2012-09977
http://wooyun.org/whitehats/CnCxzSec(衰仔)
http://wooyun.org/bugs/wooyun-2010-0358
http://wooyun.org/whitehats/路人甲
http://wooyun.org/bugs/wooyun-2010-0831
http://wooyun.org/whitehats/J.Yang
http://wooyun.org/bugs/wooyun-2011-02033
http://wooyun.org/whitehats/code0day
http://wooyun.org/bugs/wooyun-2012-04584
http://wooyun.org/whitehats/stephanie
http://wooyun.org/bugs/wooyun-2013-023234
http://wooyun.org/whitehats/科西嘉
http://wooyun.org/bugs/wooyun-2013-023733
http://wooyun.org/whitehats/B0mbErM@n
http://wooyun.org/bugs/wooyun-2011-01165
http://wooyun.org/whitehats/路人甲
http://wooyun.org/bugs/wooyun-2011-01370
http://wooyun.org/whitehats/Jannock
http://wooyun.org/bugs/wooyun-2011-03616
http://wooyun.org/whitehats/路人甲
http://wooyun.org/bugs/wooyun-2011-03692
http://wooyun.org/whitehats/小一
http://wooyun.org/bugs/wooyun-2012-05668
http://wooyun.org/whitehats/猪哥靓
http://wooyun.org/bugs/wooyun-2013-020425
http://wooyun.org/bugs/new_submit/
http://wooyun.org/feeds/submit
http://wooyun.org/bugs/wooyun-2013-027104
http://wooyun.org/bugs/wooyun-2013-027104#comment
http://wooyun.org/whitehats/Drizzle.Risk
http://wooyun.org/bugs/wooyun-2013-027098
http://wooyun.org/bugs/wooyun-2013-027098#comment
http://wooyun.org/whitehats/VIP
http://wooyun.org/bugs/wooyun-2013-027097
http://wooyun.org/bugs/wooyun-2013-027097#comment
http://wooyun.org/whitehats/爱梅小礼
http://wooyun.org/bugs/wooyun-2013-027095
http://wooyun.org/bugs/wooyun-2013-027095#comment
http://wooyun.org/whitehats/忆小冬
http://wooyun.org/bugs/wooyun-2013-027082
http://wooyun.org/bugs/wooyun-2013-027082#comment
http://wooyun.org/whitehats/带馅儿馒头
http://wooyun.org/bugs/wooyun-2013-027074
http://wooyun.org/bugs/wooyun-2013-027074#comment
http://wooyun.org/whitehats/小贱
http://wooyun.org/bugs/new_public/
http://wooyun.org/feeds/public
http://wooyun.org/bugs/wooyun-2013-023704
http://wooyun.org/bugs/wooyun-2013-023704#comment
http://wooyun.org/whitehats/梦想肥羊
http://wooyun.org/bugs/wooyun-2013-022727
http://wooyun.org/bugs/wooyun-2013-022727#comment
http://wooyun.org/whitehats/浩天
http://wooyun.org/bugs/wooyun-2013-023664
http://wooyun.org/bugs/wooyun-2013-023664#comment
http://wooyun.org/whitehats/asgoo
http://wooyun.org/bugs/wooyun-2013-020934
http://wooyun.org/bugs/wooyun-2013-020934#comment
http://wooyun.org/whitehats/路人甲
http://wooyun.org/bugs/wooyun-2013-023685
http://wooyun.org/bugs/wooyun-2013-023685#comment
http://wooyun.org/whitehats/Michael
http://wooyun.org/bugs/wooyun-2013-023668
http://wooyun.org/bugs/wooyun-2013-023668#comment
http://wooyun.org/whitehats/梧桐雨
http://wooyun.org/bugs/new_confirm/
http://wooyun.org/feeds/confirm
http://wooyun.org/bugs/wooyun-2013-027075
http://wooyun.org/bugs/wooyun-2013-027075#comment
http://wooyun.org/whitehats/路人甲
http://wooyun.org/bugs/wooyun-2013-026254
http://wooyun.org/bugs/wooyun-2013-026254#comment
http://wooyun.org/whitehats/Royal.
http://wooyun.org/bugs/wooyun-2013-026975
http://wooyun.org/bugs/wooyun-2013-026975#comment
http://wooyun.org/whitehats/小川
http://wooyun.org/bugs/wooyun-2013-027092
http://wooyun.org/bugs/wooyun-2013-027092#comment
http://wooyun.org/whitehats/VIP
http://wooyun.org/bugs/wooyun-2013-027065
http://wooyun.org/bugs/wooyun-2013-027065#comment
http://wooyun.org/whitehats/齐迹
http://wooyun.org/bugs/wooyun-2013-027051
http://wooyun.org/bugs/wooyun-2013-027051#comment
http://wooyun.org/whitehats/Sct7p
http://wooyun.org/bugs/new_unclaimed/
http://wooyun.org/feeds/unclaim
http://wooyun.org/bugs/wooyun-2013-027053
http://wooyun.org/bugs/wooyun-2013-027053#comment
http://wooyun.org/whitehats/倪明
http://wooyun.org/bugs/wooyun-2013-027023
http://wooyun.org/bugs/wooyun-2013-027023#comment
http://wooyun.org/whitehats/瞌睡龙
http://wooyun.org/bugs/wooyun-2013-026906
http://wooyun.org/bugs/wooyun-2013-026906#comment
http://wooyun.org/whitehats/霍大然
http://wooyun.org/bugs/wooyun-2013-026904
http://wooyun.org/bugs/wooyun-2013-026904#comment
http://wooyun.org/whitehats/if、so
http://wooyun.org/bugs/wooyun-2013-026821
http://wooyun.org/bugs/wooyun-2013-026821#comment
http://wooyun.org/whitehats/爱上平顶山
http://wooyun.org/bugs/wooyun-2013-026801
http://wooyun.org/bugs/wooyun-2013-026801#comment
http://wooyun.org/whitehats/M4sk
http://wooyun.org/lawer
http://wooyun.org/contactus
http://wooyun.org/help
http://wooyun.org/about
[Finished in 5.3s]