分析的是这个 lijiejie的子域名工具
https://github.com/lijiejie/subDomainsBrute
此次分析的版本是v1.1
首先导入了这些库
import multiprocessing
import gevent
from gevent import monkey
monkey.patch_all()
from gevent.queue import PriorityQueue
import re
import dns.resolver
import time
import signal
import os
import glob
from lib.cmdline import parse_args
from lib.commonimport is_intranet, load_dns_servers, load_next_sub, print_msg, get_out_file_name, \
user_abort
首先 gevent 协程
当一个gevent遇到IO操作时,比如访问网络,就自动切换到其他的greenlet,等到IO操作完成,在到合适的时候切换回来继续执行,由于IO操作非常耗时,经常使程序处于等待状态,有了gevent为我们自动切换协程
就保证总有greenlet在运行,而不是等待IO,
由于切换是在IO操作时自动完成,所以gevent需要修改python自带的一些标志库,
这一过程在启动时挺高 monkey patch完成
from gevent import monkey
monkey.patch_socket()
import gevent
def f(n):
for i in range(n):
print gevent.getcurrent(), i
g1 = gevent.spawn(f, 5)
g2 = gevent.spawn(f, 5)
g3 = gevent.spawn(f, 5)
g1.join()
g2.join()
g3.join()
执行的话输出是按顺序执行的
<Greenlet at 0x2577a80: f(5)> 0
<Greenlet at 0x2577a80: f(5)> 1
<Greenlet at 0x2577a80: f(5)> 2
<Greenlet at 0x2577a80: f(5)> 3
<Greenlet at 0x2577a80: f(5)> 4
<Greenlet at 0x2577bc0: f(5)> 0
<Greenlet at 0x2577bc0: f(5)> 1
<Greenlet at 0x2577bc0: f(5)> 2
<Greenlet at 0x2577bc0: f(5)> 3
<Greenlet at 0x2577bc0: f(5)> 4
<Greenlet at 0x2577c10: f(5)> 0
<Greenlet at 0x2577c10: f(5)> 1
<Greenlet at 0x2577c10: f(5)> 2
<Greenlet at 0x2577c10: f(5)> 3
<Greenlet at 0x2577c10: f(5)> 4
[Finished in 0.4s]
要让 greenlet交替运行,可以通过 gevent.sleep()交出控制权
def f(n):
for i in range(n):
print gevent.getcurrent(), i
gevent.sleep(0)
遇到 IO操作嘛相当于就切换
输出结果
<Greenlet at 0x2668c60: f(5)> 0
<Greenlet at 0x2668da0: f(5)> 0
<Greenlet at 0x2668df0: f(5)> 0
<Greenlet at 0x2668c60: f(5)> 1
<Greenlet at 0x2668da0: f(5)> 1
<Greenlet at 0x2668df0: f(5)> 1
<Greenlet at 0x2668c60: f(5)> 2
<Greenlet at 0x2668da0: f(5)> 2
<Greenlet at 0x2668df0: f(5)> 2
<Greenlet at 0x2668c60: f(5)> 3
<Greenlet at 0x2668da0: f(5)> 3
<Greenlet at 0x2668df0: f(5)> 3
<Greenlet at 0x2668c60: f(5)> 4
<Greenlet at 0x2668da0: f(5)> 4
<Greenlet at 0x2668df0: f(5)> 4
[Finished in 0.5s]
他们是交替执行的,实际线程数只有一个
在实际代码中,我们不会用 gevent.sleep()去切换协程,而是在执行到IO操作时,gevent自动切换。
下面是使用的 gevent的joinall传入的 gevent.spawn(传入方法,方法的参数)
from gevent import monkey; monkey.patch_all()
import gevent
import urllib2
def f(url):
print('GET: %s' % url)
resp = urllib2.urlopen(url)
data = resp.read()
print('%d bytes received from %s.' % (len(data), url))
gevent.joinall([
gevent.spawn(f, 'https://www.python.org/'),
gevent.spawn(f, 'https://www.yahoo.com/'),
gevent.spawn(f, 'https://github.com/'),
])
结果输出:
GET: https://www.python.org/
GET: https://www.yahoo.com/
GET: https://github.com/
49273 bytes received from https://www.python.org/.
505363 bytes received from https://www.yahoo.com/.
53758 bytes received from https://github.com/.
[Finished in 11.6s]
也就是遇到IO操作就自动切换了
python的多进程
multiprocessing
创建一个子进程
from multiprocessing import Process
import os
def run_proc(name):
print 'Run child process %s (%s)...' % (name, os.getpid())
if __name__=='__main__':
print 'Parent process %s.' % os.getpid()
p = Process(target=run_proc, args=('test',))
print 'Process will start.'
p.start()
p.join()
print 'Process end.'
输出结果:
Parent process 20744.
Process will start.
Run child process test (7884)...
Process end.
[Finished in 0.6s]
p.join()
然让主进程在p这个子进程结束后在结束
使用 Process创建
对应的就应该有进程池
Pool
from multiprocessing import Pool
import os, time, random
def long_time_task(name):
print 'Run task %s (%s)...' % (name, os.getpid())
start = time.time()
time.sleep(random.random() * 3)
end = time.time()
print 'Task %s runs %0.2f seconds.' % (name, (end - start))
if __name__=='__main__':
print 'Parent process %s.' % os.getpid()
p = Pool()
for i in range(5):
p.apply_async(long_time_task, args=(i,))
print 'Waiting for all subprocesses done...'
p.close()
p.join()
print 'All subprocesses done.'
用Pool创建进程池(可以传入进程池里进程数限制)
然后使用 apply_async(传入方法,传入参数)
使用进程池
最后又有join,那么最后要等子进程结束主进程在才能结束
进程间通信,比如一个往Queue里写数据,一个从Queue中读数据
#coding:utf-8
from multiprocessing import Process, Queue
import os, time, random
# 写数据进程执行的代码:
def write(q):
for value in ['A', 'B', 'C']:
print 'Put %s to queue...' % value
q.put(value)
time.sleep(random.random())
# 读数据进程执行的代码:
def read(q):
while True:
if not q.empty():
value = q.get(True)
print 'Get %s from queue.' % value
time.sleep(random.random())
else:
break
if __name__=='__main__':
# 父进程创建Queue,并传给各个子进程:
q = Queue()
pw = Process(target=write, args=(q,))
pr = Process(target=read, args=(q,))
# 启动子进程pw,写入:
pw.start()
# 等待pw结束:
pw.join()
# 启动子进程pr,读取:
pr.start()
pr.join()
# pr进程里是死循环,无法等待其结束,只能强行终止:
print '所有数据都写入并且读完'
输出
Put A to queue...
Put B to queue...
Put C to queue...
Get A from queue.
Get B from queue.
Get C from queue.
多个进程之间操作一个队列
继续
脚本是使用的Gevent的Queue通信
接下来
下一个库
signal
是信号,进程之间通信的方式,是一种软件中断,一个进程一旦接收到信号就会打断原来的程序执行流程
来处理信号
从主函数开始分析
if __name__ == '__main__':
options, args = parse_args()
start_time = time.time()
# make tmp dirs
tmp_dir = 'tmp/%s_%s' % (args[0], int(time.time()))
if not os.path.exists(tmp_dir):
os.makedirs(tmp_dir)
multiprocessing.freeze_support()
all_process = []
dns_servers = load_dns_servers()
next_subs = load_next_sub(options)
scan_count = multiprocessing.Value('i', 0)
found_count = multiprocessing.Value('i', 0)
queue_size_list = multiprocessing.Array('i', options.process)
try:
print '[+] Init %s scan process.' % options.process
for process_num in range(options.process):
p = multiprocessing.Process(target=run_process,
args=(args[0], options, process_num,
dns_servers, next_subs,
scan_count, found_count,queue_size_list,
tmp_dir)
)
all_process.append(p)
p.start()
while all_process:
for p in all_process:
if not p.is_alive():
all_process.remove(p)
groups_count = 0
for c in queue_size_list:
groups_count += c
msg = '[*] %s found, %s scanned in %.1f seconds, %s groups left' % (
found_count.value, scan_count.value, time.time() - start_time, groups_count)
print_msg(msg)
time.sleep(1.0)
except KeyboardInterrupt as e:
for p in all_process:
p.terminate()
print '[ERROR] User aborted the scan!'
except Exception as e:
print e
msg = '[+] All Done. %s found, %s scanned in %.1f seconds.' % (
found_count.value, scan_count.value, time.time() - start_time)
print_msg(msg, line_feed=True)
out_file_name = get_out_file_name(args[0], options)
with open(out_file_name, 'w') as f:
for _file in glob.glob(tmp_dir + '/*.txt'):
with open(_file,'r') as tmp_f:
content = tmp_f.read()
f.write(content)
print '[+] The output file is %s' % out_file_name
显示获取参数,命令行的参数
是写在了lib文件夹中,对命令行参数的这些定义
然后获取启动当前时间作为 开始时间
创建临时目录
tmp_dir = 'tmp/%s_%s' % (args[0], int(time.time()))
以传入的域名加上时间作为临时目录
if not os.path.exists(tmp_dir):
os.makedirs(tmp_dir)
创建临时目录 tmp/baidu.com_时间
multiprocessing.freeze_support()
multiprocessing.freeze_support()是必须加的,不然multiprocessing任务会报错
dns_servers = load_dns_servers()
加载 dns服务
def load_dns_servers():
print_msg('[+] Validate DNS servers', line_feed=True)
dns_servers = []
pool = Pool(10)
for server in open('dict/dns_servers.txt').readlines():
server = server.strip()
if server:
pool.apply_async(test_server, (server, dns_servers))
pool.join()
dns_count = len(dns_servers)
print_msg('\n[+] %s available DNS Servers found in total' % dns_count, line_feed=True)
if dns_count == 0:
print_msg('[ERROR] No DNS Servers available!', line_feed=True)
sys.exit(-1)
return dns_servers
也是写在 lib中的 common.py文件中的 load_dns_servers方法
print_msg('[+] Validate DNS servers', line_feed=True)
又写了一个输出信息的方法 print_msg
设置了左对齐还是右对齐
是否换行
def print_msg(msg=None, left_align=True, line_feed=False):
if left_align:
sys.stdout.write('\r' + msg + ' ' * (console_width - len(msg)))
else: # right align
sys.stdout.write('\r' + ' ' * (console_width - len(msg)) + msg)
if line_feed:
sys.stdout.write('\n')
sys.stdout.flush()
默认左对齐,不换行
这里的console_width 是通过获取什么系统的然后再进行操作。。。。。大佬真的大佬
专门写了一个 py在lib下的console_width
pool = Pool(10)
创建协程池
for server in open('dict/dns_servers.txt').readlines():
server = server.strip()
if server:
pool.apply_async(test_server, (server, dns_servers))
pool.join()
循环读取 dns服务读取,并使用 test_server判断是否是可用的 dns服务
协程执行,
执行后输出 可用的 dns服务
找到了4个可用的 dns服务返回
一行一行读取 字典的dns服务ip
223.5.5.5
223.6.6.6
119.29.29.29
182.254.116.116
并传入 test_server方法,传入参数 ip和创建的 dns_server列表
test_server方法 用来测试 dns服务是否可用
def test_server(server, dns_servers):
resolver = dns.resolver.Resolver(configure=False)
resolver.lifetime = resolver.timeout = 6.0
try:
resolver.nameservers = [server]
answers = resolver.query('public-dns-a.baidu.com') # test lookup an existed domain
if answers[0].address != '180.76.76.76':
raise Exception('Incorrect DNS response')
try:
resolver.query('test.bad.dns.lijiejie.com') # Non-existed domain test
with open('bad_dns_servers.txt', 'a') as f:
f.write(server + '\n')
print_msg('[+] Bad DNS Server found %s' % server)
except:
dns_servers.append(server)
print_msg('[+] Server %s < OK > Found %s' % (server.ljust(16), len(dns_servers)))
except:
print_msg('[+] Server %s <Fail> Found %s' % (server.ljust(16), len(dns_servers)))
这个地方,如何测试:
先确定一个确定的ip对应的域名,如果这个dns没有找到那么就认为是有问题的dns
然后再测试一个不存在的域名,如果没有异常就是一个有问题的dns
都正确就加入 dns服务列表中
获取到可以使用的dns服务的数量
如果等于0就抛出异常,只要大于0就返回这个可以用的dns服务列表
next_subs = load_next_sub(options)
在 common.py中
def load_next_sub(options):
next_subs = []
_set = set()
_file = 'dict/next_sub_full.txt' if options.full_scan else 'dict/next_sub.txt'
with open(_file) as f:
for line in f:
sub = line.strip()
if sub and sub not in next_subs:
tmp_set = {sub}
while tmp_set:
item = tmp_set.pop()
if item.find('{alphnum}') >= 0:
for _letter in 'abcdefghijklmnopqrstuvwxyz0123456789':
tmp_set.add(item.replace('{alphnum}', _letter, 1))
elif item.find('{alpha}') >= 0:
for _letter in 'abcdefghijklmnopqrstuvwxyz':
tmp_set.add(item.replace('{alpha}', _letter, 1))
elif item.find('{num}') >= 0:
for _letter in '0123456789':
tmp_set.add(item.replace('{num}', _letter, 1))
elif item not in _set:
_set.add(item)
next_subs.append(item)
return next_subs
此处扫描子域名下一级域名使用的字典是全部的还是常用的那种。
不使用 –full默认是不使用全部的字典
全部的字典是next_sub_full,默认的是 next_sub.txt都在 dict目录下
如果有指定的{替换的}就替换
加入 set 中,并输出判断是否有替换的字符
加入 netx_subs列表中
返回 next_subs列表 里面存的都是 子域名下一级域名的字典
回看 主函数
定义了两个进程之间的共享变量
scan_count = multiprocessing.Value('i', 0)
found_count = multiprocessing.Value('i', 0)
i的意思就是数字,设置扫描数量,找到数量初始为0
定义 三个进程间共享变量 ,扫描数,找到数,队列大小列表 是个字典
queue_size_list = multiprocessing.Array('i', options.process)
try:
print '[+] Init %s scan process.' % options.process
for process_num in range(options.process):
p = multiprocessing.Process(target=run_process,
args=(args[0], options, process_num,
dns_servers, next_subs,
scan_count, found_count,queue_size_list,
tmp_dir)
)
all_process.append(p)
p.start()
进程数默认是6
创建多进程传入方法 run_process
def run_process(target, options, process_num, dns_servers, next_subs, scan_count, found_count, queue_size_list,
tmp_dir):
signal.signal(signal.SIGINT, user_abort)
s = SubNameBrute(target=target, options=options, process_num=process_num,
dns_servers=dns_servers, next_subs=next_subs,
scan_count=scan_count, found_count=found_count, queue_size_list=queue_size_list,
tmp_dir=tmp_dir)
signal.signal(signal.SIGINT, user_abort)
这个的意思是发起信号当按下键盘的ctrl+c的时候 exit退出程序
s = SubNameBrute(target=target, options=options, process_num=process_num,
dns_servers=dns_servers, next_subs=next_subs,
scan_count=scan_count, found_count=found_count, queue_size_list=queue_size_list,
tmp_dir=tmp_dir)
[dns.resolver.Resolver(configure=False) for _ in range(options.threads)]
初始化创建的时候 值 resolvers
设置多少线程数就创建多少个 dns.resolver.Resolve
elf.queue = PriorityQueue()
定义队列为优先队列
self.local_time = time.time()
定义lcoal_time为那一刻时间
创建类 SubNameBrute
把之前创建的变量用来初始化这个类
之后调用了这个类的 run方法
def run(self):
threads = [gevent.spawn(self._scan, i) for i in range(self.options.threads)]
gevent.joinall(threads)
这个run方法就是协程调用扫描方法 当时传入的参数的来指定创建扫描任务
最主要的扫描方法
def _scan(self, j):
self.resolvers[j].nameservers = [self.dns_servers[j % self.dns_count]]
while not self.queue.empty():
try:
item = self.queue.get(timeout=3.0)[1]
self.scan_count_local += 1
if time.time() - self.local_time > 3.0:
self.scan_count.value += self.scan_count_local
self.scan_count_local = 0
self.queue_size_list[self.process_num] = self.queue.qsize()
except Exception as e:
break
try:
if item.find('{alphnum}') >= 0:
for _letter in 'abcdefghijklmnopqrstuvwxyz0123456789':
self.put_item(item.replace('{alphnum}', _letter, 1))
continue
elif item.find('{alpha}') >= 0:
for _letter in 'abcdefghijklmnopqrstuvwxyz':
self.put_item(item.replace('{alpha}', _letter, 1))
continue
elif item.find('{num}') >= 0:
for _letter in '0123456789':
self.put_item(item.replace('{num}', _letter, 1))
continue
elif item.find('{next_sub}') >= 0:
for _ in self.next_subs:
self.queue.put((0, item.replace('{next_sub}', _, 1)))
continue
else:
sub = item
if sub in self.found_subs:
continue
cur_sub_domain = sub + '.' + self.target
_sub = sub.split('.')[-1]
try:
answers = self.resolvers[j].query(cur_sub_domain)
except dns.resolver.NoAnswer, e:
answers = self.ex_resolver.query(cur_sub_domain)
if answers:
self.found_subs.add(sub)
ips = ', '.join(sorted([answer.address for answer in answers]))
if ips in ['1.1.1.1', '127.0.0.1', '0.0.0.0']:
continue
if self.options.i and is_intranet(answers[0].address):
continue
try:
self.scan_count_local += 1
answers = self.resolvers[j].query(cur_sub_domain, 'cname')
cname = answers[0].target.to_unicode().rstrip('.')
if cname.endswith(self.target) and cname not in self.found_subs:
self.found_subs.add(cname)
cname_sub = cname[:len(cname) - len(self.target) - 1] # new sub
self.queue.put((0, cname_sub))
except:
pass
if (_sub, ips) not in self.ip_dict:
self.ip_dict[(_sub, ips)] = 1
else:
self.ip_dict[(_sub, ips)] += 1
if self.ip_dict[(_sub, ips)] > 30:
continue
self.found_count_local += 1
if time.time() - self.local_time > 3.0:
self.found_count.value += self.found_count_local
self.found_count_local = 0
self.queue_size_list[self.process_num] = self.queue.qsize()
self.local_time = time.time()
msg = cur_sub_domain.ljust(30) + ips
# print_msg(msg, line_feed=True)
self.outfile.write(cur_sub_domain.ljust(30) + '\t' + ips + '\n')
self.outfile.flush()
try:
self.resolvers[j].query('lijiejietest.' + cur_sub_domain)
except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as e:
self.queue.put((999999999, '{next_sub}.' + sub))
except:
pass
except (dns.resolver.NXDOMAIN, dns.name.EmptyLabel) as e:
pass
except (dns.resolver.NoNameservers, dns.resolver.NoAnswer, dns.exception.Timeout) as e:
pass
except Exception as e:
import traceback
traceback.print_exc()
with open('errors.log', 'a') as errFile:
errFile.write('[%s] %s %s\n' % (type(e), cur_sub_domain, str(e)))
只要定义的优先队列中不是空的
while not self.queue.empty():
try:
item = self.queue.get(timeout=3.0)[1]
self.scan_count_local += 1
从队列中取出 设置超时时间为3秒
并且本地扫描数自增1
if time.time() - self.local_time > 3.0:
self.scan_count.value += self.scan_count_local
self.scan_count_local = 0
self.queue_size_list[self.process_num] = self.queue.qsize()
如果当前时间减去 上面初始化时定义的时间大于3秒(这个地方不明白。。)
将值加到类变量后再次初始化扫描值
if item.find('{alphnum}') >= 0:
for _letter in 'abcdefghijklmnopqrstuvwxyz0123456789':
self.put_item(item.replace('{alphnum}', _letter, 1))
continue
elif item.find('{alpha}') >= 0:
for _letter in 'abcdefghijklmnopqrstuvwxyz':
self.put_item(item.replace('{alpha}', _letter, 1))
continue
elif item.find('{num}') >= 0:
for _letter in '0123456789':
self.put_item(item.replace('{num}', _letter, 1))
continue
elif item.find('{next_sub}') >= 0:
for _ in self.next_subs:
self.queue.put((0, item.replace('{next_sub}', _, 1)))
continue
else:
sub = item
如果队列中的有可以替换匹配的进行匹配替换存入
调用的 put_item方法
def put_item(self, item):
num = item.count('{alphnum}') + item.count('{alpha}') + item.count('{num}')
if num == 0:
self.priority += 1
self.queue.put((self.priority, item))
else:
self.queue.put((self.priority + num * 10000000, item))
如果这几个的总数等于0设置优先级+1,存入item
如果大于0,那么就设置优先级让这个数字的总数乘上10000000 存入 item
cur_sub_domain = sub + '.' + self.target
_sub = sub.split('.')[-1]
cur_sub_domain 就是 把主域名前面加上子域名
然后去dns去查询这个子域名是否存在
如果存在:
就在找到的子域名列表中增加
并把子域名对应的ip地址排序写在 ips中,如果这个域名对应的ip中存在 1.1.1.1 127.0.0.0.1 0.0.0.0直接认为这个是不对的,
如果 这个ip是内网(common.py中的is_intranet方法判断)并且是否忽略内网
判断这个子域名是否正确
try:
self.scan_count_local += 1
answers = self.resolvers[j].query(cur_sub_domain, 'cname')
cname = answers[0].target.to_unicode().rstrip('.')
if cname.endswith(self.target) and cname not in self.found_subs:
self.found_subs.add(cname)
cname_sub = cname[:len(cname) - len(self.target) - 1] # new sub
self.queue.put((0, cname_sub))
之后对本地扫描数量+1
查询别名 cname,
如果别名正确那么加入队列中
if (_sub, ips) not in self.ip_dict:
self.ip_dict[(_sub, ips)] = 1
else:
self.ip_dict[(_sub, ips)] += 1
if self.ip_dict[(_sub, ips)] > 30:
continue
如果
对应的这个ip的这个字典是否存在,不存在就设置值为1
存在就将值加1,如果这个字典的值大于了30就结束
找到的子域名总数加1
设置找到的子域名总数加上当前找到的
清空当前找到的子域名总数
重新设置本地时间和队列
self.outfile.write(cur_sub_domain.ljust(30) + '\t' + ips + '\n')
self.outfile.flush()
try:
self.resolvers[j].query('lijiejietest.' + cur_sub_domain)
except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as e:
self.queue.put((999999999, '{next_sub}.' + sub))
except:
pass
将子域名写到文件中。
并查询一个错误的子域名,返回异常才是正确的于是设置优先级999999999,添加
'{next_sub}.+子域名
下一级的子域名加入
到队列中
回到主函数中,这个是每个进程的执行的
加入所有进行列表,启动进行
while all_process:
for p in all_process:
if not p.is_alive():
all_process.remove(p)
groups_count = 0
for c in queue_size_list:
groups_count += c
msg = '[*] %s found, %s scanned in %.1f seconds, %s groups left' % (
found_count.value, scan_count.value, time.time() - start_time, groups_count)
print_msg(msg)
time.sleep(1.0)
只要还有进程,递归处理,
判断进程是否活跃,不是则从列表中删除
从队列中递归出来子域名
输出信息,找到的域名数,扫描数,花费时间,队列数
except KeyboardInterrupt as e:
for p in all_process:
p.terminate()
print '[ERROR] User aborted the scan!'
except Exception as e:
print e
如果中断脚本(ctrl+c) 那么就把结束所有进程
out_file_name = get_out_file_name(args[0], options)
with open(out_file_name, 'w') as f:
for _file in glob.glob(tmp_dir + '/*.txt'):
with open(_file,'r') as tmp_f:
content = tmp_f.read()
f.write(content)
print '[+] The output file is %s' % out_file_name
输出到之前定义的临时文件
加载子域名方法
_load_sub_names()
def _load_sub_names(self):
if self.options.full_scan and self.options.file == 'subnames.txt':
_file = 'dict/subnames_full.txt'
else:
if os.path.exists(self.options.file):
_file = self.options.file
elif os.path.exists('dict/%s' % self.options.file):
_file = 'dict/%s' % self.options.file
else:
print_msg('[ERROR] Names file not found: %s' % self.options.file)
exit(-1)
normal_lines = []
wildcard_lines = []
wildcard_list = []
regex_list = []
lines = set()
with open(_file) as f:
for line in f.xreadlines():
sub = line.strip()
if not sub or sub in lines:
continue
lines.add(sub)
if sub.find('{alphnum}') >= 0 or sub.find('{alpha}') >= 0 or sub.find('{num}') >= 0:
wildcard_lines.append(sub)
sub = sub.replace('{alphnum}', '[a-z0-9]')
sub = sub.replace('{alpha}', '[a-z]')
sub = sub.replace('{num}', '[0-9]')
if sub not in wildcard_list:
wildcard_list.append(sub)
regex_list.append('^' + sub + '$')
else:
normal_lines.append(sub)
if regex_list:
pattern = '|'.join(regex_list)
_regex = re.compile(pattern)
for line in normal_lines[:]:
if _regex.search(line):
normal_lines.remove(line)
for item in normal_lines[self.process_num::self.options.process]:
self.priority += 1
self.queue.put((self.priority, item))
for item in wildcard_lines[self.process_num::self.options.process]:
self.queue.put((88888888, item))
先判断是否是全部扫描。
确实使用的字典
打开这个文件读取子域名字典
if sub.find('{alphnum}') >= 0 or sub.find('{alpha}') >= 0 or sub.find('{num}') >= 0:
wildcard_lines.append(sub)
sub = sub.replace('{alphnum}', '[a-z0-9]')
sub = sub.replace('{alpha}', '[a-z]')
sub = sub.replace('{num}', '[0-9]')
if sub not in wildcard_list:
wildcard_list.append(sub)
regex_list.append('^' + sub + '$')
wildcard_lines如果有可以替换的字符就加入进去
如果是没有可替换字符的就加入 normal_lines 里面
if regex_list:
pattern = '|'.join(regex_list)
_regex = re.compile(pattern)
for line in normal_lines[:]:
if _regex.search(line):
normal_lines.remove(line)
如果正则列表不是空
如果 匹配到了就从 normal_lines 中删除这个,因为 wildcard_list里面已经有了。
for item in normal_lines[self.process_num::self.options.process]:
self.priority += 1
self.queue.put((self.priority, item))
for item in wildcard_lines[self.process_num::self.options.process]:
self.queue.put((88888888, item))
从没有替换的类别中存入队列中
从右替换的存入队列中并设置优先级为 88888888
总体流程进行分析
先根据配置的获取dns服务,然后测试dns服务是否可用,解析子域名和子域名下一级
定义的是共享的协程的队列,将子域名加载到队列中,拼接成域名访问 dns服务获取是否存在,存在就记录,并添加下一级域名字符解析,之后存入队列,
将字符串解析替换为下一级子域名,再次访问。。。
try:
self.resolvers[j].query('lijiejietest.' + cur_sub_domain)
except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as e:
self.queue.put((999999999, '{next_sub}.' + sub))
except:
在成功的域名后,再将成功的域名加上 {next_sub}.
添加到 子域名队列中
elif item.find('{next_sub}') >= 0:
for _ in self.next_subs:
self.queue.put((0, item.replace('{next_sub}', _, 1)))
continue
会被替换到将所有的 next_sub 替换为 字典中的域名再次添加到域名队列中
默认6个进程数,执行创建协程,扫描是在多进程中的协程中执行的,
添加子域名的时候是创建类的时候添加的,还有确认域名正确后测试下一级域名会添加到队列中。
还有些地方不懂,如有错误希望大佬指点下。