子域名收集工具编写

子域名收集工具编写

多线程爆破

1.提取文件中的子域名字典，循环将subdomain.domain(测试子域名)压入队列中，准备传入多线程类。

q = queue.Queue()
        path = os.path.join(common.get_root_path(), 'dir\dic.txt')
        with open(path, 'r') as f:
            r = f.readlines()
        for i in r:
            i = i.strip('\n')
            url = '{subdomain}.{domain}'.format(subdomain=i, domain=self.domain)
            q.put(url)

2.多线程类:从队列中获得url，调用dns库的resolve进行A类型解析，如果能解析出A记录，即根据域名解析出了对应的ip，则认为此子域名存在存入result数组中。

class Brute(threading.Thread):
	   def __init__(self, q, result, total):
	       threading.Thread.__init__(self)
	       self.q = q
	       self.result = result
	       self.total = total
	
	   def run(self) -> None:
	       while not self.q.empty():
	           url = self.q.get()
	           self.msg()
	           try:
	               # 调用dns库中的resolve方法对url进行A类型dns查询
	               A = resolver.resolve(url, 'A')
	               if A.response.answer:
	                   self.result.append(url)
	           except Exception:
	               pass
	
	   def msg(self):
	       left = self.q.qsize()
	       total = self.total
	       per = ((total - left) / total) * 100
	       sys.stdout.write('\r{} left {} total|{:.2f}% scan'.format(left, total, per))

3.运行多线程模块爬取结果

st = time.time()
print('开始爆破子域名{}'.format(time.strftime('%X')))
threads = []
result = []
thread_count = 2000
total = q.qsize()
for i in range(thread_count):
    threads.append(self.Brute(q, result, total))

for t in threads:
    t.start()

for t in threads:
    t.join()

path = common.get_result(domain=self.domain, type='brute.json')
common.save_result(path, result)
print('\n爆破结束{}  共计用时{:.2f}s'.format(time.strftime('%X'), time.time() - st))

多进程和协程爬取

本模块采用多进程和协程爬取第三方网站子域名记录 以chinaz为例
1.先获取chinaz针对domain查询的结果总页数，把所有结果页面的url存入列表中

def get_urls(domain):
    urls = []
    url = 'https://tool.chinaz.com/subdomain?domain=' + domain
    # 得到一个总页数
    try:
        r = requests.get(url=url)
        soup = BeautifulSoup(r.text, features='lxml')
        pages = soup.find(name='span', attrs={'class': 'col-gray02'})
        s = pages.string
        index_p = s.index('页')
        page = int(s[1:index_p])
        print('chinaz查询总页数:' + str(page))
        # 得到所有的结果页面url
        for i in range(1, page + 1):
            urls.append('https://tool.chinaz.com/subdomain?domain=' + domain + '&page=' + str(i))
        return urls
    except AttributeError:
        print('chinaz收集失败，没有查询到相关的子域名')
        return None

2.协程函数:访问传入的chinaz页面并获得当前页面查询的所有子域名结果，存入subdomains列表

async def spider2(url, subdomains):
    # print('正在请求',url)
    async with(sem):
        async with aiohttp.ClientSession() as session:
            async with session.request('GET', url) as resp:
                # print(resp.status)
                # 解析网页这个行为可以被切换不用一直等待
                content = await resp.text()
                # 从aiohttp response中解析出子域名的链接
                soup = BeautifulSoup(content, features='lxml')
                subs = soup.find_all(name='div', attrs={'class': 'w23-0 subdomain'})
                for sub in subs:
                    # print(sub.a.string)
                    subdomains.append(sub.a.string)

3.定义事件循环列表，打包协程任务加入事件循环并启动

class chinaz(object):
    def __init__(self, domain):
        self.domain = domain

    def run(self):
        st = time.time()
        print('begin to collect from chinaz:{}'.format(time.strftime('%X')))
        subdomains = []
        urls = get_urls(self.domain)
        if not urls:
            print('chinaz没有查询到子域名')
            return
        # 打包任务 并开启事件循环
        coroutines = []
        for url in urls:
            coroutines.append(spider2(url, subdomains))
        loop = asyncio.get_event_loop()
        loop.run_until_complete(asyncio.wait(coroutines))
        # 将得到的subdomains结果数组保存
        path = common.get_result(domain=self.domain, type='chinaz.json')
        common.save_result(path, subdomains)
        print('end to collect from chinaz:{}   共计用时{:.2f}s'.format(time.strftime('%X'), time.time()-st))