愚公移山日记·15

愚公移山日记·15

学习进度

前天说到爬取一个网页的免费IP地址，由于昨天的学习进度很慢，仅仅弄检验IP的一点点，所以昨天没有发博客，今天本以为会把多线程解决掉，但是事与愿违，还是有点难度的。
检验IP的代码：

import requests
import re
from fake_useragent import UserAgent
def get_html(url):
    count = 0
    while True:
        headers = {'user-agent':UserAgent().random}
        response = requests.get(url,headers = headers)
        if response.status_code == 200:
            response.encoding = 'utf-8'
            return response
        else :
            count += 1
            if count == 3:
                return
            else:
                continue
 
def get_infos(response):
    num = re.findall(r'<tr>[\s\S]*?<td>(.*?)</td>',response.text)
    return num   
if __name__ == '__main__':
    urls = ['http://www.xiladaili.com/gaoni/{}/'.format(str(i)) for i in range(1,2)]
    for url in urls :
        response = get_html(url)
        num = get_infos(response)
        for i in num:
             try:
                 requests.get('http://wenshu.court.gov.cn/', proxies={"http":'http://' +  i})
             except:
                 print ('connect failed')
             else:
                 print ('success')

结果如下图

这方法很慢只能一条一条的去逐个检验，很慢。
下面我来搞一下我今天的学习结果，虽然没有解决重要的问题，但是希望路过的看官，留下您宝贵的意见。

import requests
import re
from fake_useragent import UserAgent
import threading
def get_html(url):
    count = 0
    while True:
        headers = {'user-agent':UserAgent().random}
        response = requests.get(url,headers = headers)
        if response.status_code == 200:
            response.encoding = 'utf-8'
            return response
        else :
            count += 1
            if count == 3:
                return
            else:
                continue
def get_infos(response):
    num = re.findall(r'<tr>[\s\S]*?<td>(.*?)</td>',response.text)
    return num
def try_ip(ip):
    try:
        requests.get('http://wenshu.court.gov.cn/', proxies={"http":'http://' +  ip})
    except:
        print ('connect failed')
    else:
        print ('success')
             
if __name__ == '__main__':
    urls = ['http://www.xiladaili.com/gaoni/{}/'.format(str(i)) for i in range(1,2)]
    for url in urls :
        response = get_html(url)
        num = get_infos(response)
        for i in num:
            t = threading.Thread(target=try_ip,args=(i,))
            t.start()
            t.join()

这上面是我用threading模块写的代码但是问题不知出在那里，加上这个之后速度还是很慢，而且检验IP地址出现问题，并不准确。

希望路过的看官留下您宝贵的意见。