免费代理爬虫遭遇JavaScript

  在github看到了开源项目IPProxyTool,因而以此为基础自己维护一个抓取免费代理ip的项目。个人对项目进行了一些改动，细节这里就不提了，这里简单记录下我新添加的爬虫中的goubanjia，nyloner。因为这这个网址使用了一些反爬技巧。
  爬虫过程经常会遇到JavaScript反爬虫，如果JavaScript代码可读性强，那么写出相应的python代码，效率是最高的。如果JavaScript代码可读性非常差，也可以考虑使用基于selenium的phantomjs。本文采用的是第一种。
  首先查看goubanjia的网页源码

<td class="ip">
    <span style='display: inline-block;'></span>
    <span style='display: inline-block;'>1</span>
    <p style='display: none;'>5</p>
    <span>5</span>
    <div style='display: inline-block;'>6</div>
    <p style='display: none;'></p>
    <span></span>
    <span style='display: inline-block;'>.</span>
    <p style='display: none;'></p>
    <span></span>
    <p style='display: none;'>67</p>
    <span>67</span>
    <span style='display: inline-block;'>.</span>
    <div style='display: inline-block;'></div>
    <p style='display: none;'>2</p>
    <span>2</span>
    <p style='display: none;'></p>
    <span></span>
    <div style='display: inline-block;'>19</div>
    <span style='display: inline-block;'></span>
    <span style='display: inline-block;'>.6</span>
    <div style='display: inline-block;'>1</div>:
    <span class="port CFACE">8364</span>
</td>

  代理ip好办，用xpath提取即可。

    ipList=row.xpath('./td[1]/div[@style]/text() | ./td[1]/span[@style]/text() | ./td[1]/span/text()')[:-1]
                ip = ''.join(ipList)
    item['ip']=ip
    portstr = row.xpath('./td[1]/span[contains(@class,"port")]/@class')[0] #['port EAEDCA'] 

  端口8255，8255却与实际端口3128不一样，玄机在哪呢？
  首先找个相同的端口3128看看： 8110 ，数字不同，”port CFACE”完全相同，我们把关注点放在”port XXXXX”上面，很容易知道实际端口可以通过”CFACE”解密而得。直接上解密代码：

def get_poxy(self,word): 
    num_list = [] 
    for item in word: 
        num = 'ABCDEFGHIZ'.find(item) 
        num_list.append(str(num)) 
    port = int("".join(num_list)) >> 0x3
    return port

  再来看看nyloner，这是个个人维护的网站，首先还是右键查看网页源码，会发现完全没有代理ip的信息，不必惊慌，我们先F12打开开发者工具，然后点下一页，第二页的ip显示出来了，同时会发现仅仅多出了一个请求https://www.nyloner.cn/proxy?page=2&num=15&token=7a65c88d591e3096135db3d9770f26b8&t=1508988424.显然第二页的ip信息，就在这个请求的回复里面，我们看下回复的内容:

{"status": "true", "list": "OUofAQ89MwA2BS4AKCY/SCAFJx89PU1YIi1RCiM9OQEiJjMAGitWFwExDU8lLA0fPzowWCUXEhUkSj4fDDInADYFLgAoGC8BIhZeHT8tSRUhByQKIy0DFiAPHV83FQtXKTEsTiZdAhI4FBYLJRQgCiMAWBYjMSMbNDgiGyghDQolLAoSEF0zXCUUChUkAwsYIyEzADUvLgABNQINNjwNUzstGhUiKiBBIi0pGCIxO1wwKCoWKhgnSyMFO1Y4XUkfJiYWHA8hLQEhDDMANDgmGSkYDUogBScSPz1NXCEHBgEnOggYDFY4XjMWAAkvCAkOIysvCz4tOAILKQkGNCoPWScmERc0KCpdKSYrDiA7J1c7KjwUIAQsQCETOVwkVkIdMCQcAAQqLxcjBi8LPzo8WSMELEYiEyleICZGFjQ4IgApMS8XDyhXHBYtGlomLQZGIz0LGCQcBQkzXz0eByU7FyMGLws/BDgUIRdVCiM6XBYgDDMWNDgAFysYAUgiFgpcPi07WyVcCQUkAwMIJB83FjYvW1wqIVpLITxaVDwXGh8mLQEFDEomXyQfHQkzFghaKjYJFyAsLwsWKRUYNT0GRCc6DxYjITddNTgqGSkmJ0slKysdPQQwXiAEMEEkSlwcJy0BABg0LgAqGy8XITsrVz4EPFkgPVELIhMlGCEmER0wLwsZB0EkSSYVAQI4FB4bJRcSFSRKPh8MMicANgUuACgYLwEiFl4dPy1JFSEHJAojLQMWIA8dXzcVC1cpMSxOJl0CEjgUFgslFCAKIBBYFyExGRs0BVsWKiYNFyAsLwsRKUAVCy0GRCc6D1sjJhEdMC8LXgQlXxUmFQECOBQwGyI6BgYjLS0bIw8rCTQ4KlgoGDdPIjsNCxQ9DgsKFwECDToPWScmERY0ODYbKCYFTCAFJxA/OhJeJRcSFSRKLhkNCyMANgUuACo2LxcgLC8LFikVGDU9BkQnOg8WIyE3XTU4KhkpJidLJSsrHT0EMF4gBDBBJEpcHCctAQAYNC4AKhsvFyE7IxA8FB4ZIjosQyITKV0gHBEdMC8LGQdBJEkmFQECOBQeGyAqJBwiOi0BCiIeGiM/DFgsMQ0AISsrVj46PBsjOixAJz0pFyEPO1w2BjpcL0FeCiUnHQsTNjgCIAckHCMtC14iDxlfNQYqWisbWkojLA0WOy0dGw1dL0IkAwMIJB8rXzQ4LhQvCxkeJlw8FRA5LAIgByQcIxMtFyAcQhY0L18XKxsvASE7ARw8BBZdIT0BSyI6LlgkVx4ZMxYACS8IJwwhOwlWPgQ0XSMEIAsjEA8cJyYWGRtfJV4vCAEeJhUJEj8qOAIjLSQcCj4AGzQ2EVgwLwwXKDYrSiA7KxI+OjBeJiogCiETJV0hDydfM19fHSw6HRcNNy8LPQc4AiI6DkIiEyleIg83FzQFWxcrNjcXICwvCxEpQBULLQZEJzoPXiMfO1o0PwwdLDEKSQ04Xwk4FBYLJRQsBSMtDxsjMTMaNAY2CSgmK08hBTdTPDooAgk9EhULAAgfDSYRWDAvDBYqJjcMITs7ED8EKBYjBCwFJAAbCCRWMBgaAj4AKhsvFyMrL1E/LRofJi0BQg8uXQMkHx0JMxYmGSgmDQ0hOy8RPwQgCyI6IEQjEzVZIDEjAB8/GAkACwoJDywNUzstGhQhBDQHIy01GCIPN140BVtcKDENCiUsChIQXTNcJRQKFSQDORUjDxUWMxUYCS9BPAkOODsLPQc4AiIEJAogAFwXIyZCFzcFLhYoJgEAIgUBVD8XHVUjLSdFJEsAGCQfHQkzFipbKCFaSiAFK1U/B00UIgQCHCI6LQENIksXHS8MWCwxDUghFSdRPz0aHyYtAUIPLl0DJB8dCTMWJhkoJg0NITsvET8EIAsiOiBEIxM1WSAxPwAfOl9T"}

  什么鬼？list的内容看起来像乱码，我们回头找网页的javascript，发现有如下几个脚本:

<script src="../static/js/jquery.min.js"></script>
<script src="../static/js/bootstrap.min.js"></script>
<script src="../static/js/md5.js"></script>
<script src="../static/js/proxy.js"></script>
<script src="../static/js/base64.js"></script>

  这其中md5，base64是常见的加密方法，很自然就关注到proxy.js，我们打开proxy.js看看里面的源码：

function get_proxy_ip(page, num, click_btn) {
    var timestamp = Date.parse(new Date());
    timestamp = timestamp / 1000;
    var token = md5(String(page) + String(num) + String(timestamp));
    $.get('../proxy?page=' + page + '&num=' + num + '&token=' + token + '&t=' + timestamp, function (result) {
        if (result.status === 'true') {
            var setHtml = "";
            $("#ip-list").html(setHtml);
            var encode_str = result.list;
            var items = strToJson(decode_str(encode_str));
            for (var index = 0; index < items.length; ++index) {
                item = items[index];
                setHtml += "<tr>\n<td>" + (index + 1) + "</td>\n";
                setHtml += "<td>" + item.ip.toString() + "</td>\n";
                setHtml += "<td>" + item.port.toString() + "</td>\n";
                setHtml += "<td>" + item.time.toString() + "</td>\n</tr>\n";
            }
            $("#ip-list").html(setHtml);
            if (click_btn === 'next') {
                document.getElementById("last-page").disabled = false;
                if (items.length < 15) {
                    document.getElementById("next-page").disabled = true;
                }
            } else {
                document.getElementById("next-page").disabled = false;
                if (page === 1) {
                    document.getElementById("last-page").disabled = true;
                }
            }

        }
    });
}

  很开心里面的代码可读性非常高，看到这个函数function get_proxy_ip(page, num, click_btn)了吗？里面的代码简单分为1：先根据页数+每页ip数+时间戳经过md5加密获得一个token，2：然后根据页数&每页ip数&token&时间戳组成一个url，我们依样画瓢写出get_url()函数。

   def geturl(self,page,limitnum=15):     
        timestamp = datetime.now().timestamp()
        timestamp = int(timestamp)
        token = str(page) + str(limitnum) + str(timestamp)
        token = hashlib.md5(token.encode()).hexdigest()
        url = "https://www.nyloner.cn/proxy?page={page}&num={num}&token={token}&t={t}".format(page=page,num=limitnum,token=token,t=timestamp)
        return url

  使用requests测试一下，返回了一长串类似乱码的字符串。这里面的信息应该如何解密呢？我们继续看刚才的get_proxy_ip(),可以看到解密的代码，先使用decode_str解密，再转成json。

var items = strToJson(decode_str(encode_str))

function decode_str(scHZjLUh1) {
    scHZjLUh1 = Base64["\x64\x65\x63\x6f\x64\x65"](scHZjLUh1);
    key = '\x6e\x79\x6c\x6f\x6e\x65\x72';
    len = key["\x6c\x65\x6e\x67\x74\x68"];
    code = '';
    for (i = 0; i < scHZjLUh1["\x6c\x65\x6e\x67\x74\x68"]; i++) {
        var coeFYlqUm2 = i % len;
        code += window["\x53\x74\x72\x69\x6e\x67"]["\x66\x72\x6f\x6d\x43\x68\x61\x72\x43\x6f\x64\x65"](scHZjLUh1["\x63\x68\x61\x72\x43\x6f\x64\x65\x41\x74"](i) ^ key["\x63\x68\x61\x72\x43\x6f\x64\x65\x41\x74"](coeFYlqUm2))
    }
    return Base64["\x64\x65\x63\x6f\x64\x65"](code)
}

function strToJson(str) {
    return (new Function("return " + str))();
}

  通过代码，可以知道decode_str()的原理：先把 scHZjLUh1通过base64解密，然后每7个字节分别与key=”nyloner”进行异或运算，得到一个新字符串，再base64解密。同样依样画瓢可以下出decode_str(),代码如下：

  def decode_str(self,scHZjLUh1):        
        scHZjLUh1 = base64.decodestring(scHZjLUh1)
        key=b'nyloner' 
        lenth= len(key)
        schlenth=len(scHZjLUh1)
        code=''
        for i in range(schlenth):
            coeFYlqUm2 = i % lenth
            code+= chr(scHZjLUh1[i] ^ key[coeFYlqUm2])
        code = base64.decodestring(code.encode())
        iplist = code.decode()
        return iplist

  我们运行一下代码，即可成功获取ip列表。