[Python]自学笔记36：论一只爬虫的自我修养3：隐藏

最新推荐文章于 2020-12-03 18:43:11 发布

MyLight1109

最新推荐文章于 2020-12-03 18:43:11 发布

阅读量1.1k

点赞数

分类专栏： Python自学文章标签： python

本文链接：https://blog.csdn.net/weixin_46285081/article/details/105564248

版权

Python自学专栏收录该内容

39 篇文章 1 订阅

订阅专栏

有一些网站比较讨厌爬虫，它们不想被爬虫访问，所以它们会检查链接的来源，若不是正常途径，那么就会被屏蔽。
这就需要将爬虫隐藏
有下面几种方法

一.修改headers
网站一般检查链接的headers里面的USER-Agent，如果不加以修改，爬虫的USER-Agent就会是Python的版本号，所以我们需要自己修改。
我们需要利用利用urllib.request.Request(url,data,headers={},origin_req_host=None,unverifiable=False,method=None)
来修改headers。

一共有两种方法！
**方法一:**通过Request的headers参数修改
首先，我们按照上一节的例子（爬取有道翻译功能），我们在网页的post操作中找到一个Request headers的表单，复制里面的USER-Agent
在这里插入图片描述
第二步，我们在原有的的代码基础上进行修改
我们需要定义一个head字典，并将其赋值，并传入Request方法中。

import urllib.request
import urllib.parse
import json

url = "http://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule"

head = {} #定义一个head字典
head['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36'   #将Users-Agent的值复制进去。

content = input("请输入需要翻译的内容:")

data = {}
data['i']= content
data['from']= 'AUTO'
data['to']= 'AUTO'
data['smartresult']= 'dict'
data['client']= 'fanyideskweb'
data['salt']= '15868495054215'
data['sign']= '84a37daf45c409c09312a03f5b000bc7'
data['ts']= '1586849505421'
data['bv']= '887ddef35bb193fe341b77b7709d1160'
data['doctype']= 'json'
data['version']= '2.1'
data['keyfrom']= 'fanyi.web'
data['action']= 'FY_BY_CLICKBUTTION'
data = urllib.parse.urlencode(data).encode('utf-8')

req = urllib.request.Request(url,data,head)   #此处增加一个head项，以保证将字典传入。
response = urllib.request.urlopen(req)
html = response.read().decode('utf-8')
target = json.loads(html)
print('翻译结果为：%s' % target['translateResult'][0][0]['tgt'])

第三步，我们进行调试运行。

请输入需要翻译的内容:love
翻译结果为：爱
>>> req.headers
{'User-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36'}

代码正常运行，Request方法中的headers为修改后的。

**方法二：**通过Request.add_header()方法修改
我们在上一节的代码上进行修改

import urllib.request
import urllib.parse
import json

url = "http://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule"

content = input("请输入需要翻译的内容:")

data = {}
data['i']= content
data['from']= 'AUTO'
data['to']= 'AUTO'
data['smartresult']= 'dict'
data['client']= 'fanyideskweb'
data['salt']= '15868495054215'
data['sign']= '84a37daf45c409c09312a03f5b000bc7'
data['ts']= '1586849505421'
data['bv']= '887ddef35bb193fe341b77b7709d1160'
data['doctype']= 'json'
data['version']= '2.1'
data['keyfrom']= 'fanyi.web'
data['action']= 'FY_BY_CLICKBUTTION'
data = urllib.parse.urlencode(data).encode('utf-8')

req = urllib.request.Request(url,data)   #此处先定义Request方法
req.add_header('User-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36')   #
利用add_header()方法将字典添加到req中。
response = urllib.request.urlopen(req)    #访问req
html = response.read().decode('utf-8')
target = json.loads(html)
print('翻译结果为：%s' % target['translateResult'][0][0]['tgt'])

第二步，运行代码

请输入需要翻译的内容:hate
翻译结果为：讨厌
>>> req.headers
{'User-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36'}

运行成功！

二.使用间歇式访问
我们可以利用time模块中的sleep方法，间歇性的执行程序，从而让网站认为是正常人在使用，来达到隐藏的目的。
代码如下：

import urllib.request
import urllib.parse
import json
import time

while True:

    content = input("请输入需要翻译的内容:(输入'q'退出程序)")
    if content == 'q':
        break

    url = "http://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule"

    data = {}
    data['i']= content
    data['from']= 'AUTO'
    data['to']= 'AUTO'
    data['smartresult']= 'dict'
    data['client']= 'fanyideskweb'
    data['salt']= '15868495054215'
    data['sign']= '84a37daf45c409c09312a03f5b000bc7'
    data['ts']= '1586849505421'
    data['bv']= '887ddef35bb193fe341b77b7709d1160'
    data['doctype']= 'json'
    data['version']= '2.1'
    data['keyfrom']= 'fanyi.web'
    data['action']= 'FY_BY_CLICKBUTTION'
    data = urllib.parse.urlencode(data).encode('utf-8')

    req = urllib.request.Request(url,data)
    req.add_header('User-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36')
    response = urllib.request.urlopen(req)
    html = response.read().decode('utf-8')
    target = json.loads(html)
    print('翻译结果为：%s' % target['translateResult'][0][0]['tgt'])
    time.sleep(5)

执行结果：

请输入需要翻译的内容:(输入'q'退出程序)love
翻译结果为：爱
请输入需要翻译的内容:(输入'q'退出程序)hate
翻译结果为：讨厌
请输入需要翻译的内容:(输入'q'退出程序)q

三.代理
步骤
1.参数是一个字典{‘类型’ : ‘代理ip:端口号’}
proxy_support = urllib.request.ProxyHandler({})

2.定制，创建一个opener
opener = urllib.request.build_opener(proxy_support)

3a.安装opener
urllib.request.install_opener(opener)
3b.调用opener
opener.open(url)

当然，opener也可以自己DIY，下面代码中就自己定义了headers。

import urllib.request

url = 'https://tool.lu/ip/'

proxy_support = urllib.request.ProxyHandler({'http':'221.230.143.62:80'})

opener = urllib.request.build_opener(proxy_support)
opener.addheaders = [('USER-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36')]

urllib.request.install_opener(opener)

response = urllib.request.urlopen(url)
html = response.read().decode('utf-8')

print(html)

执行结果：

============== RESTART: C:/Users/Administrator/Desktop/proxy_eg.py =============
<!DOCTYPE html>
<html lang="zh-cmn-Hans">
<head>
    <meta charset="UTF-8">
    <link rel="dns-prefetch" href="//s1.tool.lu">
    <link rel="dns-prefetch" href="//s2.tool.lu">
    <link rel="dns-prefetch" href="//s3.tool.lu">
    <link rel="dns-prefetch" href="//s4.tool.lu">
    <link rel="dns-prefetch" href="//qn11.tool.lu">
    <link rel="dns-prefetch" href="//qn12.tool.lu">
    <link rel="dns-prefetch" href="//qn13.tool.lu">
    <link rel="dns-prefetch" href="//qn14.tool.lu">
    <link rel="dns-prefetch" href="//analytics.tool.lu">
    <title>IP地址查询 - 在线工具</title>
    <link rel="search" type="application/opensearchdescription+xml" title="在线工具"
          href="https://tool.lu/opensearch.xml">
    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=5.0, viewport-fit=cover">
    <meta name="author" content="xiaozi">
    <link rel="publisher" href="https://plus.google.com/113621341737546487026">
    <link rel="canonical" href="https://tool.lu/ip/">
    <meta property="og:locale" content="zh_CN">
    <meta property="og:title" content="IP地址查询">
    <meta property="og:site_name" content="在线工具">
    <meta property="og:description"
          content="IP地址查询,签名图片">
    <meta property="og:type" content="website">
    <meta property="og:image" content="//qn13.tool.lu/201908/25/232334CvNgX3l2slPFu8DK_1200x630.png">
    <meta property="og:url" content="https://tool.lu/ip/">
    <meta name="keywords" content="IP地址查询,签名图片,IP签名档,个性动态签名IP">
    <meta name="description" content="IP地址查询,签名图片">
    <link rel="shortcut icon" href="https://tool.lu/favicon.ico">
    <meta name="apple-mobile-web-app-title" content="在线工具">
    <meta name="format-detection" content="telephone=no">
    <meta name="apple-mobile-web-app-capable" content="yes">
    <meta name="apple-mobile-web-app-status-bar-style" content="grey">
    <link rel="apple-touch-icon" sizes="57x57" href="//qn11.tool.lu/201711/08/002818pvCvPDnAZK6IioEb_57x57.png">
    <link rel="apple-touch-icon" sizes="72x72" href="//qn11.tool.lu/201711/08/002819Dt982ULrn0pC7AL1_72x72.png">
    <link rel="apple-touch-icon" sizes="114x114" href="//qn11.tool.lu/201711/08/002818ZlzSK5t2HM5icUJN_114x114.png">
    <link rel="apple-touch-icon" sizes="144x144" href="//qn11.tool.lu/201711/08/002819v0Gaydtvy2P4y03G_144x144.png">
    <!--[if lt IE 9]>
    <script src="//s1.tool.lu/__/6f8c881821420275fc59b30c6999a75b.js"></script>
    <![endif]-->
    <link rel="stylesheet" href="//s2.tool.lu/css/fontawesome/css/all.min.css">
    <link rel="stylesheet" href="//s3.tool.lu/__/a5b2a3fd62df3d5d62a029e6d212697d.css">
    <link rel="stylesheet" href="//s4.tool.lu/__/081a8409ef017a2f484b1874c506db4d.css">
            <script src="//analytics.tool.lu/te.js" crossorigin="anonymous"></script>
        <script>
        var _hmt = _hmt || [];
        (function () {
            var hm = document.createElement("script");
            hm.src = "https://hm.baidu.com/hm.js?0fba23df1ee7ec49af558fb29456f532";
            var s = document.getElementsByTagName("script")[0];
            s.parentNode.insertBefore(hm, s);
        })();
    </script>
    <script src="//s1.tool.lu/__/060febbce34c518547c352ebc00588bd.js"></script>
    </head>
<body>
<!--[if lt IE 9]>
<div class="notice chromeframe">您的浏览器版本<strong>很旧很旧</strong>，为了正常地访问网站，请升级您的浏览器 <a target="_blank"
                                                                                   href="http://browsehappy.com">立即升级</a>
</div>
<![endif]-->
<div class="g-nav clearfix">
    <ul>
        <li><a href="//type.so/">技术</a></li>
        <li class="hidden-xs"><a href="//tool.lu/">在线工具</a></li>
    </ul>
    <ul class="fr" id="js_login_bar" data-url="https://tool.lu/hello?callback=?">
    </ul>
</div>
<div id="wrap">
    <div id="hdr">
        <div class="w">
            <a class="logo" rel="nofollow" href="https://tool.lu/">
                在线工具<!--<span class="sup"></span>--></a>
            <h1 class="hidden">IP地址查询 | 签名图片</h1>
            <form name="search" id="srch" class="hidden-xs hidden-sm" action="https://tool.lu/search/"
                  method="GET">
                <input type="text" class="text" name="query" value=""
                       placeholder="搜索其实很简单">
                <button type="submit">搜索</button>
                <p class="hot-keywords">
                    <a target="_blank"
                             href="https://tool.lu/search/?query=%E6%AD%A3%E5%88%99">正则</a><a target="_blank"
                             href="https://tool.lu/search/?query=%E8%A7%A3%E5%AF%86">解密</a><a target="_blank"
                             href="https://tool.lu/search/?query=%E8%BF%90%E8%A1%8C%E4%BB%A3%E7%A0%81">运行代码</a>                </p>
            </form>
            <div id="eles">
                                <ul>
                                        <li>
                        <a href="https://tool.lu/top/">
                            <i class="fas fa-list-ol"></i>
                            <span>排行榜</span>
                        </a>
                    </li>
                </ul>
            </div>
        </div>
    </div>
    <div id="nav" class="hidden-xs hidden-sm">
        <div class="w">
            <div class="nav-inner clearfix">
                <ul class="clearfix" style="float: left;">
                    <li><a class="nav-mine" style="color: #34495e;" href="https://tool.lu/mine/"
                            rel="nofollow">我的</a></li>
                    <li><a href="https://tool.lu/" rel="nofollow">所有</a>
                    </li>
                                            <li class="hidden-xs ">
                            <a href="https://tool.lu/c/developer/">开发类</a>
                        </li>
                                            <li class="hidden-xs ">
                            <a href="https://tool.lu/c/webmaster/">站长类</a>
                        </li>
                                            <li class="hidden-xs ">
                            <a href="https://tool.lu/c/geek/">极客类</a>
                        </li>
                                            <li class="hidden-xs ">
                            <a href="https://tool.lu/c/academic/">学术类</a>
                        </li>
                                            <li class="hidden-xs  active">
                            <a href="https://tool.lu/c/other/">其它</a>
                        </li>
                                        <li><a href="https://tool.lu/article/"
                            style="color: #8E44AD;">码农文库</a></li>
                    <li><a href="https://tool.lu/tip/"
                            style="color: #E74C3C;">奇淫巧技</a></li>
                    <li class=""><a href="https://tool.lu/software/"
                             style="color: #F39C12;">软件推荐</a></li>
                    <!-- <li class=""><a href="https://tool.lu/mall/" style="color: #3498DB;">好物</a></li> -->
                    <li class=""><a href="https://tool.lu/nav/"
                             style="">网址导航</a></li>
                    <li class=" hidden-xs"><a href="https://tool.lu/wiki/">Wiki</a>
                    </li>
                                    </ul>
                <a rel="noreferrer noopener" target="_blank"
                   href="https://tool.lu/user/messages/2" class="feedback">反馈</a>
            </div>
        </div>
    </div>
    <div id="btm-nav" class="visible-xs visible-sm">
        <div class="opts">
            <a class="opts-group active" href="https://tool.lu/">
                <i class="fa fa-home"></i>
                <span>工具首页</span>
            </a>
            <a class="opts-group "
               href="https://tool.lu/article/">
                <i class="fa fa-pen"></i>
                <span>码农文库</span>
            </a>
            <a class="opts-group "
               href="https://tool.lu/mine/">
                <i class="fa fa-user"></i>
                <span>我的</span>
            </a>
            <div class="opts-group">
                <div class="btn-group dropup">
                    <i class="fa fa-bars" data-toggle="dropdown"></i>
                    <span data-toggle="dropdown">更多</span>
                    <ul class="dropdown-menu">
                        <li>
                            <a href="https://tool.lu/tip/">奇淫巧技</a>
                        </li>
                        <li>
                            <a href="https://tool.lu/wiki/">Wiki</a>
                        </li>
                        <li>
                            <a href="https://tool.lu/software/">软件推荐</a>
                        </li>
                        <li>
                            <a href="https://tool.lu/nav/">网址导航</a>
                        </li>
                    </ul>
                </div>
            </div>
        </div>
    </div>
    <div id="bdy">
        <div class="w">
<link rel="stylesheet" href="//s2.tool.lu/__/ae909e055caee9d9feb6f90f1361a321.css">
<div class="main app_ip">
    <div class="inner">
        <form id="main_form" action="https://tool.lu/ip/ajax.html"
              method="POST">
            <p class="clearfix">
                <label style="float: left;">请输入IP或网站域名：</label>
                <input type="text" name="ip" class="text" style="float: left; margin-right: 15px;"
                       value="112.8.242.127"/>
                <button type="submit">查询</button>
            </p>
            <div class="notice" style="margin-right: 40px;">
                <table class="tbl" width="100%" cellspacing="0" cellpadding="0">
                    <tbody>
                    <tr>
                        <th width="30%">内网IP 地址：</th>
                        <td id="inner_ip">-</td>
                    </tr>
                    <tr>
                        <th width="30%">IP 地址：</th>
                        <td id="ipaddress">127.0.0.1</td>
                    </tr>
                    <tr>
                        <th width="30%">IP Long：</th>
                        <td id="iplong">0</td>
                    </tr>
                    <tr>
                        <th width="30%">归属地(纯真数据)：</th>
                        <td id="chunzhen_address">本机地址</td>
                    </tr>
                    <tr>
                        <th width="30%">归属地(ipip)：</th>
                        <td id="ipip_address">本机地址</td>
                    </tr>
                    <tr>
                        <th width="30%">归属地(淘宝数据)：</th>
                        <td id="taobao_address">本机地址</td>
                    </tr>
                    <tr>
                        <th width="30%">归属地(IP2REGION)：</th>
                        <td id="ip2region_address">本机地址</td>
                    </tr>
                    </tbody>
                </table>
            </div>
        </form>
        <h3>IP签名图</h3>
        <div class="pic_item clearfix">
            <img width="300" height="126" src="https://tool.lu/netcard/"/>
            <div>
                <label for="htmlcode">html代码：</label><br/>
                <input id="htmlcode" type="text" class="text"
                       value="&lt;img src=&quot;https://tool.lu/netcard/&quot;&gt;"/>
            </div>
            <div>
                <label for="ubbcode">ubb代码：</label><br/>
                <input id="ubbcode" type="text" class="text"
                       value="[img]https://tool.lu/netcard/[/img]"/>
            </div>
            <div>
                <label for="mdcode">markdown代码：</label><br/>
                <input id="mdcode" type="text" class="text"
                       value="![IP签名](https://tool.lu/netcard/)"/>
            </div>
            <div>
                <label for="mdcode">asciidoc代码：</label><br/>
                <input id="mdcode" type="text" class="text"
                       value="image:https://tool.lu/netcard/[IP签名]"/>
            </div>
        </div>
        <h3>命令行获取外网IP</h3>
        <div>
            <pre><code>curl -L tool.lu/ip</code></pre>
        </div>
        <p>😉  阿里云幸运红包，<a target="_blank" rel="noopener noreferrer" class="js-track" href="https://promotion.aliyun.com/ntms/yunparter/invite.html?userCode=c9eztko9">戳我领取</a>！</p><iframe id="comment_frame" frameborder="0" scrolling="no" style="width:100%; height: 350px;" src="https://tool.lu/comment/b/e" loading="lazy"></iframe><script src="//s3.tool.lu/js/comment/top.js"></script>    </div>
</div>
<script src="//s4.tool.lu/__/ded71b846304789d8a78a2bc406ba04a.js"></script>
<div class="aside hidden-xs hidden-sm">
            <div class="inner mgb10">
            <input id="shorturl" type="text" readonly value="https://tool.lu/ip/"
                   class="copy-btn" data-clipboard-text="https://tool.lu/ip/">
        </div>
        <div class="inner mgb10">
            <!-- 生成 url 的二维码 -->
            <div class="js-share-bar">
                <a class="js-share-weixin" href="javascript:;" data-url="https://tool.lu/qrcode/basic.html?text=https%3A%2F%2Ftool.lu%2Fip%2F&size=140"><i
                            class="fab fa-weixin"></i></a>
                <a class="js-share-weibo" href="javascript:;" data-url="http://service.weibo.com/share/share.php?url=https%3A%2F%2Ftool.lu%2Fip%2F&title=IP%E5%9C%B0%E5%9D%80%E6%9F%A5%E8%AF%A2&appkey=576866350&searchPic=true"><i
                            class="fab fa-weibo"></i></a>
                <a class="js-share-qq" href="javascript:;" data-url="http://connect.qq.com/widget/shareqq/index.html?url=https%3A%2F%2Ftool.lu%2Fip%2F&title=IP%E5%9C%B0%E5%9D%80%E6%9F%A5%E8%AF%A2&desc=IP%E5%9C%B0%E5%9D%80%E6%9F%A5%E8%AF%A2&summary=&pics=&flash=&site=%E5%9C%A8%E7%BA%BF%E5%B7%A5%E5%85%B7"><i
                            class="fab fa-qq"></i></a>
            </div>
            <div>据说喜欢分享的,后来都成了大神</div>
        </div>
                <div class="inner mgb10">
            <div class="panel">
                <div class="panel-heading">
                                            <a title="收藏" class="collectable collect" href="javascript:;"
                           data-slug="ip" data-url="https://tool.lu/collect.html"><i
                                    class="icon collect"></i><var>加入收藏</var></a>
                                    </div>
                <div class="panel-body">
                                            <span class="author-info"><a target="_blank"
                                                     href="http://type.so/author/3/">xiaozi</a> <a
                                    target="_blank"
                                    style="color: #e67e22;"
                                    href="https://tool.lu/user/messages/2"><i
                                        class="far fa-envelope"></i></a> <i class="medal-zz"></i></span><br>
                                    </div>
            </div>
        </div>
        <div class="inner">
        <div class="ad-container swiper-container">
            <div class="swiper-wrapper">
                <div class="swiper-slide"><a target="_blank" rel="noopener noreferrer" class="js-track"
                                             title="阿里云产品限量2000红包"
                                             href="https://www.aliyun.com/minisite/goods?userCode=c9eztko9"><img
                                width="278" height="149" alt="阿里云产品限量2000红包"
                                src="//qn14.tool.lu/202003/05/165505IXjxAQPEgLoFM4Na_280x150.png"
                                srcset="//qn11.tool.lu/202003/05/165503u6RFaLixr5wVv7Ti_560x300.png 2x"></a></div>
                <!-- <div class="swiper-slide"><a target="_blank" rel="noopener noreferrer" class="js-track" title="vultr"
                                             href="https://www.vultr.com/?ref=6887961"><img width="278" height="149"
                                                                                            alt="vultr"
                                                                                            src="//qn12.tool.lu/202003/05/165543QAa8BDZRasYR52N1_280x150.png"
                                                                                            srcset="//qn11.tool.lu/202003/05/165603zhQOPMj6OsI1bQlH_560x300.png 2x"></a>
                </div> -->
                <div class="swiper-slide"><a target="_blank" rel="noopener noreferrer" class="js-track" title="vultr activity"
                                             href="https://www.vultr.com/?ref=8505945-6G"><img width="278" height="149"
                                                                                            alt="vultr activity"
                                                                                            src="//qn11.tool.lu/202003/23/121504EJ0nby64UxJDoTOZ_280x150.png"
                                                                                            srcset="//qn12.tool.lu/202003/23/121504Dzecp6vk8B6gQars_560x300.png 2x"></a>
                </div>
                <div class="swiper-slide"><a target="_blank" rel="noopener noreferrer" class="js-track" title="七牛云" href="https://portal.qiniu.com/signup?code=1hcnwqbyzvdjm"><img width="278" height="149" alt="qiniu" src="//qn12.tool.lu/202003/05/170440Qumu8ygODbc2VhHS_280x150.png" srcset="//qn14.tool.lu/202003/05/170443d6xidQiuAgBokYoT_560x300.png 2x"></a></div>
                <div class="swiper-slide"><a target="_blank" rel="noopener noreferrer" class="js-track" title="腾讯云"
                                             href="https://cloud.tencent.com/redirect.php?redirect=1025&cps_key=bdf94f38b1db499e7fff3c66aa6753d6&from=console"><img
                                width="278" height="149" alt="tencent"
                                src="//qn14.tool.lu/202003/05/1656399ip4cR5R7yHU5oQc_280x150.png"
                                srcset="//qn11.tool.lu/202003/05/165557zMuqbqfQkSXO25Re_560x300.png 2x"></a></div>
                <div class="swiper-slide"><a target="_blank" rel="noopener noreferrer" class="js-track" title="virmach" href="https://billing.virmach.com/aff.php?aff=7437"><img width="278" height="149" alt="virmach" src="//qn13.tool.lu/202003/06/1216058ObomJXy4cSW7z8c_280x150.png" srcset="//qn12.tool.lu/202003/06/121607DWYH2i81V1rxWAHS_560x300.png 2x"></a></div>
            </div>
            <div class="swiper-pagination"></div>
        </div>
    </div>
    <div class="note">
        <a class="note-add-btn" target="_blank" href="https://tool.lu/sentence/add"><i class="fa fa-plus"></i>
            提交句子</a>
        <div class="note-container">
            生活就像海绵里的水，只要你不愿意挤，总有一天会蒸发完的。        </div>
        <div class="note-bottom"></div>
    </div>
    <div class="inner">
        <div class="panel">
        <div class="panel-heading">码农文库<a class="fr" target="_blank" href="https://tool.lu/article/">更多</a></div>
            <div class="panel-body">
                <ul class="box-list">
                                            <li>
                            <a target="_blank" title="干货来袭！淘宝设计师谈动效的高效设计与交付" href="https://tool.lu/article/1Zf/detail">
                                <span class="top-num">1</span>
                                <span class="title">干货来袭！淘宝设计师谈动效的高效设计与交付</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="阿里高管的思考方式比一般人强在哪？（内部员工7000字深度干货）" href="https://tool.lu/article/1Ze/detail">
                                <span class="top-num">2</span>
                                <span class="title">阿里高管的思考方式比一般人强在哪？（内部员工7000字深度干货）</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="十种图像模糊算法的总结与实现" href="https://tool.lu/article/1Zd/detail">
                                <span class="top-num">3</span>
                                <span class="title">十种图像模糊算法的总结与实现</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="达达集团实时计算任务SQL化实践" href="https://tool.lu/article/1Zc/detail">
                                <span class="top-num">4</span>
                                <span class="title">达达集团实时计算任务SQL化实践</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="数据库连接配置策略和实践" href="https://tool.lu/article/1Zb/detail">
                                <span class="top-num">5</span>
                                <span class="title">数据库连接配置策略和实践</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="第02期：MySQL 数据类型的艺术 - 大对象字段" href="https://tool.lu/article/1Za/detail">
                                <span class="top-num">6</span>
                                <span class="title">第02期：MySQL 数据类型的艺术 - 大对象字段</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="优酷“互动”剧的核心体验设计" href="https://tool.lu/article/1Z9/detail">
                                <span class="top-num">7</span>
                                <span class="title">优酷“互动”剧的核心体验设计</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="Linux select 一网打尽" href="https://tool.lu/article/1Z8/detail">
                                <span class="top-num">8</span>
                                <span class="title">Linux select 一网打尽</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="为什么越来越多设计师不想做设计？" href="https://tool.lu/article/1Z7/detail">
                                <span class="top-num">9</span>
                                <span class="title">为什么越来越多设计师不想做设计？</span>
                            </a>
                        </li>
                                            <li>
                            <a target="_blank" title="倒排索引/全文搜索基本原理 - 郭大侠1 - 博客园" href="https://tool.lu/article/1Z6/detail">
                                <span class="top-num">10</span>
                                <span class="title">倒排索引/全文搜索基本原理 - 郭大侠1 - 博客园</span>
                            </a>
                        </li>
                                    </ul>
            </div>
        </div>
    </div>
</div>
</div>
</div>
<div id="ftr">
    <div class="w">
        <p class="declare">
            <a target="_blank" href="mailto:245565986@qq.com">联系我</a> - <a rel="nofollow" href="https://tool.lu/">工具首页</a>
            <br>
            Copyright &copy; 2011-2020 <a rel="noopener noreferrer" href="http://type.so/" target="_blank">iteam</a>.
            Current version is <a style="color: #009a61;" href="https://tool.lu/changelog.html">2.64.2</a>.
            UTC+08:00, 2020-04-16 19:23<br>
            <a rel="noopener noreferrer" target="_blank" href="http://www.beian.miit.gov.cn/">浙ICP备14020137号</a>
            <a href="https://tool.lu/visitor/">$访客地图$</a>
        </p>
    </div>
</div>
</div>
<div id="t_fixed">
    <a rel="noopener noreferrer" target="_blank" class="btn qq-qun copy-btn js-tip"
        data-clipboard-text="227310278"
       href="http://shang.qq.com/wpa/qunwpa?idkey=556567ae633f9715b4849f7277bbc37d0f657eab2a6366b1b4b528cece2cbe74"
       title="QQ群(二): 227310278"><i class="fab fa-qq"></i></a>
    <a rel="noopener noreferrer" target="_blank" class="btn weibo js-tip" href="http://weibo.com/245565986" title="微博: 小子欠扁"><i class="fab fa-weibo"></i></a>
    <a rel="noopener noreferrer" target="_blank" class="btn github js-tip" href="https://github.com/xiaozi" title="github: xiaozi"><i class="fab fa-github-alt"></i></a>
    <a class="btn gotop js-tip" href="javascript:;" title="返回顶部"><i class="fas fa-arrow-up"></i></a>
</div>
<script>
    (function () {
        var a = document.createElement("script");
        a.async = !0;
        a.src = ("https:" == document.location.protocol ? "https:" : "http:") + "//analytics.tool.lu/ta.js";
        var b = document.getElementsByTagName("script")[0];
        b.parentNode.insertBefore(a, b)
    })();
</script>
<script src="//s1.tool.lu/__/e7dc1f2927665000350c2b5c1177e821.js"></script>
<script src="//s2.tool.lu/__/044f9466a86bcd5f9abef2dbd779218e.js"></script>
</body>
</html>

MyLight1109

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
[Python]自学笔记36：论一只爬虫的自我修养3：隐藏

有一些网站比较讨厌爬虫，它们不想被爬虫访问，所以它们会检查链接的来源，若不是正常途径，那么就会被屏蔽。这就需要将爬虫隐藏有下面几种方法一.修改headers网站一般检查链接的headers里面的USER-Agent，如果不加以修改，爬虫的USER-Agent就会是Python的版本号，所以我们需要自己修改。我们需要利用利用urllib.request.Request(url,data,h...
复制链接

扫一扫

专栏目录