爬虫dns cache设置

版权声明:本文为博主原创文章,遵循 CC 4.0 by-sa 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/nimade511/article/details/79502869

设置代理服务器squid的dns

  • positive_dns_ttl 6 hours #设置为6小时,这个值取dns返回的ttl最小值
  • dns_nameservers 8.8.8.8 #设置google dns服务器

设置nscd

  • enable-cache hosts yes #和squid设置差不多

python requests设置

from urllib3.util import connection

    _orig_create_connection = connection.create_connection

    def patched_create_connection(address, *args, **kwargs):
        """Wrap urllib3's create_connection to resolve the name elsewhere"""
        # resolve hostname to an ip address; use your own
        # resolver here, as otherwise the system resolver will be used.
        host, port = address
        hostname = your_dns_resolver(host)

        return _orig_create_connection((hostname, port), *args, **kwargs)

    connection.create_connection = patched_create_connection

这里是stackoverflow回答

展开阅读全文

没有更多推荐了,返回首页