一、报错形式
记录一下,今天继续做我的毕设,爬crt.sh站记录时,发现我的IP被封掉了……
requests.get('https://crt.sh/?q=' + domain, headers=headers,verify=False)
我得到了这样的结果,requests请求返回如下:
This user agent has been blocked due to abuse. Can we interest you direct access to the crt.sh DB to fetch the data you want? https://groups.google.com/g/crtsh/c/sUmV0mBz8bQ/m/K-6Vymd_AAAJ
以及手动请求:
二、报错原因
爬虫导致IP被封
解决方案
1.PPPOE路由器重置,自动更换本机公网IP(直接)
def public(): # 查看本机公网地址
with urlopen(r'http://ip.42.pl/raw') as fd:
f = fd.read().decode()
print(f)
if __name__ == '__main__':
public()
以上代码可直接查到本机的公网IP地址。
2.伪造User-Agent
1.代码
import random
import socket
import struct
from fake_useragent import UserAgent # pip install fake-useragent
HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'User-Agent': "", # 构造用户代理
'Referer': 'https://www.google.com',
'X-Forwarded-For': "", # 构造
'X-Real-IP': "", # 构造
'Connection': 'keep-alive',
}
def get_ua():
ua = UserAgent()
ip = socket.inet_ntoa(struct.pack('>I', random.randint(1, 0xffffffff)))
HEADERS["User-Agent"] = ua.random
HEADERS["X-Forwarded-For"] = HEADERS["X-Real-IP"] = ip
pyHEADERS = [
'User-Agent: {}'.format(ua.random),
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Referer: https://www.google.com', 'X-Forwarded-For: {}'.format(ip),
'X-Real-IP: {}'.format(ip), 'Connection: close'
]
return HEADERS
if __name__ == "__main__":
print(get_ua())
返回结果:
可得:四次请求构造出不同请求头部。
2.User-Agent介绍
User-Agent又称UA,是用户代理。使得服务器能够识别客户使用的操作系统及版本、CPU类型、浏览器及版本、浏览器渲染引擎、浏览器语言、浏览器插件等等。
常见用户代理:
user_agents = ['Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1','Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50','Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11']