python3爬虫使用代理爬取页面noscript标签问题

最新推荐文章于 2024-07-12 16:42:46 发布

xzlAwin

最新推荐文章于 2024-07-12 16:42:46 发布

阅读量1k

点赞数 1

分类专栏： python 文章标签：爬虫 python noscript python爬虫

本文链接：https://blog.csdn.net/xzlAwin/article/details/124967058

版权

python 专栏收录该内容

13 篇文章 0 订阅

订阅专栏

python3爬虫使用代理爬取页面noscript标签问题

操作

运行python爬虫，使用代理爬取网页

from urllib.error import URLError  
from urllib.request import ProxyHandler, build_opener  

url = 'https://www.baidu.com'
proxy_handler = ProxyHandler({  
  'http': 'http://127.0.0.1:8888',
  'https': 'https://127.0.0.1:8888'  
})  
opener = build_opener(proxy_handler)
try:  
  response = opener.open(url)
  data = response.read().decode('utf-8')
  # print(data)
  fhandle = open("./baidu.html","wb")
  fhandle.write(data.encode('utf-8'))
  fhandle.close()
except URLError as e:
  print(e.reason)

代理爬取页面

<html>
<head>
        <script>
                location.replace(location.href.replace("https://","http://"));
        </script>
</head>
<body>
        <noscript><meta http-equiv="refresh" content="0;url=http://www.baidu.com/"></noscript>
</body>
</html>

分析原因

爬取百度页面，如果百度不识别浏览器，就会使用location.replace()方法使页面重定向
noscript标签，表示无法识别的浏览器

解决方法

http协议的headers头，添加浏览器标识即可

from urllib.error import URLError  
from urllib.request import ProxyHandler, build_opener  

url = 'https://www.baidu.com'
headers = (
	{
		"User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36"
	}
)
proxy_handler = ProxyHandler({  
  'http': 'http://127.0.0.1:8888',
  'https': 'https://127.0.0.1:8888'  
})  
opener = build_opener(proxy_handler)
opener.addheaders = [headers]
try:  
  response = opener.open(url)
  data = response.read().decode('utf-8')
  # print(data)
  fhandle = open("./baidu.html","wb")
  fhandle.write(data.encode('utf-8'))
  fhandle.close()
except URLError as e:
  print(e.reason)

xzlAwin

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
python3爬虫使用代理爬取页面noscript标签问题

python3爬虫使用代理爬取页面noscript标签问题操作运行python爬虫，使用代理爬取网页from urllib.error import URLError from urllib.request import ProxyHandler, build_opener url = 'https://www.baidu.com'proxy_handler = ProxyHandler({ 'http': 'http://127.0.0.1:8888', 'https':
复制链接

扫一扫