User-agent反爬

最新推荐文章于 2024-09-16 22:23:03 发布

以梦为马越骑越傻

最新推荐文章于 2024-09-16 22:23:03 发布

阅读量227

点赞数

分类专栏：反爬

本文链接：https://blog.csdn.net/weixin_42670402/article/details/110840786

版权

反爬专栏收录该内容

11 篇文章 0 订阅

订阅专栏

该博客介绍了如何在Python中实现反爬机制，特别是针对校园新闻网列表页的User-Agent反爬。通过使用`fake_useragent`库生成随机User-Agent，避免被网站识别为爬虫。代码展示了如何结合`requests`库发送带User-Agent的HTTP请求，并利用`Scrapy`的`Selector`解析响应内容，提取页面数据。

摘要由CSDN通过智能技术生成

'''
校园新闻网列表页User-agent反爬
'''
from common.tools import Tool
import requests
from scrapy.selector import Selector
class User_Agent:
    def __init__(self):
        self.tool = Tool.get_tool()

    def run(self):
        url = 'http://www.porters.vip/verify/uas/index.html'
        headers = {}
        headers = self.tool.get_user_agent(headers)
        r = requests.get(url=url,headers=headers)
        if r.status_code == 200:
            titles = Selector(r).xpath("//ul[@class='list-group']/li/text()").extract()
            for title in titles:
                print(title)

if __name__ == '__main__':
    User_Agent().run()

Tools：

import os

from PIL import Image
import io
import pytesseract
import requests
from fake_useragent import UserAgent
class Tool:

    @classmethod
    def get_tool(cls):
        return cls()

    def get_user_agent(self,headers):
        ua = UserAgent()
        headers['User-Agent'] = ua.random
        return headers
    def get_cookie(self,cookie_str):
        c_b = cookie_str.split(',')
        cookies = {}
        for cookie in c_b:
            k,v = cookie.split('=')
            cookies[k] = v
        return cookies
    def tessseract_img(self,img_url):
        # 识别图片文字数字
        img_body = requests.get(img_url).content
        image_stream = Image.open(io.BytesIO(img_body))

        image_stream = image_stream.convert('L')
        if os.path.exists('a.jpg'):
            os.remove('a.jpg')
        image_stream.save('a.jpg')
        Image.open(r'a.jpg')
        return pytesseract.image_to_string(image_stream,lang='eng')

----参考python3反爬虫原理与绕过实战