restful framework 访问频率源码分析

一.简介

承接上篇文章Django Rest Framework权限源码剖析，当服务的接口被频繁调用，导致资源紧张怎么办呢？当然或许有很多解决办法，比如：负载均衡、提高服务器配置、通过代理限制访问频率等，但是django rest framework自身就提供了访问频率的控制，可以从代码本身做控制。

二.频率控制内部原理概述

django rest framework 中频率控制基本原理基于访问次数和时间，通过计算实现，当然我们也可以自己定义频率控制方法。基本原理如下：

启用频率，DRF内部会有一个字典记录来访者的IP，以及访问时间最近几（通过配置）次的访问时间，这样确保每次列表中最后一个元素都是该用户请求的最早时间，形式如下：

{
IP1:[第三次请求时间，第二次请求时间,第一次请求时间,],
IP2:[第二次请求时间,第一次请求时间,],
.....
}

举例说明，比如我现在配置了5秒内只能访问2次，每次请求到达频率控制时候先判断请求者IP是否已经在这个请求字典中，若存在，在判断用户请求5秒内的请求次数，若次数小于等于2，则允许请求，若大于2，则超过频率，不允许请求。

关于请求频率的的算法（以5秒内最多访问两次为例）：
1.首先删除掉列表里5秒之前的请求，循环判断当前请求时间和最早请求时间之差记作t1，若t1大于5则代表列表中最早的请求已经在5秒外了，删除掉，继续判断倒数第二个请求，直到t1小于5.

2.当确保请求列表中只有5秒内请求时候，接着判断其请求次数(列表长度)，若长度大于2，则证明超过5秒内访问超过2次了，则不允许，否则，通过并将此次访问时间插入到列表最前面，作为最新访问时间。

三.频率控制源码分析

1.在前面几篇文章中已经分析了DRF的认证、权限源码，频率控制也一样也从APIView的dispatch方法说起：

def dispatch(self, request, *args, **kwargs):
        """
        `.dispatch()` is pretty much the same as Django's regular dispatch,
        but with extra hooks for startup, finalize, and exception handling.
        """
        self.args = args
        self.kwargs = kwargs
        #对原始request进行加工，丰富了一些功能
        #Request(
        #     request,
        #     parsers=self.get_parsers(),
        #     authenticators=self.get_authenticators(),
        #     negotiator=self.get_content_negotiator(),
        #     parser_context=parser_context
        # )
        #request(原始request,[BasicAuthentications对象，])
        #获取原生request,request._request
        #获取认证类的对象，request.authticators
        #1.封装request
        request = self.initialize_request(request, *args, **kwargs)
        self.request = request
        self.headers = self.default_response_headers  # deprecate?

        try:
            self.initial(request, *args, **kwargs)

            # Get the appropriate handler method
            if request.method.lower() in self.http_method_names:
                handler = getattr(self, request.method.lower(),
                                  self.http_method_not_allowed)
            else:
                handler = self.http_method_not_allowed

            response = handler(request, *args, **kwargs)

        except Exception as exc:
            response = self.handle_exception(exc)

        self.response = self.finalize_response(request, response, *args, **kwargs)
        return self.response

2.执行inital方法，initial方法中执行check_throttles则开始频率控制

def initial(self, request, *args, **kwargs):
        """
        Runs anything that needs to occur prior to calling the method handler.
        """
        self.format_kwarg = self.get_format_suffix(**kwargs)

        # Perform content negotiation and store the accepted info on the request
        neg = self.perform_content_negotiation(request)
        request.accepted_renderer, request.accepted_media_type = neg

        # Determine the API version, if versioning is in use.
        version, scheme = self.determine_version(request, *args, **kwargs)
        request.version, request.versioning_scheme = version, scheme

        # Ensure that the incoming request is permitted
        #2.实现认证
        self.perform_authentication(request)
        #3.权限判断
        self.check_permissions(request)
        #4.频率限制
        self.check_throttles(request)

3.下面是check_throttles源码，与认证、权限一样采用列表对象方式，通过判断allow_request方法返回值判断频率是否通过

def check_throttles(self, request):
        """
        Check if request should be throttled.
        Raises an appropriate exception if the request is throttled.
        """
        for throttle in self.get_throttles(): #循环频率控制类结果
            if not throttle.allow_request(request, self): #判断其中的allow_request返回结果，true则频率通过，否则返回等待多少秒可以访问
                self.throttled(request, throttle.wait())

4.get_throttles方法，采用列表生成式生成频率控制对象，与认证、权限一样

 def get_throttles(self):
        """
        Instantiates and returns the list of throttles that this view uses.
        """
        return [throttle() for throttle in self.throttle_classes]  #这里面的self.throttle_classes有两种情况1.局部使用频率控制类 在函数前面定义,2.全局使用频率控制类 在settings里面配置

5.self.throttle_classes属性获取

class APIView(View):

    # The following policies may be set at either globally, or per-view.
    renderer_classes = api_settings.DEFAULT_RENDERER_CLASSES
    parser_classes = api_settings.DEFAULT_PARSER_CLASSES
    authentication_classes = api_settings.DEFAULT_AUTHENTICATION_CLASSES
    throttle_classes = api_settings.DEFAULT_THROTTLE_CLASSES     #频率控制全局配置
    permission_classes = api_settings.DEFAULT_PERMISSION_CLASSES
    content_negotiation_class = api_settings.DEFAULT_CONTENT_NEGOTIATION_CLASS
    metadata_class = api_settings.DEFAULT_METADATA_CLASS
    versioning_class = api_settings.DEFAULT_VERSIONING_CLASS

6.通过以上分析，知道了频率控制是通过判断每个类中的allow_request放法的返回值来判断频率是否通过，下面来看看我们所使用的SimpleRateThrottle怎么实现的,分析部分请看注解：
SimpleRateThrottle类源码：

class SimpleRateThrottle(BaseThrottle):
    """
    A simple cache implementation, that only requires `.get_cache_key()`
    to be overridden.

    The rate (requests / seconds) is set by a `throttle` attribute on the View
    class.  The attribute is a string of the form 'number_of_requests/period'.

    Period should be one of: ('s', 'sec', 'm', 'min', 'h', 'hour', 'd', 'day')

    Previous request information used for throttling is stored in the cache.
    """
    cache = default_cache # 存放请求时间，类似与示例中的大字典，这里使用的是django的缓存
    timer = time.time
    cache_format = 'throttle_%(scope)s_%(ident)s'
    scope = None
    THROTTLE_RATES = api_settings.DEFAULT_THROTTLE_RATES  

    def __init__(self):
        if not getattr(self, 'rate', None): #对象初始化的时候获取rate, 如果获取不到 执行get_rate()
            self.rate = self.get_rate()  #'5/m'
        self.num_requests, self.duration = self.parse_rate(self.rate)
   # 获取请求的key标识，必须要有否则会报错，这里可以重写，使用用户的用户名、或其他作为key，在示例中使用的get_ident方法用户获取用户IP作为key
    def get_cache_key(self, request, view):
        """
        Should return a unique cache-key which can be used for throttling.
        Must be overridden.

        May return `None` if the request should not be throttled.
        """
        raise NotImplementedError('.get_cache_key() must be overridden')

    def get_rate(self):
        """
        Determine the string representation of the allowed request rate.
        """
        if not getattr(self, 'scope', None):  #要获取rate首先要获取Scope scope是必须要的
            msg = ("You must set either `.scope` or `.rate` for '%s' throttle" %
                   self.__class__.__name__)
            raise ImproperlyConfigured(msg)

        try:
            return self.THROTTLE_RATES[self.scope]
        except KeyError:
            msg = "No default throttle rate set for '%s' scope" % self.scope
            raise ImproperlyConfigured(msg)

    def parse_rate(self, rate):#格式化速率
        """
        Given the request rate string, return a two tuple of:
        <allowed number of requests>, <period of time in seconds>
        """
        if rate is None:
            return (None, None)
        num, period = rate.split('/') #'3/m'  num=3  period=m
        num_requests = int(num)  #3
        duration = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400}[period[0]]  #60
        return (num_requests, duration)

    def allow_request(self, request, view):
        """
        Implement the check to see if the request should be throttled.

        On success calls `throttle_success`.
        On failure calls `throttle_failure`.
        """
        if self.rate is None:  #
            return True

        self.key = self.get_cache_key(request, view)  #username
        if self.key is None:
            return True

        self.history = self.cache.get(self.key, [])
        self.now = self.timer()

        # Drop any requests from the history which have now passed the
        # throttle duration
        while self.history and self.history[-1] <= self.now - self.duration:
            self.history.pop()  #将不在duration范围的时间去掉
        if len(self.history) >= self.num_requests: #如果访问次数大于num_requests 访问不被允许
            return self.throttle_failure()
        return self.throttle_success()

    def throttle_success(self): #频率通过 返回True
        """
        Inserts the current request's timestamp along with the key
        into the cache.
        """
        self.history.insert(0, self.now)
        self.cache.set(self.key, self.history, self.duration)
        return True

    def throttle_failure(self):#不通过返回false
        """
        Called when a request to the API has failed due to throttling.
        """
        return False

    def wait(self): # 返回等待时间
        """
        Returns the recommended next request time in seconds.
        """
        if self.history:
            remaining_duration = self.duration - (self.now - self.history[-1])
        else:
            remaining_duration = self.duration

        available_requests = self.num_requests - len(self.history) + 1
        if available_requests <= 0:
            return None

        return remaining_duration / float(available_requests)

get_ident方法源码，该方法用于获取请求的IP：

def get_ident(self, request):
        """
        Identify the machine making the request by parsing HTTP_X_FORWARDED_FOR
        if present and number of proxies is > 0. If not use all of
        HTTP_X_FORWARDED_FOR if it is available, if not use REMOTE_ADDR.
        """
        xff = request.META.get('HTTP_X_FORWARDED_FOR')
        remote_addr = request.META.get('REMOTE_ADDR')
        #这里request是封装以后的requst，django原生的是request._request.META 这样也可以获取
        num_proxies = api_settings.NUM_PROXIES

        if num_proxies is not None:
            if num_proxies == 0 or xff is None:
                return remote_addr
            addrs = xff.split(',')
            client_addr = addrs[-min(num_proxies, len(addrs))]
            return client_addr.strip()

        return ''.join(xff.split()) if xff else remote_addr

四.自定义频率控制类

自定义频率控制无非实现request_allow方法和wait方法，你可以根据实际需求来定制你的频率控制，
urls.py

 url(r'^api/v1/manage/$', views.ManageView.as_view()),

views.py

class ManageView(views.APIView):
    #authentication_classes = [MyAuthentication,]
    #permission_classes = [Mypermision,ManagePermission]
    throttle_classes = [UserVisitThrottle]
    def get(self,request):

        return Response('管理人员访问页面')

    def throttled(self, request, wait):
        """
        定制方法设置中文错误
        :param request:
        :param wait:
        :return:
        """
        class MyThrottle(exceptions.Throttled):
            default_detail = '请求被限制'
            extra_detail_singular = 'Expected available in {wait} second.'
            extra_detail_plural = 'Expected available in {wait} seconds.'
            default_code = '还需要再等{wait}秒'

        raise MyThrottle(wait)

utils.py–>throttle.py

from rest_framework.throttling import BaseThrottle, SimpleRateThrottle
import time
request_record = {}  # 访问记录，

# 自定义频率控制 全部都要自己写
class VisitThrottle(BaseThrottle):
    scope = "WD"

    def __init__(self):
        self.history = None

    def allow_request(self, request, view):
        remote_addr = self.get_ident(request)  # 继承BaseThrottle
        self.now = self.timer()
        if remote_addr not in request_record:
            request_record[remote_addr] = [self.now, ]
            return True

        history = request_record.get(remote_addr)
        self.history = history

        while history and history[-1] < self.now - 60:
            history.pop()

        if len(history) < 3:
            history.insert(0, self.now)
            return True

    def wait(self):
        ctime = time.time()
        history = self.history[-1]
        wt = 60 - (ctime - history)
        return wt


# 对匿名用户自定义频率控制 针对的是IP
class AnVisitThrottle(SimpleRateThrottle):
    scope = "Thro_anon"

    def get_cache_key(self, request, view):
        if request.user:
            return None
        return self.get_ident(request)


# 对登录用户自定义频率控制 针对的是用户名
class UserVisitThrottle(SimpleRateThrottle):
     scope="Thro_user"

     def get_cache_key(self, request, view):
         if request.user:
              return request.auth
         return None
注意：如果我们的自定义类继承BaseThrottle，allow_request,wait方法都要自己写，如果我们自定义的类继承SimpleRateThrottle，get_cache_key()需要自己写，allow_request()可以继承父类

五.配置自定义认证类

1.局部配置

 throttle_classes = [AnVisitThrottle]

2.全局配置

REST_FRAMEWORK = {
    'DEFAULT_THROTTLE_CLASSES': ['app01.utils.auth.AnVisitThrottle',],
    'DEFAULT_THROTTLE_RATES': {
        'Thro_anon': '5/m',
        'Thro_user':'10/m',
    }
}

六.案例

对匿名用户进行限制，每个用户1分钟允许访问5次，对于登录的普通用户1分钟访问10次，VIP用户一分钟访问20次

比如首页可以匿名访问
#先认证，只有认证了才知道是不是匿名的，
#权限登录成功之后才能访问， ，index页面就不需要权限了
If request.user #判断登录了没有
urls.py

  url(r'^api/v1/index/$', views.IndexView.as_view()),

views.py

#首页支持匿名访问,
#无需要登录就可以访问
class IndexView(views.APIView):

    #authentication_classes = [MyAuthentication]
    #permission_classes = []
    throttle_classes = [AnVisitThrottle,UserVisitThrottle,VipuserVisitThrottle]
    def get(self,request):
        return Response('访问首页')

    def throttled(self, request, wait):
        """
        定制方法设置中文错误
        :param request:
        :param wait:
        :return:
        """
        class MyThrottle(exceptions.Throttled):
            default_detail = '请求被限制'
            extra_detail_singular = 'Expected available in {wait} second.'
            extra_detail_plural = 'Expected available in {wait} seconds.'
            default_code = '还需要再等{wait}秒'
        raise MyThrottle(wait)

utils.py–>throttle.py

# 对匿名用户自定义频率控制 针对的是IP
class AnVisitThrottle(SimpleRateThrottle):
    scope = "Thro_anon"

    def get_cache_key(self, request, view):
        if request.user:
            return None
        return self.get_ident(request)


# 对登录用户自定义频率控制 针对的是用户名
class UserVisitThrottle(SimpleRateThrottle):
     scope="Thro_user"

     def get_cache_key(self, request, view):
         if request.user:
              return request.auth
         return None
class VipuserVisitThrottle(SimpleRateThrottle):
    scope = "Throvip_user"

    def get_cache_key(self, request, view):
        if request.user.user_type==3:
            return request.auth
        return None

settings.py

REST_FRAMEWORK = {
    
    'DEFAULT_THROTTLE_CLASSES': [],
    'DEFAULT_THROTTLE_RATES': {
        'Thro_anon': '5/m',
        'Thro_user':'10/m',
        'Throvip_user':'20/m',
    }
}

六.总结

1.认证：就是检查用户是否存在；如果存在返回(request.user,request.auth)；不存在request.user/request.auth=None
2.权限：进行职责的划分
3.限制访问频率

认证
    - 类：authenticate/authenticate_header ##验证不成功的时候执行的
    - 返回值：
        - return None,
        - return (user,auth),
        - raise 异常
    - 配置：
        - 视图：
            class IndexView(APIView):
                authentication_classes = [MyAuthentication,]
        - 全局：
            REST_FRAMEWORK = {
                    'UNAUTHENTICATED_USER': None,
                    'UNAUTHENTICATED_TOKEN': None,
                    "DEFAULT_AUTHENTICATION_CLASSES": [
                        # "app02.utils.MyAuthentication",
                    ],
            }

权限 
    - 类：has_permission/has_object_permission
    - 返回值： 
        - True、#有权限
        - False、#无权限
        - exceptions.PermissionDenied(detail="错误信息")  #异常自己随意，想抛就抛，错误信息自己指定
    - 配置：
        - 视图：
            class IndexView(APIView):
                permission_classes = [MyPermission,]
        - 全局：
            REST_FRAMEWORK = {
                    "DEFAULT_PERMISSION_CLASSES": [
                        # "app02.utils.MyAuthentication",
                    ],
            }
限流
    - 类：allow_request/wait PS: scope = "wdp_user"
    - 返回值：
　　　　　　return True、#不限制
　　　　　　return False  #限制
    - 配置： 
            - 视图： 
                class IndexView(APIView):
                    
                    throttle_classes=[AnonThrottle,UserThrottle,]
                    def get(self,request,*args,**kwargs):
                        self.dispatch
                        return Response('访问首页')
            - 全局
                REST_FRAMEWORK = {
                    "DEFAULT_THROTTLE_CLASSES":[
                    
                    ],
                    'DEFAULT_THROTTLE_RATES':{
                        'wdp_anon':'5/minute',
                        'wdp_user':'10/minute',
                    }
                }

4.优先级：局部>全局>默认