Moziila Location Service-7

自从有了pdb调试之后，跟踪过程顺利多了。目前已经弄清楚了提交位置信息的运行流程，记录如下：
1.开启服务（前面有介绍其过程）
2.发送请求：在浏览器地址栏输入http://127.0.0.1:7001/v2/geosumbit?key=mytest(注意key 是我根据邮件回复的做法自己插入数据库的)，同时在restclient的body框里填上位置信息的json字符串（可从官方文档复制），如图：

{"items": [{
    "timestamp": 1405602028568,
    "position": {
        "latitude": -22.7539192,
        "longitude": -43.4371081,
        "accuracy": 10.0,
        "age": 1000,
        "altitude": 100.0,
        "altitudeAccuracy": 50.0,
        "heading": 45.0,
        "pressure": 1013.25,
        "speed": 3.6,
        "source": "gps"
    },
    "bluetoothBeacons": [
        {
            "macAddress": "ff:23:45:67:89:ab",
            "age": 2000,
            "name": "beacon",
            "signalStrength": -110
        }
    ],
    "cellTowers": [
        {
            "radioType": "lte",
            "mobileCountryCode": 208,
            "mobileNetworkCode": 1,
            "locationAreaCode": 2,
            "cellId": 12345,
            "age": 3000,
            "asu": 31,
            "primaryScramblingCode": 5,
            "serving": 1,
            "signalStrength": -51,
            "timingAdvance": 1
        }
    ],
    "wifiAccessPoints": [
        {
            "macAddress": "01:23:45:67:89:ab",
            "age": 5000,
            "channel": 6,
            "frequency": 2437,
            "radioType": "802.11n",
            "signalToNoiseRatio": 13,
            "signalStrength": -77
        },
        {
            "macAddress": "23:45:67:89:ab:cd"
        }
    ]
}]}

3./ProgFile/ichnaea-for-liuqiao/ichnaea/lib/python2.7/site-packages/ichnaea-1.5-py2.7-linux-x86_64.egg/ichnaea/webapp/app.py：

    if _APP is None:
        #print 'ready to settrace...'
        print "_APP is none"
        conf = read_config()
        # print conf
        # pdb.set_trace()
        _APP = main(conf, ping_connections=True)
        if environ is None and start_response is None:
            print "environ is None and start_response is None"
            # Called as part of gunicorn's post_worker_init
            return _APP
    print "_APP is not none", start_response
    return _APP(environ, start_response)

开启服务后，_APP不为空，在服务没有断开之前，每次发送请求，都会直接执行：
return _APP(environ, start_response)

4._APP是Router类的一个实例，该类带有call方法，所以_APP实例可以当做函数使用：
/ProgFile/ichnaea-for-liuqiao/ichnaea/lib/python2.7/site-packages/pyramid/router.py:

    def __call__(self, environ, start_response):
        """
        Accept ``environ`` and ``start_response``; create a
        :term:`request` and route the request to a :app:`Pyramid`
        view based on introspection of :term:`view configuration`
        within the application registry; call ``start_response`` and
        return an iterable.
        """
        request = self.request_factory(environ)
        response = self.invoke_subrequest(request, use_tweens=True)
        return response(request.environ, start_response)

准备request没什么好说的就是把我们输入的地址，请求体等内容包装润色好，以备后面使用。
重点在如何处理请求。
5.response = self.invoke_subrequest(request, use_tweens=True)：

    def invoke_subrequest(self, request, use_tweens=False):
        """Obtain a response object from the Pyramid application based on
        information in the ``request`` object provided.  The ``request``
        object must be an object that implements the Pyramid request
        interface (such as a :class:`pyramid.request.Request` instance).  If
        ``use_tweens`` is ``True``, the request will be sent to the
        :term:`tween` in the tween stack closest to the request ingress.  If
        ``use_tweens`` is ``False``, the request will be sent to the main
        router handler, and no tweens will be invoked.

        See the API for pyramid.request for complete documentation
        从基于请求信息的Pyramid应用处获得一个相应对象。请求对象必须实现request接口。
        """
        registry = self.registry
        has_listeners = self.registry.has_listeners
        notify = self.registry.notify
        threadlocals = {'registry':registry, 'request':request}
        manager = self.threadlocal_manager
        manager.push(threadlocals)
        request.registry = registry
        request.invoke_subrequest = self.invoke_subrequest

        if use_tweens:
            handle_request = self.handle_request
        else:
            handle_request = self.orig_handle_request

        try:

            try:
                extensions = self.request_extensions
                if extensions is not None:
                    apply_request_extensions(request, extensions=extensions)
                #开始调用处理请求的函数
                response = handle_request(request)

                if request.response_callbacks:
                    request._process_response_callbacks(response)

                has_listeners and notify(NewResponse(request, response))

                return response

            finally:
                if request.finished_callbacks:
                    request._process_finished_callbacks()

        finally:
            manager.pop()

6.response = handle_request(request)
该步骤在前面也有解释。

    def handle_request(self, request):
        attrs = request.__dict__
        registry = attrs['registry']

        request.request_iface = IRequest
        context = None
        routes_mapper = self.routes_mapper
        debug_routematch = self.debug_routematch
        adapters = registry.adapters
        has_listeners = registry.has_listeners
        notify = registry.notify
        logger = self.logger

        has_listeners and notify(NewRequest(request))
        # find the root object
        root_factory = self.root_factory
        if routes_mapper is not None:
            info = routes_mapper(request)
            match, route = info['match'], info['route']
            if route is None:
                if debug_routematch:
                    msg = ('no route matched for url %s' %
                           request.url)
                    logger and logger.debug(msg)
            else:
                attrs['matchdict'] = match
                attrs['matched_route'] = route

                if debug_routematch:
                    msg = (
                        'route matched for url %s; '
                        'route_name: %r, '
                        'path_info: %r, '
                        'pattern: %r, '
                        'matchdict: %r, '
                        'predicates: %r' % (
                            request.url,
                            route.name,
                            request.path_info,
                            route.pattern,
                            match,
                            ', '.join([p.text() for p in route.predicates]))
                        )
                    logger and logger.debug(msg)

                request.request_iface = registry.queryUtility(
                    IRouteRequest,
                    name=route.name,
                    default=IRequest)

                root_factory = route.factory or self.root_factory

        root = root_factory(request)
        attrs['root'] = root

        # find a context
        traverser = adapters.queryAdapter(root, ITraverser)
        if traverser is None:
            traverser = ResourceTreeTraverser(root)
        tdict = traverser(request)

        context, view_name, subpath, traversed, vroot, vroot_path = (
            tdict['context'],
            tdict['view_name'],
            tdict['subpath'],
            tdict['traversed'],
            tdict['virtual_root'],
            tdict['virtual_root_path']
            )

        attrs.update(tdict)
        has_listeners and notify(ContextFound(request))

        # find a view callable
        context_iface = providedBy(context)
        #调用view，重点
        response = _call_view(
            registry,
            request,
            context,
            context_iface,
            view_name
            )

        if response is None:
            if self.debug_notfound:
                msg = (
                    'debug_notfound of url %s; path_info: %r, '
                    'context: %r, view_name: %r, subpath: %r, '
                    'traversed: %r, root: %r, vroot: %r, '
                    'vroot_path: %r' % (
                        request.url, request.path_info, context,
                        view_name, subpath, traversed, root, vroot,
                        vroot_path)
                    )
                logger and logger.debug(msg)
            else:
                msg = request.path_info
            raise HTTPNotFound(msg)

        return response

7._call_view
/ProgFile/ichnaea-for-liuqiao/ichnaea/lib/python2.7/site-packages/pyramid/view.py:

def _call_view(
    registry,
    request,
    context,
    context_iface,
    view_name,
    view_types=None,
    view_classifier=None,
    secure=True,
    request_iface=None,
    ):
    if request_iface is None:
        request_iface = getattr(request, 'request_iface', IRequest)
    view_callables = _find_views(
        registry,
        request_iface,
        context_iface,
        view_name,
        view_types=view_types,
        view_classifier=view_classifier,
        )

    pme = None
    response = None

    for view_callable in view_callables:
        # look for views that meet the predicate criteria
        try:
            if not secure:
                # the view will have a __call_permissive__ attribute if it's
                # secured; otherwise it won't.
                view_callable = getattr(
                    view_callable,
                    '__call_permissive__',
                    view_callable
                    )

            # if this view is secured, it will raise a Forbidden
            # appropriately if the executing user does not have the proper
            # permission
            #重点，view_callable的类型是multiview 
            response = view_callable(context, request)
            return response
        except PredicateMismatch as _pme:
            pme = _pme

    if pme is not None:
        raise pme

    return response

8.response = view_callable(context, request)
/ProgFile/ichnaea-for-liuqiao/ichnaea/lib/python2.7/site-packages/pyramid/config/views.py:

    def __call__(self, context, request):
        for order, view, phash in self.get_views(request):
            try:
            #重点:view类型：ichnaea.api.locate.views
                return view(context, request)
            except PredicateMismatch:
                continue
        raise PredicateMismatch(self.name)

get_views(request)：

    def get_views(self, request):
        if self.accepts and hasattr(request, 'accept'):
            accepts = self.accepts[:]
            views = []
            while accepts:
                match = request.accept.best_match(accepts)
                if match is None:
                    break
                subset = self.media_views[match]
                views.extend(subset)
                accepts.remove(match)
            views.extend(self.views)
            return views
        return self.views

9.return view(context, request)：
前面也说了。api的view继承层数比较多，观察控制台输出结果：
pyramid.config.views
ichnaea.api.locate.views
webapp/baseview init start…….
webapp/baseview init finish…….
BaseAPIView init……
BaseAPIView call…….
locate/views view(self,api_key)……..
可以得到以下步骤：
（1）最老祖宗baseview初始化：
/ProgFile/ichnaea-for-liuqiao/ichnaea/lib/python2.7/site-packages/ichnaea-1.5-py2.7-linux-x86_64.egg/ichnaea/webapp/view.py：

    def __init__(self, request):
        """
        Instantiate the view with a request object.
        """
        print 'webapp/baseview __init__ start.......'
        self.request = request
        if self._cors_headers:
            request.response.headers.update(self._cors_headers)
        print 'webapp/baseview __init__ finish.......'

（2）BaseApiView初始化：
/ProgFile/ichnaea-for-liuqiao/ichnaea/lib/python2.7/site-packages/ichnaea-1.5-py2.7-linux-x86_64.egg/ichnaea/api/views.py：

    def __init__(self, request):
        super(BaseAPIView, self).__init__(request)
        print 'BaseAPIView __init__......'
        self.raven_client = request.registry.raven_client
        self.redis_client = request.registry.redis_client
        self.stats_client = request.registry.stats_client

（3）BaseApiView call

    def __call__(self):
        """Execute the view and return a response."""
        很明显，执行view并返回响应
        print 'BaseAPIView __call__.......'
        if self.check_api_key:
            return self.check()
        else:
            api_key = ApiKey(
                valid_key=None, allow_fallback=False,
                allow_locate=True, allow_transfer=False)
            # Only use the unchecked API key in the request for simple
            # logging purposes.
            self.log_count(self.parse_apikey(), False)
            #重点
            return self.view(api_key)

（4）return self.view(api_key)
此时的方法在哪里呢？当然是在：/ProgFile/ichnaea-for-liuqiao/ichnaea/lib/python2.7/site-packages/ichnaea-1.5-py2.7-linux-x86_64.egg/ichnaea/api/submit/views.py：

    def view(self, api_key):
        """
        Execute the view code and return a response.
        """
        try:
            self.submit(api_key)
        except RedisError:
            raise self.prepare_exception(ServiceUnavailable())

        return self.prepare_exception(self.success())

10.终于要提交数据了：self.submit(api_key)

    def submit(self, api_key):
        print 'start to submit........'
        # may raise HTTP error
        # pdb.set_trace()
        #这一步解析请求
        request_data = self.preprocess()

        valid_key = api_key.valid_key
        data = []
        for report in request_data['items']:
            print 'report is:', report
            #此处可打印出值
            source = 'gnss'
            if report is not None:
                position = report.get('position')
                if position is not None:
                    source = position.get('source', 'gnss')

            data.append({
                'api_key': valid_key,
                'report': report,
                'source': source,
            })
        print 'date is:', data
        # pdb.set_trace()
        #重点
        self.queue.enqueue(data)
        self.emit_upload_metrics(len(data), api_key)

11.self.queue.enqueue(data)
这个看起来是把我们整好的数据入队列。那么是站到了哪一队呢？又是怎么加入队列的呢？队列保存在哪儿？

    def __init__(self, request):
        super(BaseSubmitView, self).__init__(request)
        self.queue = self.request.registry.data_queues['update_incoming']

可以看到self.queue队列，名字’update_incoming’刚好是redis里的一个key,这个key的类型是list.说明我们提交的数据存储在了update_incoming里。
接下来看入队函数：

    def enqueue(self, items, batch=None, pipe=None):
        """
        Put items into the queue.

        The items will be pushed into Redis as part of a single (given)
        pipe in batches corresponding to the given batch argument.
        """
        print 'start to enqueue'
        if batch is None:
            batch = self.batch

        if batch == 0:
            batch = len(items)

        if self.json:
            # simplejson.dumps returns Unicode strings
            items = [simplejson.dumps(item, encoding='utf-8').encode('utf-8')
                     for item in items]

        if self.compress:
            print 'compress is true!'
            #items = [util.encode_gzip(item, encoding=None) for item in items]
        #pdb.set_trace()
        if pipe is not None:
            self._push(pipe, items, batch)
        else:
            with redis_pipeline(self.redis_client) as pipe:
                self._push(pipe, items, batch)

这里有几个问题，第一个是编码。

if self.json:
            # simplejson.dumps returns Unicode strings
            items = [simplejson.dumps(item, encoding='utf-8').encode('utf-8')
                     for item in items]
这一段将数据转成utf-8编码存储。

 if self.compress:
            print 'compress is true!'
            #items = [util.encode_gzip(item, encoding=None) for item in items]
            这一段貌似是去除编码

经过观察，以上两段都会执行。执行后，在控制台答应update_incoming 的值，将会出现一种乱乱的看不懂的编码。
把第二段注释掉就好了。
原因似乎是 终端的编码格式需要和redis存入时的格式保持一致 。locale命令请百度。
第二个问题是
with redis_pipeline(self.redis_client) as pipe:
self._push(pipe, items, batch)
with …. as 是个什么鬼？这里有解释

http://blog.csdn.net/wusuopubupt/article/details/29369601

/ProgFile/ichnaea-for-liuqiao/ichnaea/lib/python2.7/site-packages/ichnaea-1.5-py2.7-linux-x86_64.egg/ichnaea/cache.py:

@contextmanager
def redis_pipeline(redis_client, execute=True):
    """
    Return a Redis pipeline usable as a context manager.

    :param execute: Should the pipeline be executed or aborted at the end?
    :type execute: bool
    """
    with redis_client.pipeline() as pipe:
        yield pipe
        if execute:
            pipe.execute()

总而言之，数据被存到了update_incoming里。
然后再一层一层地返回。

接下来的问题是：
1.redis,mysql,celery这三个东西各自职责是什么？他们如何配合的呢？
2.定位的细节是？