Swift源码分析----swift-account-audit(2)

感谢朋友支持本博客，欢迎共同探讨交流，由于能力和时间有限，错误之处在所难免，欢迎指正！

如果转载，请保留作者信息。
博客地址：http://blog.csdn.net/gaoxingnengjisuan
邮箱地址：dong.liu@siat.ac.cn

PS：最近没有登录博客，很多朋友的留言没有看见，这里道歉！还有就是本人较少上QQ，可以邮件交流。

接续上篇：

转到3.2，来看方法audit_container的实现：

def audit_container(self, account, name, recurse=False):
        """
        指定container的审计验证，并实现递归验证container下每个object；
        """
        if (account, name) in self.in_progress:
            self.in_progress[(account, name)].wait()
        if (account, name) in self.list_cache:
            return self.list_cache[(account, name)]
        self.in_progress[(account, name)] = Event()
        print 'Auditing container "%s"' % name
        
        # # 指定指定account下的容器具体路径；
        path = '/%s/%s' % (account, name)
        
        # 获取指定account下的容器列表；
        account_listing = self.audit_account(account)
        
        consistent = True
        if name not in account_listing:
            consistent = False
            print "  Container %s not in account listing!" % path
        
        # 获取指定name容器的所有副本的相关节点和分区号；
        # 获取account/container/object所对应的分区号和节点（可能是多个，因为分区副本有多个，可能位于不同的节点上）；
        # 返回元组（分区，节点信息列表）；
        # 在节点信息列表中至少包含id、weight、zone、ip、port、device、meta；
        part, nodes = self.container_ring.get_nodes(account, name.encode('utf-8'))
        rec_d = {}
        responses = {}
        for node in nodes:
            marker = ''
            results = True
            while results:
                try:
                    conn = http_connect(node['ip'], node['port'],
                                        node['device'], part, 'GET',
                                        path.encode('utf-8'), {},
                                        'format=json&marker=%s' %
                                        quote(marker.encode('utf-8')))
                    # 获取来自服务器的响应；
                    resp = conn.getresponse()
                    if resp.status // 100 != 2:
                        self.container_not_found += 1
                        consistent = False
                        print('  Bad status GETting container "%s" on %s/%s' %
                              (path, node['ip'], node['device']))
                        break
                    
                    if node['id'] not in responses:
                        responses[node['id']] = dict(resp.getheaders())
                    results = simplejson.loads(resp.read())
                except Exception:
                    self.container_exceptions += 1
                    consistent = False
                    print '  Exception GETting container "%s" on %s/%s' % \
                        (path, node['ip'], node['device'])
                    break
                if results:
                    marker = results[-1]['name']
                    for obj in results:
                        obj_name = obj['name']
                        if obj_name not in rec_d:
                            rec_d[obj_name] = obj
                        if (obj['last_modified'] !=
                                rec_d[obj_name]['last_modified']):
                            self.container_obj_mismatch += 1
                            consistent = False
                            print("  Different versions of %s/%s "
                                  "in container dbs." % (name, obj['name']))
                            if (obj['last_modified'] >
                                    rec_d[obj_name]['last_modified']):
                                rec_d[obj_name] = obj
        obj_counts = [int(header['x-container-object-count'])
                      for header in responses.values()]
        if not obj_counts:
            consistent = False
            print "  Failed to fetch container %s at all!" % path
        else:
            if len(set(obj_counts)) != 1:
                self.container_count_mismatch += 1
                consistent = False
                print "  Container databases don't agree on number of objects."
                print "  Max: %s, Min: %s" % (max(obj_counts), min(obj_counts))
        self.containers_checked += 1
        self.list_cache[(account, name)] = rec_d
        self.in_progress[(account, name)].send(True)
        del self.in_progress[(account, name)]
        
        # 递归验证container下每个object；
        if recurse:
            for obj in rec_d.keys():
                self.pool.spawn_n(self.audit_object, account, name, obj)
        if not consistent and self.error_file:
            print >>open(self.error_file, 'a'), path
        return rec_d

3.2.1 获取指定account下的容器具体路径；
3.2.2 调用方法audit_account实现获取指定account下的容器列表，验证当前指定容器是否包含其中；
3.2.3 获取指定name容器的所有副本的相关节点和分区号；
3.2.4 针对容器的所有副本相关节点，进行遍历，对于每个节点执行以下操作：
      通过HTTP应用GET方法远程获取节点的验证响应信息，通过响应信息的状态值，判断远程节点副本容器是否存在；
3.2.5 递归调用方法audit_object实现审计验证容器下每个object；

转到3.3，来看方法audit_account的实现：

def audit_account(self, account, recurse=False):
        """
        指定account的审计验证，并实现递归验证account下每个container，并且进一步实现递归验证container下每个object；
        """
        if account in self.in_progress:
            self.in_progress[account].wait()
        if account in self.list_cache:
            return self.list_cache[account]
        self.in_progress[account] = Event()
        print 'Auditing account "%s"' % account
        consistent = True
        path = '/%s' % account
        
        # 获取指定name账户的所有副本的相关节点和分区号；
        # 获取account所对应的分区号和节点（可能是多个，因为分区副本有多个，可能位于不同的节点上）；
        # 返回元组（分区，节点信息列表）；
        # 在节点信息列表中至少包含id、weight、zone、ip、port、device、meta；
        part, nodes = self.account_ring.get_nodes(account)
        
        responses = {}
        for node in nodes:
            marker = ''
            results = True
            while results:
                node_id = node['id']
                try:   
                    # 建立一个HTTPConnection类的对象；
                    # 并获取来自服务器的响应信息；
                    conn = http_connect(node['ip'], node['port'],
                                        node['device'], part, 'GET', path, {},
                                        'format=json&marker=%s' %
                                        quote(marker.encode('utf-8')))
                    resp = conn.getresponse()
                    if resp.status // 100 != 2:
                        self.account_not_found += 1
                        consistent = False
                        print("  Bad status GETting account '%s' "
                              " from %ss:%ss" %
                              (account, node['ip'], node['device']))
                        break
                    results = simplejson.loads(resp.read())    
                except Exception:
                    self.account_exceptions += 1
                    consistent = False
                    print("  Exception GETting account '%s' on %ss:%ss" %
                          (account, node['ip'], node['device']))
                    break
                
                if node_id not in responses:
                    responses[node_id] = [dict(resp.getheaders()), []]
                responses[node_id][1].extend(results)
                if results:
                    marker = results[-1]['name']
        
        headers = [resp[0] for resp in responses.values()]
        cont_counts = [int(header['x-account-container-count'])
                       for header in headers]
        if len(set(cont_counts)) != 1:
            self.account_container_mismatch += 1
            consistent = False
            print("  Account databases for '%s' don't agree on"
                  " number of containers." % account)
            if cont_counts:
                print "  Max: %s, Min: %s" % (max(cont_counts),
                                              min(cont_counts))
        obj_counts = [int(header['x-account-object-count'])
                      for header in headers]
        if len(set(obj_counts)) != 1:
            self.account_object_mismatch += 1
            consistent = False
            print("  Account databases for '%s' don't agree on"
                  " number of objects." % account)
            if obj_counts:
                print "  Max: %s, Min: %s" % (max(obj_counts),
                                              min(obj_counts))
        containers = set()
        for resp in responses.values():
            containers.update(container['name'] for container in resp[1])
        self.list_cache[account] = containers
        self.in_progress[account].send(True)
        del self.in_progress[account]
        self.accounts_checked += 1
        if recurse:
            for container in containers:
                self.pool.spawn_n(self.audit_container, account,
                                  container, True)
        if not consistent and self.error_file:
            print >>open(self.error_file, 'a'), path
        return containers

3.3.1 获取指定账户的具体路径；
3.3.2 获取指定name账户的所有副本的相关节点和分区号；
3.3.4 针对账户的所有副本相关节点，进行遍历，对于每个节点执行以下操作：
      通过HTTP应用GET方法远程获取节点的验证响应信息，通过响应信息的状态值，判断远程副本账户是否存在；
3.3.5 递归调用方法audit_container实现审计验证账户下每个container；


至此，这个脚本的账户/容器/脚本的审计验证流程分析完成，当然在具体审计验证过程中还有很多其他细节，这里不再赘述。