Python装饰器的知识请参考:12步轻松搞定python装饰器
@property函数的知识请参考:Python进阶之“属性(property)”详解
看下面一段代码:
定义一个类,然后创建一个实例对象mc,
class Myclass(object):
... def gt_roidb(self):
... print 'gt_roidb...'
... @property
... def ss_roidb(self):
... print 'ss_roidb...'
mc = Myclass()
然后检查Myclass.dict 和 mc.dict
>>> Myclass.__dict__
dict_proxy({'__module__': '__main__', 'gt_roidb': <function gt_roidb at 0x7f43966be668>, 'ss_roidb': <property object at 0x7f43966b5578>, '__dict__': <attribute '__dict__' of 'Myclass' objects>, '__weakref__': <attribute '__weakref__' of 'Myclass' objects>, '__doc__': None})
>>> mc.__dict__
{}
这些都不难理解,而且应该都知道。
接下来看下这些:
>>> Myclass.gt_roidb
<unbound method Myclass.gt_roidb>
>>> Myclass.ss_roidb
<property object at 0x7fbb79490578>
>>> mc.gt_roidb
<bound method Myclass.gt_roidb of <__main__.Myclass object at 0x7fbb794a0250>>
>>> mc.ss_roidb
ss_roidb...
因为ss_roidb是一个属性,所以mc.ss_roidb会直接运行相应的函数def ss_roidb(self)。
接着测试一下调用,
>>> mc.gt_roidb()
gt_roidb...
>>> mc.ss_roidb()
ss_roidb...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable
根据提示,之所以会报错,是因为mc.ss_roidb**没有返回**,它只是打印出了一些东西,当不是返回。所以是’NoneType’, 当然就not callable 了。
修改一下类:
class Myclass(object):
... def gt_roidb(self):
... gt_list = ['a','b','c']
... print 'gt_roidb...'
... return gt_list
... @property
... def ss_roidb(self):
... ss_list = ['d','e','f']
... print 'ssroidb...'
... return ss_list
mc = Myclass()
测试一下,
>>> mc.gt_roidb()
gt_roidb...
['a', 'b', 'c']
>>> mc.ss_roidb
ssroidb...
['d', 'e', 'f']
>>> mc.ss_roidb()
ssroidb...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'list' object is not callable
这里虽然也报错了,但是错误跟之前那个case是不一样的, mc.ss_roidb 返回 一个list,list不能callable, 但是list可以取其中的元素,如下:
>>> mc.ss_roidb[2]
ssroidb...
'f'
所以,当用一个实例对象来调用类的属性的时候,mc.ss_roidb,它等价运行了相应的那个方法,如果它有返回,则也许还能够综合其他一些操作,比如callable(), 或者getitem[] 等。可以参考一下imdb.py中的代码,比如:
@property
def roidb(self):
# A roidb is a list of dictionaries, each with the following keys:
# boxes
# gt_overlaps
# gt_classes
# flipped
if self._roidb is not None:
return self._roidb
self._roidb = self.roidb_handler()
return self._roidb
其中self.roidb_handler
返回的是一个函数对象,是可调用的,(也需要进行调用)所以才在后面加上()
下面贴一段pycaffe.py中的代码
@property
def _Net_blobs(self):
"""
An OrderedDict (bottom to top, i.e., input to output) of network
blobs indexed by name
"""
return OrderedDict(zip(self._blob_names, self._blobs))
@property
def _Net_blob_loss_weights(self):
"""
An OrderedDict (bottom to top, i.e., input to output) of network
blob loss weights indexed by name
"""
return OrderedDict(zip(self._blob_names, self._blob_loss_weights))
@property
def _Net_params(self):
"""
An OrderedDict (bottom to top, i.e., input to output) of network
parameters indexed by name; each is a list of multiple blobs (e.g.,
weights and biases)
"""
return OrderedDict([(name, lr.blobs)
for name, lr in zip(self._layer_names, self.layers)
if len(lr.blobs) > 0])
@property
def _Net_inputs(self):
return [list(self.blobs.keys())[i] for i in self._inputs]
@property
def _Net_outputs(self):
return [list(self.blobs.keys())[i] for i in self._outputs]
def _Net_forward(self, blobs=None, start=None, end=None, **kwargs):
"""
Forward pass: prepare inputs and run the net forward.
Parameters
----------
blobs : list of blobs to return in addition to output blobs.
kwargs : Keys are input blob names and values are blob ndarrays.
For formatting inputs for Caffe, see Net.preprocess().
If None, input is taken from data layers.
start : optional name of layer at which to begin the forward pass
end : optional name of layer at which to finish the forward pass
(inclusive)
Returns
-------
outs : {blob name: blob ndarray} dict.
"""
if blobs is None:
blobs = []
if start is not None:
start_ind = list(self._layer_names).index(start)
else:
start_ind = 0
if end is not None:
end_ind = list(self._layer_names).index(end)
outputs = set([end] + blobs)
else:
end_ind = len(self.layers) - 1
outputs = set(self.outputs + blobs)
if kwargs:
if set(kwargs.keys()) != set(self.inputs):
raise Exception('Input blob arguments do not match net inputs.')
# Set input according to defined shapes and make arrays single and
# C-contiguous as Caffe expects.
for in_, blob in kwargs.iteritems():
if blob.shape[0] != self.blobs[in_].num:
raise Exception('Input is not batch sized')
# 将实参中的blob传递给网络的blobs(通过blob的name即in_来索引相应的blob)
self.blobs[in_].data[...] = blob
self._forward(start_ind, end_ind)
# Unpack blobs to extract
return {out: self.blobs[out].data for out in outputs}
def _Net_backward(self, diffs=None, start=None, end=None, **kwargs):
"""
Backward pass: prepare diffs and run the net backward.
Parameters
----------
diffs : list of diffs to return in addition to bottom diffs.
kwargs : Keys are output blob names and values are diff ndarrays.
If None, top diffs are taken from forward loss.
start : optional name of layer at which to begin the backward pass
end : optional name of layer at which to finish the backward pass
(inclusive)
Returns
-------
outs: {blob name: diff ndarray} dict.
"""
if diffs is None:
diffs = []
if start is not None:
start_ind = list(self._layer_names).index(start)
else:
start_ind = len(self.layers) - 1
if end is not None:
end_ind = list(self._layer_names).index(end)
outputs = set([end] + diffs)
else:
end_ind = 0
outputs = set(self.inputs + diffs)
if kwargs:
if set(kwargs.keys()) != set(self.outputs):
raise Exception('Top diff arguments do not match net outputs.')
# Set top diffs according to defined shapes and make arrays single and
# C-contiguous as Caffe expects.
for top, diff in kwargs.iteritems():
if diff.shape[0] != self.blobs[top].num:
raise Exception('Diff is not batch sized')
#
self.blobs[top].diff[...] = diff
self._backward(start_ind, end_ind)
# Unpack diffs to extract
return {out: self.blobs[out].diff for out in outputs}
初次看的时候,也看不懂,没办法,只能靠自己研究。以下纯属个人理解,学识有限,求指正!
此处分割线
在Python进阶之“属性(property)”详解中有提到:
Python中有一个被称为属性函数(property)的小概念,它可以做一些有用的事情, 比如:1. 将类方法转换为只读属性 2. 重新实现一个属性的setter和getter方法。所以:
1. 很自然的一个问题就是:在pycaffe.py中,并没有声明一个类,而且其中的一些函数的声明都包含了self参数,self是跟具体某个类的实例对象绑定的,那么这个类的定义在哪里呢???
首先,Python中可以在类外定义方法(我觉得这应该没有问题,毕竟python是一门非常强大的语言)找到一篇文章Python class 入门,可以参考一下:下面是我自己定义的一个类:
#! /usr/bin/env python
@property
def upperName(self):
return self.name.upper()
class Person:
Year = 2016 # 类属性,而不是实例属性
def __init__(self, name):
self.name = name
self.age = 23
@property
def name(self):
return self.name
在解释器中创建实例对象:p = Person("sam")
,然后查看一些属性, 如下:
>>> Person.__dict__
{'__module__': '__main__', '__doc__': None, 'name': <property object at 0x7f094fa965d0>, '__init__': <function __init__ at 0x7f094fa9f6e0>, 'Year': 2016}
可以看到此时的类对象Person并没有upperName这个属性
>>> p.__dict__
{'age': 23, 'name': 'sam'}
这是实例对象p的属性
然后,将upperName这个对象赋值给Person.upper,此行代码同时给类Person创建了一个新的属性upper。这就是python的强大之处:能够在“运行时”创建属性。
注意在python中,一切皆为对象,类是对象,类的实例也是对象,函数是对象,变量是对象等等。。。
Person.upper = upperName
在运行这行代码之前,运行p.upperName
或者Person.upperName
都会报错:
>>> p.upperName
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: Person instance has no attribute 'upperName'
>>> Person.upperName
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: class Person has no attribute 'upperName'
但,如果运行了 Person.upper = upperName 呢?我们先看Person的属性:
>>> Person.__dict__
{'upper': <property object at 0x7f094fa96578>, '__module__': '__main__', 'name': <property object at 0x7f094fa965d0>, 'Year': 2016, '__doc__': None, '__init__': <function __init__ at 0x7f094fa9f6e0>}
显然多了一个属性:'upper': <property object at 0x7f094fa96578>
, 而实例对象p就没有变化:
>>> p.__dict__
{'age': 23, 'name': 'sam'}
接下来,调用:
>>> p.upper
'SAM'
OK,可见python的强大之处。
注意这是利用实例对象调用的upper,如用类来调用,并不会运行def upperName(self), 只是输出一些信息:
>>> Person.upper
<property object at 0x7f094fa96578>
那么,在pycaffe.py中,情况如何呢???请看下面的代码:
# Attach methods to Net.
Net.blobs = _Net_blobs
Net.blob_loss_weights = _Net_blob_loss_weights
Net.params = _Net_params
Net.forward = _Net_forward
Net.backward = _Net_backward
Net.forward_all = _Net_forward_all
Net.forward_backward_all = _Net_forward_backward_all
Net.set_input_arrays = _Net_set_input_arrays
Net._batch = _Net_batch
Net.inputs = _Net_inputs
Net.outputs = _Net_outputs
Net是从_caffe中导入的一个类(个人理解):from ._caffe import Net,那么上述代码就是将在pycaffe.py中定义的属性“绑定”到Net上,也就是Person.upper = upperName
这个操作。
最后,用pacaffe.py中的注释作为总结: Wrap the internal caffe C++ module (_caffe.so) with a clean, Pythonic
interface.
以_Net_blobs说下这些属性
@property
def _Net_blobs(self):
"""
An OrderedDict (bottom to top, i.e., input to output) of network
blobs indexed by name
"""
return OrderedDict(zip(self._blob_names, self._blobs))
a) self._blob_names
调用Caffe.Net的blob_names属性(底层调用Net::blob_names方法),返回blob_names , 即整个网络中所有非参数blob的name
b) self._blobs
调用Caffe.Net的blobs属性(底层调用Net::blobs方法)返回blobs, 即整个网络中所有非参数blob
c) return OrderedDict(zip(self._blob_names, self._blobs))
以字典的方式返回
还有一个类:
class _Net_IdNameWrapper:
"""
A simple wrapper that allows the ids propery to be accessed as a dict
indexed by names. Used for top and bottom names
"""
def __init__(self, net, func):
self.net, self.func = net, func
def __getitem__(self, name):
# Map the layer name to id
ids = self.func(self.net, list(self.net._layer_names).index(name))
# Map the blob id to name
id_to_name = list(self.net.blobs)
return [id_to_name[i] for i in ids]
对应的有:
# property为内建属性函数
Net.top_names = property(lambda n: _Net_IdNameWrapper(n, Net._top_ids))
Net.bottom_names = property(lambda n: _Net_IdNameWrapper(n, Net._bottom_ids))
在底层,top blob, bottom blob 是按id来索引访问的,所以在底层就相应的有top_id_vecs_ 和 bottom_id_vecs_ 这两个变量。而class _Net_IdNameWrapper 的功能就是将这种按id访问,改为按blob的name来访问,在训练的时候带来方便性。
下面以 Net.top_names = property(lambda n: _Net_IdNameWrapper(n, Net._top_ids)) 为例,解释一下。
a) _Net_IdNameWrapper(n, Net._top_ids) 创建一个_Net_IdNameWrapper对象, 那么它是如何实现利用blob name来访问blob的呢?
b) 其实是通过def getitem(self, name)来定制的: ids = self.func(self.net, list(self.net._layer_names).index(name))
这里self,表示Net_IdNameWrapper实例对象。self.net._layer_names调用Caffe.Net的_layer_names属性,返回网络中所有layer的名字,index(name)找到对应name的layer的id,而self.func即为Net._top_ids方法(定义在_caffe.cpp中),它调用底层的Net::top_ids方法,返回Net::top_id_vecs, 其类型为:vector<vector<int> >
,所以综合一下可知,ids为名字为name的layer的所以top blob 的id。 这里需要注意的是list(dict) 返回的是dict的键的列表
c) id_to_name = list(self.net.blobs)
:self.net.blobs调用caffe.Net的blobs属性返回网络中所有的非参数blob所组成的字典,这是因为caffe.Net的blobs属性已经经过wraped .[参见def _Net_blobs(self)]。即id_to_name是一个blob name 的列表,里面存储了网络的所有非参数blob的名字。
d) return [id_to_name[i] for i in ids]
将所有需要的blob name以列表的形式返回。
总之Net.top_names这个属性的功能就是返回给定名字为name的层的所有top blob。