本文深入解析YOLOv3目标检测模型，探讨其设计理念，网络结构，锚框（anchor）的概念及其在训练和预测阶段的应用。文章还介绍了交并比（IoU）的意义，非最大值抑制（Non-Maximum Suppression, NMS）的作用，并展示了Darknet53模型结构。此外，提供了YOLOv3和Tiny YOLOv3的模型构建、损失函数以及训练过程中的数据预处理。最后，详述了如何使用YOLOv3进行图像和视频的检测。
yolov3详细解释
yolo3是如何构想的 并且网络结构是什么样的
论文地址：https://pjreddie.com/media/files/papers/YOLOv3.pdf
yolov3在yolov1和v2的基础上进行了改进，如果想了解v1和v2可以自行查阅
推荐链接：
https://blog.csdn.net/litt1e/article/details/88814417
https://blog.csdn.net/litt1e/article/details/88852745
v3将原来的图片进行不同细致程度的检测层划分，然后利用每一个小网格来负责检测相应位置的信息。并且v3得到了三种不同细致程度的检测层，在原始图像为416x416的前提下，三种划分为：13x13 26x26 52x52（这些都是不同检测层的行列数）。这种不同维度划分，有利于对不同尺度大小的物体进行检测。13x13对应最大物体的检测，26x26对应中等物体的检测，52x52对应最小物体的检测。每一个网格只能用来检测一个物体，所以我们用标记方框的中心点来确定此物体的位置，这样就可以很大程度的解决一个网格有很多的物体而不知道该检测谁的问题。每一种尺度的每一个检测层拥有三个bounding box检测块，每一个bounding box检测块对应一个anchor锚框，一共有九个anchor锚框，每个检测划分层对应三个anchor锚框。（上图以13x13为例 假设有5分类）
锚框anchor的含义（锚框anchor的构建用于训练时期 和预测时期）
锚框anchor是 通过聚类总结出来的，每张图片都有对自己所包含的物体的标记框，每个框的大小也不同，有为了标记猫而画的框，也有为马标记的框，所以大小各有不同。因此v3通过聚类的方式，将所有不同大小的进行了k聚类，一共得到九个锚框anchor。于是，锚框的含义也就显然了，就是把被标记的图片中的物体分为了九种大小格式，一种对应一个bounding box检测块层，这样就相当于提前对需检测的物体进行了大小划分，对大小不一的物体，负责检测的检测层也不一样，并且bound ing box也不一样。（如果还是不理解，仔细看代码preprocess_true_boxes过程）
交并比的意义
交并比是两个方框的重合面积（交）比上两个方框所占总面积（并）得到的。
非最大值抑制是什么（用在模型训练完以后，进行预测时）
非最大值抑制，从字面意义上很容易看出是将不是最大可能的预测给否定了，只留下最有可能（confidence）的预测。非最大值抑制是按类别classes来进行操作的，不同类别不会互相干扰。假设一共有20类预测的物体，首先从第一类开始，找到此类中所有预测可能性confidence最大的预测，找到与此预测的检测网格交并比在某个阈值以上的所有检测网格并删除（v3中设置非最大值抑制的阈值），记录这个最可能预测，并在后面的抑制中排除此项。但是此时第一类还没有进行完，还有其他位置有此类物体的预测（换种说法就是 与前面的预测交并比小于所设阈值的这些预测还存在），同样的进行前面的操作，直到最后。（三种不同大小的检测层会同时参加非最大值抑制，但不会影响排除的过程，随然不同检测层的物体可能会同时属于同一类物体，且中心点同时在划分较密检测层同一个网格中，但因为大小差别很大，所以交并比不会太大，很难大于所设定的阈值。）
（可参看吴恩达作业题
推荐链接：https://blog.csdn.net/u013733326/article/details/80341740）
主要的模型model的构建 darknet53
其中DarknetConv2D_BN_Leaky是基础的网络块，包括卷积层，正则化层，激活函数层。minblock是叠加卷积层的基本块，并且进行下采样（add）。resblock_body构建Darknet53的内层块，一共有5个rb块，所以进行五次缩小尺度。所以原图片分辨率是最后的检测层的32倍大小。
make_last_layers:作用是将原输出进行处理，一支分流做几次卷积后直接输出，来做检测层，另一支分流做下一个处理的上采样层。整体的网络结构图就是上面的图片。
损失函数loss function 锚框anchor在损失函数中的应用
图中bx,by,bw,bh是真实框的中心坐标和宽高，pw,ph是这个物体所对应的锚框的宽高，加入激活函数sigmoid来确定中心位置，先求出中心对于网格的相对位置，再加整体相对位置，减少数值过大带来的误差。
损失函数在绝大程度上只关注有物体的检测快，没有物体的检测快只是在confidence中添加了一个可操控参数来控制这项损失对模型的影响。
代码的每一步作用
model.py
"""YOLO_v3 Model Defined in Keras."""
from functools import wraps
import numpy as np
import tensorflow as tf
from keras import backend as K
from keras.layers import Conv2D, Add, ZeroPadding2D, UpSampling2D, Concatenate, MaxPooling2D
from keras.layers.advanced_activations import LeakyReLU
from keras.layers.normalization import BatchNormalization
from keras.models import Model
from keras.regularizers import l2
from yolo3.utils import compose
@wraps(Conv2D)
def DarknetConv2D(*args, **kwargs):
"""Wrapper to set Darknet parameters for Convolution2D."""
#为卷积2d设置参数的包装器。
#设置正则项,最后会在损失函数中体现
darknet_conv_kwargs = {'kernel_regularizer': l2(5e-4)}
#设置是否自动填充，s=2则用ZeroPadding来填充
darknet_conv_kwargs['padding'] = 'valid' if kwargs.get('strides')==(2,2) else 'same'
#更新字典参数项，如果输入里有其他项，则更新添加到字典中
darknet_conv_kwargs.update(kwargs)
#返回一个二维的卷积层
return Conv2D(*args, **darknet_conv_kwargs)
def DarknetConv2D_BN_Leaky(*args, **kwargs):
"""Darknet Convolution2D followed by BatchNormalization and LeakyReLU."""
#Darknet Convolution2D紧接着是BatchNormalization和LeakyReLU
#设置卷积中不带有偏执
no_bias_kwargs = {'use_bias': False}
no_bias_kwargs.update(kwargs)
#返回一个卷积 正则 弱Relu 层
return compose(
DarknetConv2D(*args, **no_bias_kwargs),
BatchNormalization(),
LeakyReLU(alpha=0.1))
def resblock_body(x, num_filters, num_blocks):
'''A series of resblocks starting with a downsampling Convolution2D'''
#进行降采样卷积2d的一系列块
# Darknet uses left and top padding instead of 'same' mode
#对图像进行填补，s=2，得到层的高和宽缩小一半
x = ZeroPadding2D(((1,0),(1,0)))(x)
x = DarknetConv2D_BN_Leaky(num_filters, (3,3), strides=(2,2))(x)
#num_blocks控制着卷积层的数量
for i in range(num_blocks):
y = compose(
DarknetConv2D_BN_Leaky(num_filters//2, (1,1)),
DarknetConv2D_BN_Leaky(num_filters, (3,3)))(x)
#进行下采样，x,y的维数相同
x = Add()([x,y])
return x
#这里可以进行更改，替换层数
def darknet_body(x):
'''Darknent body having 52 Convolution2D layers'''
#拥有52个Convolution2D层的darknet_body
x = DarknetConv2D_BN_Leaky(32, (3,3))(x)
x = resblock_body(x, 64, 1)
x = resblock_body(x, 128, 2)
x = resblock_body(x, 256, 8)
x = resblock_body(x, 512, 8)
x = resblock_body(x, 1024, 4)
return x
def make_last_layers(x, num_filters, out_filters):
'''6 Conv2D_BN_Leaky layers followed by a Conv2D_linear layer'''
#x用来做上采样，y用来输出检测层
x = compose(
DarknetConv2D_BN_Leaky(num_filters, (1,1)),
DarknetConv2D_BN_Leaky(num_filters*2, (3,3)),
DarknetConv2D_BN_Leaky(num_filters, (1,1)),
DarknetConv2D_BN_Leaky(num_filters*2, (3,3)),
DarknetConv2D_BN_Leaky(num_filters, (1,1)))(x)
y = compose(
DarknetConv2D_BN_Leaky(num_filters*2, (3,3)),
DarknetConv2D(out_filters, (1,1)))(x)
return x, y
#在train和yolo预测时进行运用
def yolo_body(inputs, num_anchors, num_classes):
"""Create YOLO_V3 model CNN body in Keras."""
#yolo_v3的模型层
#建立一个模型，保存darknet的所有层的运行结果
darknet = Model(inputs, darknet_body(inputs))
#输出第一次上采样的x，并且输出大目标的检测层y1
x, y1 = make_last_layers(darknet.output, 512, num_anchors*(num_classes+5))
#将x高和宽放大两倍，建立用作上采样的x
x = compose(
DarknetConv2D_BN_Leaky(256, (1,1)),
UpSampling2D(2))(x)
#将x与darknet的第152层叠加concat，输出中等目标的检测层
x = Concatenate()([x,darknet.layers[152].output])
x, y2 = make_last_layers(x, 256, num_anchors*(num_classes+5))
#同理，进行第二次上采样，输出小目标的检测层
x = compose(
DarknetConv2D_BN_Leaky(128, (1,1)),
UpSampling2D(2))(x)
x = Concatenate()([x,darknet.layers[92].output])
x, y3 = make_last_layers(x, 128, num_anchors*(num_classes+5))
return Model(inputs, [y1,y2,y3])
#在train和yolo预测时进行运用
def tiny_yolo_body(inputs, num_anchors, num_classes):
'''Create Tiny YOLO_v3 model CNN body in keras.'''
x1 = compose(
DarknetConv2D_BN_Leaky(16, (3,3)),
MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
DarknetConv2D_BN_Leaky(32, (3,3)),
MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
DarknetConv2D_BN_Leaky(64, (3,3)),
MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
DarknetConv2D_BN_Leaky(128, (3,3)),
MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
DarknetConv2D_BN_Leaky(256, (3,3)))(inputs)
x2 = compose(
MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
DarknetConv2D_BN_Leaky(512, (3,3)),
MaxPooling2D(pool_size=(2,2), strides=(1,1), padding='same'),
DarknetConv2D_BN_Leaky(1024, (3,3)),
DarknetConv2D_BN_Leaky(256, (1,1)))(x1)
y1 = compose(
DarknetConv2D_BN_Leaky(512, (3,3)),
DarknetConv2D(num_anchors*(num_classes+5), (1,1)))(x2)
x2 = compose(
DarknetConv2D_BN_Leaky(128, (1,1)),
UpSampling2D(2))(x2)
y2 = compose(
Concatenate(),
DarknetConv2D_BN_Leaky(256, (3,3)),
DarknetConv2D(num_anchors*(num_classes+5), (1,1)))([x2,x1])
return Model(inputs, [y1,y2])
#在loss中和yolo_boxes_and_scores进行运用
def yolo_head(feats, anchors, num_classes, input_shape, calc_loss=False):
#参数解释 feats 是yolo body输出的三个检测层中的一个
"""Convert final layer features to bounding box parameters."""
#转换最终层特征为边界框参数。
num_anchors = len(anchors)
# Reshape to batch, height, width, num_anchors, box_params.
#重塑为批，高度，宽度，num_anchors, box_params。
anchors_tensor = K.reshape(K.constant(anchors), [1, 1, 1, num_anchors, 2])
#获得feats的高度和宽度 13 13 或 26 26 或 52 52
grid_shape = K.shape(feats)[1:3] # height, width
#先获得一列高的值，这个列是相对于四维向量说的列，在将这列铺成宽的长度得到[h,w,1,1]的量
grid_y = K.tile(K.reshape(K.arange(0, stop=grid_shape[0]), [-1, 1, 1, 1]),
[1, grid_shape[1], 1, 1])
#先获得一行宽的值，这个行是相对于思维向量说的行，在将这行铺成高的长度得到[h,w,1,1]的量
grid_x = K.tile(K.reshape(K.arange(0, stop=grid_shape[1]), [1, -1, 1, 1]),
[grid_shape[0], 1, 1, 1])
#再将它俩按最后一维度进行拼接 得到[h,w,1,2]的量 其中其实就是行列的坐标,但是坐标第一位代表第几列 w
grid = K.concatenate([grid_x, grid_y])
#得到和feats一样类型的坐标张量,并且必须得是tf量才能有dtype ，所以feats是tf张量，tf.~.array也有关系
grid = K.cast(grid, K.dtype(feats))
#将feats转换为[batch,h,w,3,25] 用的是yolo_body 类为20类 表示
feats = K.reshape(
feats, [-1, grid_shape[0], grid_shape[1], num_anchors, num_classes + 5])
# Adjust preditions to each spatial grid point and anchor size.
#调整预测到每个空间网格点和锚的大小。::-1 取从后向前相反元素 变为[2,1,w,h]
#有公式bx = sigmoid(tx) + cx, by = sigmoid(ty) + cy, bw = pw*exp(tw), bh = ph*exp(th)， p为锚框
#除以长度和宽度进行归一化，对训练数据也进行同样的归一化处理 preprocess_true_boxes
box_xy = (K.sigmoid(feats[..., :2]) + grid) / K.cast(grid_shape[::-1], K.dtype(feats))#grid_shape是二维的 (1,2)
#一个box对应一个anchor 根据下面的方法preprocess_true_boxes推出 每个检测器中有3个box 和对应 3个锚框
box_wh = K.exp(feats[..., 2:4]) * anchors_tensor / K.cast(input_shape[::-1], K.dtype(feats))
box_confidence = K.sigmoid(feats[..., 4:5])#预测时才进行逻辑回归 使得概率在0到1之间
box_class_probs = K.sigmoid(feats[..., 5:])#预测时才进行逻辑回归 使得概率在0到1之间
#True的时候是为了损失函数的计算得到预测值
if calc_loss == True:
return grid, feats, box_xy, box_wh#没有返回confidence和class
#虽然是在yolo_boxes_and_scores中应用，但是最终是为了预测而返回
return box_xy, box_wh, box_confidence, box_class_probs
#在yolo_boxes_and_scores中应用
def yolo_correct_boxes(box_xy, box_wh, input_shape, image_shape):
'''Get corrected boxes'''
#因为在训练时输入的大小要统一规划到 input_shape 所以必须将图片返回到原来的尺寸
box_yx = box_xy[..., ::-1]
box_hw = box_wh[..., ::-1]
#输入图像的大小 例 (416,416)
input_shape = K.cast(input_shape, K.dtype(box_yx))
#原图像的大小 例 (720,570)
image_shape = K.cast(image_shape, K.dtype(box_yx))
#得到输入图像高宽与原图像高宽比中较小的 与原图像相乘
new_shape = K.round(image_shape * K.min(input_shape/image_shape))
#得到处理图像时增补的像素大小，并且因为在head中进行归一化，所以这里要除以Input_shape，
#2是因为图像两边同时填充相同大小，可参照utils中的letterbox_image
offset = (input_shape-new_shape)/2./input_shape
#scale的作用是还原图像的大小，在训练时处理图像用的是这里scale的倒数
scale = input_shape/new_shape#此处的input_shape可以抵消归一化时除以的input_shape
box_yx = (box_yx - offset) * scale
box_hw *= scale
#得到另一种方框表达形式，两点定一框
box_mins = box_yx - (box_hw / 2.)
box_maxes = box_yx + (box_hw / 2.)
boxes = K.concatenate([
box_mins[..., 0:1], # y_min
box_mins[..., 1:2], # x_min
box_maxes[..., 0:1], # y_max
box_maxes[..., 1:2] # x_max
])
# Scale boxes back to original image shape.
#将盒子缩放回原始图像形状，乘以image_shape为了返回原图大小对应的box
boxes *= K.concatenate([image_shape, image_shape])
return boxes
#在yolo_eval中进行应用
def yolo_boxes_and_scores(feats, anchors, num_classes, input_shape, image_shape):
'''Process Conv layer output'''
#参数解释：anchors的作用是用在yolo_head中的
box_xy, box_wh, box_confidence, box_class_probs = yolo_head(feats,
anchors, num_classes, input_shape)
boxes = yolo_correct_boxes(box_xy, box_wh, input_shape, image_shape)
#得到一列boxes和对应的一列scores
boxes = K.reshape(boxes, [-1, 4])
box_scores = box_confidence * box_class_probs
box_scores = K.reshape(box_scores, [-1, num_classes])
return boxes, box_scores
#在yolo 预测是进行应用
def yolo_eval(yolo_outputs,
anchors,
num_classes,
image_shape,
max_boxes=20,
score_threshold=.6,
iou_threshold=.5):
"""Evaluate YOLO model on given input and return filtered boxes."""
#参数解释：max_boxes是预测最大数量的物体，score_threshold以下的分数直接抛弃，iou_threshold大于此交并比的抛弃
#得到yolo输出的检测层数
num_layers = len(yolo_outputs)
#分配锚框给每个检测器
anchor_mask = [[6,7,8], [3,4,5], [0,1,2]] if num_layers==3 else [[3,4,5], [1,2,3]] # default setting
#得到最小那个检测器的高和宽 乘以32 得到输入图像的大小
input_shape = K.shape(yolo_outputs[0])[1:3] * 32
boxes = []
box_scores = []
#循环每一个检测器
for l in range(num_layers):
_boxes, _box_scores = yolo_boxes_and_scores(yolo_outputs[l],
anchors[anchor_mask[l]], num_classes, input_shape, image_shape)
boxes.append(_boxes)
box_scores.append(_box_scores)
#将三个检测器的boxes合并成一列 box_scores合并成一列
boxes = K.concatenate(boxes, axis=0)
box_scores = K.concatenate(box_scores, axis=0)
#将大于指定分数阈值的对应框的mask定义为1 其他定为0 [...,20] 假定有20类
mask = box_scores >= score_threshold
#将max_boxes变为tensor量
max_boxes_tensor = K.constant(max_boxes, dtype='int32')
boxes_ = []
scores_ = []
classes_ = []
#对每一个类进行预测
for c in range(num_classes):
# TODO: use keras backend instead of tf.
#c这一列的高过阈值的boxes
class_boxes = tf.boolean_mask(boxes, mask[:, c])
#c这一列的高过阈值的scores
class_box_scores = tf.boolean_mask(box_scores[:, c], mask[:, c])
#进行最值抑制，得到剩下的boxes标记 去除交并比阈值以上的其他boxes
nms_index = tf.image.non_max_suppression(
class_boxes, class_box_scores, max_boxes_tensor, iou_threshold=iou_threshold)
#聚合所有标记下的框和分数
class_boxes = K.gather(class_boxes, nms_index)
class_box_scores = K.gather(class_box_scores, nms_index)
#标记是哪一类
classes = K.ones_like(class_box_scores, 'int32') * c
boxes_.append(class_boxes)
scores_.append(class_box_scores)
classes_.append(classes)
boxes_ = K.concatenate(boxes_, axis=0)
scores_ = K.concatenate(scores_, axis=0)
classes_ = K.concatenate(classes_, axis=0)
#得到最终需要的预测框和类以及分数
return boxes_, scores_, classes_
#在train里进行运用
def preprocess_true_boxes(true_boxes, input_shape, anchors, num_classes):
'''Preprocess true boxes to training input format
Parameters
----------
true_boxes: array, shape=(m, T, 5)
Absolute x_min, y_min, x_max, y_max, class_id relative to input_shape.
input_shape: array-like, hw, multiples of 32
anchors: array, shape=(N, 2), wh
num_classes: integer
Returns
-------
y_true: list of array, shape like yolo_outputs, xywh are reletive value
'''
#true_boxes m是batch T是几个boxes 5是boxes的值
#确保预测类必须在定义类的范围内
assert (true_boxes[..., 4]<num_classes).all(), 'class id must be less than num_classes'
num_layers = len(anchors)//3 # default setting
anchor_mask = [[6,7,8], [3,4,5], [0,1,2]] if num_layers==3 else [[3,4,5], [1,2,3]]
#转换为tensor量
true_boxes = np.array(true_boxes, dtype='float32')
input_shape = np.array(input_shape, dtype='int32')
#得到xy 和 wh
boxes_xy = (true_boxes[..., 0:2] + true_boxes[..., 2:4]) // 2
boxes_wh = true_boxes[..., 2:4] - true_boxes[..., 0:2]
#归一化 true_bosex是 wh 但 input_shape 是 hw 所以要倒叙
true_boxes[..., 0:2] = boxes_xy/input_shape[::-1]
true_boxes[..., 2:4] = boxes_wh/input_shape[::-1]
#得到batch_size的值
m = true_boxes.shape[0]
#得到三个检测器尺寸 如 13,13 26,26 52,52
grid_shapes = [input_shape//{0:32, 1:16, 2:8}[l] for l in range(num_layers)]
#得到三个 0 张量 与三个检测器对应
y_true = [np.zeros((m,grid_shapes[l][0],grid_shapes[l][1],len(anchor_mask[l]),5+num_classes),
dtype='float32') for l in range(num_layers)]
# Expand dim to apply broadcasting.
#扩大维数 适应广播
#在anchors的0维处加如一个维度
anchors = np.expand_dims(anchors, 0)
#为什么除以二 因为要使true_boxes中心点和锚框中心点重合
anchor_maxes = anchors / 2.
anchor_mins = -anchor_maxes
#因为每一张图片都有20个存储物体的位置，但有的存不满，那些全是0 ，所以抛弃0的这些位置
valid_mask = boxes_wh[..., 0]>0
for b in range(m):
# Discard zero rows.
#boxes_wh是[m,T,5]维度 valid_mask是[m,T,1]维度
#输出的wh是(k,2)维度的，k是此行所拥有的非零 wh对
wh = boxes_wh[b, valid_mask[b]]
if len(wh)==0: continue
# Expand dim to apply broadcasting.
#扩大维度，适应广播 必须是-2 0不行 ？？？？？？？？？？
wh = np.expand_dims(wh, -2)
#这里的除以2与上面的锚框除以2对应
box_maxes = wh / 2.
box_mins = -box_maxes
#输出的k对wh 每一对 都要和所有的锚框进行比较，得到(k,N,2)维量 N是锚框个数
intersect_mins = np.maximum(box_mins, anchor_mins)
intersect_maxes = np.minimum(box_maxes, anchor_maxes)
intersect_wh = np.maximum(intersect_maxes - intersect_mins, 0.)
intersect_area = intersect_wh[..., 0] * intersect_wh[..., 1]
box_area = wh[..., 0] * wh[..., 1]
anchor_area = anchors[..., 0] * anchors[..., 1]
#iou为1是百分百重合 所以越是靠近1 越好 所有的iou都小于等于1
iou = intersect_area / (box_area + anchor_area - intersect_area)
# Find best anchor for each true box
#为每个真正的盒子找到最佳锚
#得到一个(k,1)或者(1,k)维度的量 。。。k是wh对的数量
best_anchor = np.argmax(iou, axis=-1)
#t代表第几个boxes_wh n是最好锚框的位置
for t, n in enumerate(best_anchor):
#检测层数量的循环
for l in range(num_layers):
#将true_boxes_wh分配到和自己搭配最好的锚框的检测层
if n in anchor_mask[l]:
#i代表宽 j代表高 (i,j)坐标 floor是向下取整
i = np.floor(true_boxes[b,t,0]*grid_shapes[l][1]).astype('int32')
j = np.floor(true_boxes[b,t,1]*grid_shapes[l][0]).astype('int32')
#k代表对应检测层下，锚框所代表的box的位置 是一个不超过2的数 因为每个检测器里只有3个box或锚框
#所以从这里可以得出 一个box对应一个锚框 一共九个 或六个(tiny时)
k = anchor_mask[l].index(n)
#c代表类别
c = true_boxes[b,t, 4].astype('int32')
#将true_box的所有信息填入y_true中 最终用来训练 用在损失函数中
y_true[l][b, j, i, k, 0:4] = true_boxes[b,t, 0:4]
y_true[l][b, j, i, k, 4] = 1
y_true[l][b, j, i, k, 5+c] = 1
return y_true
#在loss中进行应用
def box_iou(b1, b2):
'''Return iou tensor
Parameters
----------
b1: tensor, shape=(i1,...,iN, 4), xywh
b2: tensor, shape=(j, 4), xywh
Returns
-------
iou: tensor, shape=(i1,...,iN, j)
'''#b1的维度是一列boxes
#这里的b1 b2 与 上面的preprocess_true_boxes中的boxes与锚框类似 也是每一个b1中的框都要与b2中的所有框有交并比
# Expand dim to apply broadcasting.
#扩大维度，适应广播
b1 = K.expand_dims(b1, -2)
#取b1的中心点 高和宽
b1_xy = b1[..., :2]
b1_wh = b1[..., 2:4]
b1_wh_half = b1_wh/2.
#得到两点确定一个框b1的这两点
b1_mins = b1_xy - b1_wh_half
b1_maxes = b1_xy + b1_wh_half
# Expand dim to apply broadcasting.
#扩大维度，适应广播
b2 = K.expand_dims(b2, 0)
#取b2的中心点 高和宽
b2_xy = b2[..., :2]
b2_wh = b2[..., 2:4]
b2_wh_half = b2_wh/2.
#得到两点确定一个框b2的这两点
b2_mins = b2_xy - b2_wh_half
b2_maxes = b2_xy + b2_wh_half
#得到交并比
intersect_mins = K.maximum(b1_mins, b2_mins)
intersect_maxes = K.minimum(b1_maxes, b2_maxes)
intersect_wh = K.maximum(intersect_maxes - intersect_mins, 0.)
intersect_area = intersect_wh[..., 0] * intersect_wh[..., 1]
b1_area = b1_wh[..., 0] * b1_wh[..., 1]
b2_area = b2_wh[..., 0] * b2_wh[..., 1]
iou = intersect_area / (b1_area + b2_area - intersect_area)
return iou
#在train里进行运用
def yolo_loss(args, anchors, num_classes, ignore_thresh=.5, print_loss=False):
#参数解释 args包括了yolo_outputs和y_true
'''Return yolo_loss tensor
Parameters
----------
yolo_outputs: list of tensor, the output of yolo_body or tiny_yolo_body
y_true: list of array, the output of preprocess_true_boxes
anchors: array, shape=(N, 2), wh
num_classes: integer
ignore_thresh: float, the iou threshold whether to ignore object confidence loss
Returns
-------
loss: tensor, shape=(1,)
'''
num_layers = len(anchors)//3 # default setting
#区分yolo_outputs和y_true数据值
yolo_outputs = args[:num_layers]
y_true = args[num_layers:]
anchor_mask = [[6,7,8], [3,4,5], [0,1,2]] if num_layers==3 else [[3,4,5], [1,2,3]]
#得到输入图像的大小，例如(416,416)
input_shape = K.cast(K.shape(yolo_outputs[0])[1:3] * 32, K.dtype(y_true[0]))
#得到每个检测框的尺寸 (13,13) (26,26) (52,52)
grid_shapes = [K.cast(K.shape(yolo_outputs[l])[1:3], K.dtype(y_true[0])) for l in range(num_layers)]
loss = 0
#batch size大小 输入图像数量
m = K.shape(yolo_outputs[0])[0] # batch size, tensor
mf = K.cast(m, K.dtype(yolo_outputs[0]))
#对每一个检测层的损失进行叠加
for l in range(num_layers):
#物体标记 是否有物体 为0或1 这里会缩减一个维度 将最后一个维度直接放到倒数第二个维度
object_mask = y_true[l][..., 4:5]
#训练数据的类 是一个向量
true_class_probs = y_true[l][..., 5:]
#所有预测值 这里都被归一化的量 这里的raw_pred(feats)维度是 [m(batch_size), h, w, 3, num_classes + 5]
grid, raw_pred, pred_xy, pred_wh = yolo_head(yolo_outputs[l],
anchors[anchor_mask[l]], num_classes, input_shape, calc_loss=True)
#粘合 得到预测框
pred_box = K.concatenate([pred_xy, pred_wh])
# Darknet raw box to calculate loss.
#得到相对于单个box的xy sigmoid(tx) sigmoid(ty)
raw_true_xy = y_true[l][..., :2]*grid_shapes[l][::-1] - grid
#利用y_true得到tw和th
#有公式bx = sigmoid(tx) + cx, by = sigmoid(ty) + cy, bw = pw*exp(tw), bh = ph*exp(th)， p为锚框
raw_true_wh = K.log(y_true[l][..., 2:4] / anchors[anchor_mask[l]] * input_shape[::-1])
#将标记中为1的保留raw_true_wh 为0的raw_true_wh替换成0 因为log(0)是无穷大
raw_true_wh = K.switch(object_mask, raw_true_wh, K.zeros_like(raw_true_wh)) # avoid log(0)=-inf
#2-bw*bh
box_loss_scale = 2 - y_true[l][...,2:3]*y_true[l][...,3:4]
# Find ignore mask, iterate over each of batch.
#用来 确定 实际没有对象 的confidence的 忽略度
ignore_mask = tf.TensorArray(K.dtype(y_true[0]), size=1, dynamic_size=True)
#将object_mask原来的0和1 换为 False和True
object_mask_bool = K.cast(object_mask, 'bool')
#
def loop_body(b, ignore_mask):
#对每一个输入的图像boxes进行处理，这里会得到所有标记为true的boxes 排成一列
true_box = tf.boolean_mask(y_true[l][b,...,0:4], object_mask_bool[b,...,0])
#pred_box[b]是(h,w,3,4) true_box是(j,4)
iou = box_iou(pred_box[b], true_box)
#找到最大的iou
best_iou = K.max(iou, axis=-1)
#在b的位置写入 cast 如果最大iou比忽略值小则为true则损失会随之增大
#反之则为false损失将不会考虑没有对象的boxes 有对象的可能极大 会将全部精力放在有对象的box上
ignore_mask = ignore_mask.write(b, K.cast(best_iou<ignore_thresh, K.dtype(true_box)))
return b+1, ignore_mask
#将batch_size 循环确定ignore_mask 此时ignore_mask大小为 m
_, ignore_mask = K.control_flow_ops.while_loop(lambda b,*args: b<m, loop_body, [0, ignore_mask])
#stack()为什么都变成0了 懂了 不能用读取 就是栈 读了就没有，所以，不用read就行
ignore_mask = ignore_mask.stack()
ignore_mask = K.expand_dims(ignore_mask, -1)
#最终确定 没有对象 的confidence的 忽略度
# K.binary_crossentropy is helpful to avoid exp overflow.
#真正的损失计算，包括中心点 高宽 置信度 类别 与论文中的损失函数对应
xy_loss = object_mask * box_loss_scale * K.binary_crossentropy(raw_true_xy, raw_pred[...,0:2], from_logits=True)
wh_loss = object_mask * box_loss_scale * 0.5 * K.square(raw_true_wh-raw_pred[...,2:4])
confidence_loss = object_mask * K.binary_crossentropy(object_mask, raw_pred[...,4:5], from_logits=True)+ \
(1-object_mask) * K.binary_crossentropy(object_mask, raw_pred[...,4:5], from_logits=True) * ignore_mask
class_loss = object_mask * K.binary_crossentropy(true_class_probs, raw_pred[...,5:], from_logits=True)
#将所有损失相加，并且算平均，得到最终损失
xy_loss = K.sum(xy_loss) / mf
wh_loss = K.sum(wh_loss) / mf
confidence_loss = K.sum(confidence_loss) / mf
class_loss = K.sum(class_loss) / mf
loss += xy_loss + wh_loss + confidence_loss + class_loss
if print_loss:
loss = tf.Print(loss, [loss, xy_loss, wh_loss, confidence_loss, class_loss, K.sum(ignore_mask)], message='loss: ')
return loss
utils.py
"""Miscellaneous utility functions."""
#各种各样的实用函数。
from functools import reduce
from PIL import Image
import numpy as np
from matplotlib.colors import rgb_to_hsv, hsv_to_rgb
#在model中进行应用
def compose(*funcs):
"""Compose arbitrarily many functions, evaluated left to right.
Reference: https://mathieularose.com/function-composition-in-python/
"""
#组合任意多个函数，从左到右求值。
# return lambda x: reduce(lambda v, f: f(v), funcs, x)
#进行卷积层组合的函数，例：[a,b,c,d,e]代表五个函数 e(d(c(b(a(函数自变量)))))
if funcs:
return reduce(lambda f, g: lambda *a, **kw: g(f(*a, **kw)), funcs)
else:
raise ValueError('Composition of empty sequence not supported.')
#在yolo中进行应用
def letterbox_image(image, size):
'''resize image with unchanged aspect ratio using padding'''
#得到真实图片的宽高 例：(400,600)
iw, ih = image.size
#得到需要缩放到的宽高 例：(200,200)
w, h = size
#得到需要缩放的比例，找最小的意义在于，可以将wh中大的缩减下来 例：1/3
scale = min(w/iw, h/ih)
#得到按比例scale缩放的高和宽 例： nw 400/3约为133 nh 200
nw = int(iw*scale)
nh = int(ih*scale)
#将图像缩小 scale 倍 例：(133,200)
image = image.resize((nw,nh), Image.BICUBIC)
#得到一个填充色的面板 大小为 size 最终需要的大小
new_image = Image.new('RGB', size, (128,128,128))
#将(nw,nh)的图像平铺到这个面板上，左上角对应的点是((w-nw)//2, (h-nh)//2) 例：(73//2,0)
new_image.paste(image, ((w-nw)//2, (h-nh)//2))
return new_image
#随机数，应用在get_random_data中
def rand(a=0, b=1):
return np.random.rand()*(b-a) + a
#在train中进行应用
def get_random_data(annotation_line, input_shape, random=True, max_boxes=20, jitter=.3, hue=.1, sat=1.5, val=1.5, proc_img=True):
#proc_img不知何用？？？？？？？？？？？？？？？？？？？？？？？？？
'''random preprocessing for real-time data augmentation'''
#得到图像存储地址和图像 以及box
line = annotation_line.split()
image = Image.open(line[0])
iw, ih = image.size
h, w = input_shape
box = np.array([np.array(list(map(int,box.split(',')))) for box in line[1:]])
#得到原图像和boxes 对应的缩放后的图像和boxes，不生成新图
if not random:
# resize image
scale = min(w/iw, h/ih)
nw = int(iw*scale)
nh = int(ih*scale)
dx = (w-nw)//2
dy = (h-nh)//2
image_data=0
if proc_img:
image = image.resize((nw,nh), Image.BICUBIC)
new_image = Image.new('RGB', (w,h), (128,128,128))
new_image.paste(image, (dx, dy))
image_data = np.array(new_image)/255.
# correct boxes
box_data = np.zeros((max_boxes,5))
if len(box)>0:
np.random.shuffle(box)
if len(box)>max_boxes: box = box[:max_boxes]
box[:, [0,2]] = box[:, [0,2]]*scale + dx
box[:, [1,3]] = box[:, [1,3]]*scale + dy
box_data[:len(box)] = box
return image_data, box_data
# resize image 调整原图大小
new_ar = w/h * rand(1-jitter,1+jitter)/rand(1-jitter,1+jitter)
scale = rand(.25, 2)
if new_ar < 1:
nh = int(scale*h)
nw = int(nh*new_ar)
else:
nw = int(scale*w)
nh = int(nw/new_ar)
image = image.resize((nw,nh), Image.BICUBIC)
# place image 平移
dx = int(rand(0, w-nw))
dy = int(rand(0, h-nh))
new_image = Image.new('RGB', (w,h), (128,128,128))
new_image.paste(image, (dx, dy))
image = new_image
# flip image or not 镜像翻转否
flip = rand()<.5
if flip: image = image.transpose(Image.FLIP_LEFT_RIGHT)
# distort image 将图像进行扭曲
hue = rand(-hue, hue)
sat = rand(1, sat) if rand()<.5 else 1/rand(1, sat)
val = rand(1, val) if rand()<.5 else 1/rand(1, val)
x = rgb_to_hsv(np.array(image)/255.)
x[..., 0] += hue
x[..., 0][x[..., 0]>1] -= 1
x[..., 0][x[..., 0]<0] += 1
x[..., 1] *= sat
x[..., 2] *= val
x[x>1] = 1
x[x<0] = 0
image_data = hsv_to_rgb(x) # numpy array, 0 to 1
# correct boxes 得到随机图像以后的 缩放比例后 图像与boxes 注意：在经过平移 旋转 缩放 后boxes的参数要变
box_data = np.zeros((max_boxes,5))
if len(box)>0:
np.random.shuffle(box)
box[:, [0,2]] = box[:, [0,2]]*nw/iw + dx
box[:, [1,3]] = box[:, [1,3]]*nh/ih + dy
if flip: box[:, [0,2]] = w - box[:, [2,0]]
box[:, 0:2][box[:, 0:2]<0] = 0
box[:, 2][box[:, 2]>w] = w
box[:, 3][box[:, 3]>h] = h
box_w = box[:, 2] - box[:, 0]
box_h = box[:, 3] - box[:, 1]
box = box[np.logical_and(box_w>1, box_h>1)] # discard invalid box
if len(box)>max_boxes: box = box[:max_boxes]
box_data[:len(box)] = box
return image_data, box_data
train.py
"""
Retrain the YOLO model for your own dataset.
"""
import numpy as np
import keras.backend as K
from keras.layers import Input, Lambda
from keras.models import Model
from keras.optimizers import Adam
from keras.callbacks import TensorBoard, ModelCheckpoint, ReduceLROnPlateau, EarlyStopping
from yolo3.model import preprocess_true_boxes, yolo_body, tiny_yolo_body, yolo_loss
from yolo3.utils import get_random_data
#主函数
def _main():
annotation_path = 'train.txt'
log_dir = 'logs/000/'
classes_path = 'model_data/voc_classes.txt'
anchors_path = 'model_data/yolo_anchors.txt'
#得到类名，类总数，锚框
class_names = get_classes(classes_path)
num_classes = len(class_names)
anchors = get_anchors(anchors_path)
input_shape = (416,416) # multiple of 32, hw
#判断是否用tiny_body
is_tiny_version = len(anchors)==6 # default setting
if is_tiny_version:
#建立模型
model = create_tiny_model(input_shape, anchors, num_classes,
freeze_body=2, weights_path='model_data/tiny_yolo_weights.h5')
else:
#建立模型
model = create_model(input_shape, anchors, num_classes,
freeze_body=2, weights_path='model_data/yolo_weights.h5') # make sure you know what you freeze
#设置日志路径
logging = TensorBoard(log_dir=log_dir)
#设置检查点，3回检查一次，按验证集损失对比 保留最号的
checkpoint = ModelCheckpoint(log_dir + 'ep{epoch:03d}-loss{loss:.3f}-val_loss{val_loss:.3f}.h5',
monitor='val_loss', save_weights_only=True, save_best_only=True, period=3)
#学习衰减率 monitor 监测值 factor 缩放学习率的值，学习率将以lr = lr*factor的形式被减少
#当patience个epoch过去而模型性能不提升时，学习率减少的动作会被触发
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=3, verbose=1)
#早停 min_delta增大或减小的阈值，只有大于这个部分才算作improvement
#能够容忍多少个epoch内都没有improvement
early_stopping = EarlyStopping(monitor='val_loss', min_delta=0, patience=10, verbose=1)
#分给验证集的比例
val_split = 0.1
with open(annotation_path) as f:
lines = f.readlines()
np.random.seed(10101)
np.random.shuffle(lines)
np.random.seed(None)
#分配训练集和验证集的数量
num_val = int(len(lines)*val_split)
num_train = len(lines) - num_val
# Train with frozen layers first, to get a stable loss.
# Adjust num epochs to your dataset. This step is enough to obtain a not bad model.
#先用冰冻层进行训练，得到稳定的损失。
#调整num epoch到您的数据集。这一步足以获得一个不错的模型。
if True:
#设置优化器 loss设置不太懂？？？？？？？？？？？？？？？？？？？？？？？？？？？？？？？？？？？？/
model.compile(optimizer=Adam(lr=1e-3), loss={
# use custom yolo_loss Lambda layer.
'yolo_loss': lambda y_true, y_pred: y_pred})
batch_size = 32
print('Train on {} samples, val on {} samples, with batch size {}.'.format(num_train, num_val, batch_size))
#设置训练器
model.fit_generator(data_generator_wrapper(lines[:num_train], batch_size, input_shape, anchors, num_classes),
steps_per_epoch=max(1, num_train//batch_size),
validation_data=data_generator_wrapper(lines[num_train:], batch_size, input_shape, anchors, num_classes),
validation_steps=max(1, num_val//batch_size),
epochs=50,
initial_epoch=0,
callbacks=[logging, checkpoint])
model.save_weights(log_dir + 'trained_weights_stage_1.h5')
# Unfreeze and continue training, to fine-tune.
# Train longer if the result is not good.
#解冻并继续训练，以进行微调。
#如果效果不好，就多训练。
if True:
#将模型层全部解冻
for i in range(len(model.layers)):
model.layers[i].trainable = True
model.compile(optimizer=Adam(lr=1e-4), loss={'yolo_loss': lambda y_true, y_pred: y_pred}) # recompile to apply the change
print('Unfreeze all of the layers.')
batch_size = 32 # note that more GPU memory is required after unfreezing the body
print('Train on {} samples, val on {} samples, with batch size {}.'.format(num_train, num_val, batch_size))
model.fit_generator(data_generator_wrapper(lines[:num_train], batch_size, input_shape, anchors, num_classes),
steps_per_epoch=max(1, num_train//batch_size),
validation_data=data_generator_wrapper(lines[num_train:], batch_size, input_shape, anchors, num_classes),
validation_steps=max(1, num_val//batch_size),
epochs=100,
initial_epoch=50,
callbacks=[logging, checkpoint, reduce_lr, early_stopping])
model.save_weights(log_dir + 'trained_weights_final.h5')
# Further training if needed.
#如有需要，可接受进一步训练。
#在_main中应用，得到类名一个列表
def get_classes(classes_path):
'''loads the classes'''
with open(classes_path) as f:
class_names = f.readlines()
class_names = [c.strip() for c in class_names]
return class_names
#在_main中应用，得到锚框(9,2)维度 或者 (6,2)维度
def get_anchors(anchors_path):
'''loads the anchors from a file'''
with open(anchors_path) as f:
anchors = f.readline()
anchors = [float(x) for x in anchors.split(',')]
return np.array(anchors).reshape(-1, 2)
#在_train 中应用
def create_model(input_shape, anchors, num_classes, load_pretrained=True, freeze_body=2,
weights_path='model_data/yolo_weights.h5'):
'''create the training model'''
#参数解释：load_pretrained 是否下载预训练，
#freeze_body 是否冻结预训练的层 1代表冻结darknet 2代表冻结yolo_body除3个检测层的其他层
K.clear_session() # get a new session
#占位参数 image_input 在经过Input后 shape会变成四个维度 多加一个batch_size维度 (None,None,None,3)
image_input = Input(shape=(None, None, 3))
h, w = input_shape#(416,416)
num_anchors = len(anchors)
#得到占位 y_true
y_true = [Input(shape=(h//{0:32, 1:16, 2:8}[l], w//{0:32, 1:16, 2:8}[l], \
num_anchors//3, num_classes+5)) for l in range(3)]
#建立模型本体
model_body = yolo_body(image_input, num_anchors//3, num_classes)
print('Create YOLOv3 model with {} anchors and {} classes.'.format(num_anchors, num_classes))
#是否加载预训练权重
if load_pretrained:
model_body.load_weights(weights_path, by_name=True, skip_mismatch=True)
print('Load weights {}.'.format(weights_path))
#是否冻结 代表不训练这些层
if freeze_body in [1, 2]:
# Freeze darknet53 body or freeze all but 3 output layers.
#冻结暗网53的身体或冻结除3个输出层以外的所有层。
num = (185, len(model_body.layers)-3)[freeze_body-1]
for i in range(num): model_body.layers[i].trainable = False
print('Freeze the first {} layers of total {} layers.'.format(num, len(model_body.layers)))
#设置损失函数 *的作用是进行迭代，将列表中的元素按第一维度迭代处出来，实际就是减少一个维度
model_loss = Lambda(yolo_loss, output_shape=(1,), name='yolo_loss',
arguments={'anchors': anchors, 'num_classes': num_classes, 'ignore_thresh': 0.5})(
[*model_body.output, *y_true])
#将Model 中 [model_body.input, *y_true]设置为输入 model_loss设置为输出
model = Model([model_body.input, *y_true], model_loss)
return model
#在_train 中应用 与create_model相似
def create_tiny_model(input_shape, anchors, num_classes, load_pretrained=True, freeze_body=2,
weights_path='model_data/tiny_yolo_weights.h5'):
'''create the training model, for Tiny YOLOv3'''
K.clear_session() # get a new session
image_input = Input(shape=(None, None, 3))
h, w = input_shape
num_anchors = len(anchors)
y_true = [Input(shape=(h//{0:32, 1:16}[l], w//{0:32, 1:16}[l], \
num_anchors//2, num_classes+5)) for l in range(2)]
model_body = tiny_yolo_body(image_input, num_anchors//2, num_classes)
print('Create Tiny YOLOv3 model with {} anchors and {} classes.'.format(num_anchors, num_classes))
if load_pretrained:
model_body.load_weights(weights_path, by_name=True, skip_mismatch=True)
print('Load weights {}.'.format(weights_path))
if freeze_body in [1, 2]:
# Freeze the darknet body or freeze all but 2 output layers.
num = (20, len(model_body.layers)-2)[freeze_body-1]
for i in range(num): model_body.layers[i].trainable = False
print('Freeze the first {} layers of total {} layers.'.format(num, len(model_body.layers)))
model_loss = Lambda(yolo_loss, output_shape=(1,), name='yolo_loss',
arguments={'anchors': anchors, 'num_classes': num_classes, 'ignore_thresh': 0.7})(
[*model_body.output, *y_true])
model = Model([model_body.input, *y_true], model_loss)
return model
def data_generator(annotation_lines, batch_size, input_shape, anchors, num_classes):
'''data generator for fit_generator'''
#读取所有的数据行
n = len(annotation_lines)
i = 0
while True:
image_data = []
box_data = []
#对batch_size中的每一行数据进行处理
for b in range(batch_size):
#洗牌 打乱
if i==0:
np.random.shuffle(annotation_lines)
#得到经过调整后的数据
image, box = get_random_data(annotation_lines[i], input_shape, random=True)
image_data.append(image)
box_data.append(box)
#确保i在n里 也保证i一直增大
i = (i+1) % n
image_data = np.array(image_data)
box_data = np.array(box_data)
#对输入进行预处理 得到y_true
y_true = preprocess_true_boxes(box_data, input_shape, anchors, num_classes)
#yield作用是输出后继续循环
yield [image_data, *y_true], np.zeros(batch_size)
def data_generator_wrapper(annotation_lines, batch_size, input_shape, anchors, num_classes):
n = len(annotation_lines)
if n==0 or batch_size<=0: return None
return data_generator(annotation_lines, batch_size, input_shape, anchors, num_classes)
if __name__ == '__main__':
_main()
yolo.py
# -*- coding: utf-8 -*-
"""
Class definition of YOLO_v3 style detection model on image and video
图像和视频YOLO_v3风格检测模型的类定义
"""
import colorsys
import os
from timeit import default_timer as timer
import numpy as np
from keras import backend as K
from keras.models import load_model
from keras.layers import Input
from PIL import Image, ImageFont, ImageDraw
from yolo3.model import yolo_eval, yolo_body, tiny_yolo_body
from yolo3.utils import letterbox_image
import os
from keras.utils import multi_gpu_model
class YOLO(object):
_defaults = {
"model_path": 'model_data/yolo.h5',
"anchors_path": 'model_data/yolo_anchors.txt',
"classes_path": 'model_data/coco_classes.txt',
"score" : 0.3,
"iou" : 0.45,
"model_image_size" : (416, 416),
"gpu_num" : 1,
}#路径参数
@classmethod
def get_defaults(cls, n):
if n in cls._defaults:
return cls._defaults[n]
else:
return "Unrecognized attribute name '" + n + "'"
#无法识别的属性名称
def __init__(self, **kwargs):
self.__dict__.update(self._defaults) # set up default values
self.__dict__.update(kwargs) # and update with user overrides
self.class_names = self._get_class()
self.anchors = self._get_anchors()
self.sess = K.get_session()
self.boxes, self.scores, self.classes = self.generate()
#获得类名
def _get_class(self):
classes_path = os.path.expanduser(self.classes_path)#将 ~ 路径展开
with open(classes_path) as f:
class_names = f.readlines()
class_names = [c.strip() for c in class_names]
return class_names
#获得锚框
def _get_anchors(self):
anchors_path = os.path.expanduser(self.anchors_path)
with open(anchors_path) as f:
anchors = f.readline()
anchors = [float(x) for x in anchors.split(',')]
return np.array(anchors).reshape(-1, 2)
#获得模型
def generate(self):
model_path = os.path.expanduser(self.model_path)
assert model_path.endswith('.h5'), 'Keras model or weights must be a .h5 file.'
# Load model, or construct model and load weights.
#负载模型，或构造模型和负载权重。
num_anchors = len(self.anchors)
num_classes = len(self.class_names)
is_tiny_version = num_anchors==6 # default setting
try:
self.yolo_model = load_model(model_path, compile=False)# 读取网络、权重
except:
self.yolo_model = tiny_yolo_body(Input(shape=(None,None,3)), num_anchors//2, num_classes) \
if is_tiny_version else yolo_body(Input(shape=(None,None,3)), num_anchors//3, num_classes)
self.yolo_model.load_weights(self.model_path) # make sure model, anchors and classes match
else:
assert self.yolo_model.layers[-1].output_shape[-1] == \
num_anchors/len(self.yolo_model.output) * (num_classes + 5), \
'Mismatch between model and given anchor and class sizes'#确保模型与给定的锚点和类大小匹配
print('{} model, anchors, and classes loaded.'.format(model_path))
# Generate colors for drawing bounding boxes.为绘制边框生成颜色。
hsv_tuples = [(x / len(self.class_names), 1., 1.)
for x in range(len(self.class_names))]
self.colors = list(map(lambda x: colorsys.hsv_to_rgb(*x), hsv_tuples))
self.colors = list(
map(lambda x: (int(x[0] * 255), int(x[1] * 255), int(x[2] * 255)),
self.colors))#几种类型就几种边框去画
np.random.seed(10101) # Fixed seed for consistent colors across runs.固定的种子一致的颜色跨运行。
np.random.shuffle(self.colors) # Shuffle colors to decorrelate adjacent classes.打乱颜色以消除相邻类的关联。
np.random.seed(None) # Reset seed to default.将种子重置为默认值。
# Generate output tensor targets for filtered bounding boxes.为过滤过的边界框生成输出张量目标。
self.input_image_shape = K.placeholder(shape=(2, ))#实例化一个占位符张量并返回它。
if self.gpu_num>=2:
self.yolo_model = multi_gpu_model(self.yolo_model, gpus=self.gpu_num)
boxes, scores, classes = yolo_eval(self.yolo_model.output, self.anchors,
len(self.class_names), self.input_image_shape,
score_threshold=self.score, iou_threshold=self.iou)#这个输入图像形状是什么 是否是写错了
return boxes, scores, classes
def detect_image(self, image):
start = timer()
if self.model_image_size != (None, None):
assert self.model_image_size[0]%32 == 0, 'Multiples of 32 required'
assert self.model_image_size[1]%32 == 0, 'Multiples of 32 required'
boxed_image = letterbox_image(image, tuple(reversed(self.model_image_size)))
#我们的model size是高宽 letter用的是 宽高 所以要逆序
else:
new_image_size = (image.width - (image.width % 32),
image.height - (image.height % 32))
boxed_image = letterbox_image(image, new_image_size)
image_data = np.array(boxed_image, dtype='float32')
print(image_data.shape)
image_data /= 255.
image_data = np.expand_dims(image_data, 0) # Add batch dimension.批量添加维度
out_boxes, out_scores, out_classes = self.sess.run(
[self.boxes, self.scores, self.classes],
feed_dict={
self.yolo_model.input: image_data,
self.input_image_shape: [image.size[1], image.size[0]],
K.learning_phase(): 0
})#这个时候就是喂入input image shape的时候
print('Found {} boxes for {}'.format(len(out_boxes), 'img'))
font = ImageFont.truetype(font='font/FiraMono-Medium.otf',
size=np.floor(3e-2 * image.size[1] + 0.5).astype('int32'))
thickness = (image.size[0] + image.size[1]) // 300
for i, c in reversed(list(enumerate(out_classes))):
predicted_class = self.class_names[c]
box = out_boxes[i]
score = out_scores[i]
label = '{} {:.2f}'.format(predicted_class, score)
draw = ImageDraw.Draw(image)
label_size = draw.textsize(label, font)
top, left, bottom, right = box
top = max(0, np.floor(top + 0.5).astype('int32'))
left = max(0, np.floor(left + 0.5).astype('int32'))
bottom = min(image.size[1], np.floor(bottom + 0.5).astype('int32'))
right = min(image.size[0], np.floor(right + 0.5).astype('int32'))
print(label, (left, top), (right, bottom))
if top - label_size[1] >= 0:
text_origin = np.array([left, top - label_size[1]])
else:
text_origin = np.array([left, top + 1])
# My kingdom for a good redistributable image drawing library.
for i in range(thickness):
draw.rectangle(
[left + i, top + i, right - i, bottom - i],
outline=self.colors[c])
draw.rectangle(
[tuple(text_origin), tuple(text_origin + label_size)],
fill=self.colors[c])
draw.text(text_origin, label, fill=(0, 0, 0), font=font)
del draw
end = timer()
print(end - start)
return image
def close_session(self):
self.sess.close()
def detect_video(yolo, video_path, output_path=""):
import cv2
vid = cv2.VideoCapture(video_path)
if not vid.isOpened():
raise IOError("Couldn't open webcam or video")
video_FourCC = int(vid.get(cv2.CAP_PROP_FOURCC))
video_fps = vid.get(cv2.CAP_PROP_FPS)
video_size = (int(vid.get(cv2.CAP_PROP_FRAME_WIDTH)),
int(vid.get(cv2.CAP_PROP_FRAME_HEIGHT)))
isOutput = True if output_path != "" else False
if isOutput:
print("!!! TYPE:", type(output_path), type(video_FourCC), type(video_fps), type(video_size))
out = cv2.VideoWriter(output_path, video_FourCC, video_fps, video_size)
accum_time = 0
curr_fps = 0
fps = "FPS: ??"
prev_time = timer()
while True:
return_value, frame = vid.read()
image = Image.fromarray(frame)
image = yolo.detect_image(image)
result = np.asarray(image)
curr_time = timer()
exec_time = curr_time - prev_time
prev_time = curr_time
accum_time = accum_time + exec_time
curr_fps = curr_fps + 1
if accum_time > 1:
accum_time = accum_time - 1
fps = "FPS: " + str(curr_fps)
curr_fps = 0
cv2.putText(result, text=fps, org=(3, 15), fontFace=cv2.FONT_HERSHEY_SIMPLEX,
fontScale=0.50, color=(255, 0, 0), thickness=2)
cv2.namedWindow("result", cv2.WINDOW_NORMAL)
cv2.imshow("result", result)
if isOutput:
out.write(result)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
yolo.close_session()
初学，如有侵权或错误，请指出，定及时修改
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
