NMS 与 Soft-NMS

NMS

算法流程

假设有有一个候选的boxes的集合B和其对应的scores集合S：

1. 从B中找出分数最高box，记为M；

2. 将M对应的box从B中删除；

3. 将删除的box添加到集合D中；

4. 从B中删除与M对应的box重叠区域大于阈值$N_t$的其他框；

5. 重复上述步骤1-4。
   

code

# https://github.com/rbgirshick/py-faster-rcnn/blob/master/lib/nms/py_cpu_nms.py 
import numpy as np

def py_cpu_nms(dets, thresh):
    """Pure Python NMS baseline."""
    x1 = dets[:, 0]
    y1 = dets[:, 1]
    x2 = dets[:, 2]
    y2 = dets[:, 3]
    scores = dets[:, 4]

    areas = (x2 - x1 + 1) * (y2 - y1 + 1)
    order = scores.argsort()[::-1]

    keep = []
    while order.size > 0:
        i = order[0]
        keep.append(i)
        xx1 = np.maximum(x1[i], x1[order[1:]])
        yy1 = np.maximum(y1[i], y1[order[1:]])
        xx2 = np.minimum(x2[i], x2[order[1:]])
        yy2 = np.minimum(y2[i], y2[order[1:]])

        w = np.maximum(0.0, xx2 - xx1 + 1)
        h = np.maximum(0.0, yy2 - yy1 + 1)
        inter = w * h
        ovr = inter / (areas[i] + areas[order[1:]] - inter)

        inds = np.where(ovr <= thresh)[0]
        order = order[inds + 1]

    return keep
if __name__ == '__main__':
    boxes = np.array([[100, 100, 150, 168, 0.63],
                      [166, 70, 312, 190, 0.55],
                      [221, 250, 389, 500, 0.79],
                      [12, 190, 300, 399, 0.9],
                      [28, 130, 134, 302, 0.3]])
    thresh = 0.1
    keep = py_cpu_nms(boxes, thresh)
    print(keep)

Soft-NMS

Soft-NMS – Improving Object Detection With One Line of Code

NMS存在的问题

NMS在密集遮挡场景会出现很大的问题，漏检很多，例如下图，红色框和绿色框是当前的检测结果，二者的得分分别是0.95和0.80。如果按照传统的NMS进行处理，首先选中得分最高的红色框，然后绿色框就会因为与之重叠面积过大而被删掉。另外，NMS的阈值也不太容易确定，设小了会出现下图的情况（绿色框因为和红色框重叠面积较大而被删掉），设置过高又容易增大误检。

Soft-NMS的改进

Soft-NMS的思路：不要粗鲁地删除所有IOU大于阈值的框，而是降低其置信度。
Soft-NMS的做法：$M$为当前得分最高框，$b_i$ 为待处理框，$b_i$ 和$M$的$IOU$越大，$b_i$ 的得分$s_i$ 就下降的越厉害。

分数的降低方式可以选择线性加权或高斯加权

code

# https://github.com/bharatsingh430/soft-nms
# coding:utf-8
import numpy as np
def soft_nms(boxes, sigma=0.5, Nt=0.1, threshold=0.001, method=1):
    N = boxes.shape[0]
    pos = 0
    maxscore = 0
    maxpos = 0

    for i in range(N):
        maxscore = boxes[i, 4]
        maxpos = i

        tx1 = boxes[i,0]
        ty1 = boxes[i,1]
        tx2 = boxes[i,2]
        ty2 = boxes[i,3]
        ts = boxes[i,4]

        pos = i + 1
    # get max box
        while pos < N:
            if maxscore < boxes[pos, 4]:
                maxscore = boxes[pos, 4]
                maxpos = pos
            pos = pos + 1

    # add max box as a detection
        boxes[i,0] = boxes[maxpos,0]
        boxes[i,1] = boxes[maxpos,1]
        boxes[i,2] = boxes[maxpos,2]
        boxes[i,3] = boxes[maxpos,3]
        boxes[i,4] = boxes[maxpos,4]

    # swap ith box with position of max box
        boxes[maxpos,0] = tx1
        boxes[maxpos,1] = ty1
        boxes[maxpos,2] = tx2
        boxes[maxpos,3] = ty2
        boxes[maxpos,4] = ts

        tx1 = boxes[i,0]
        ty1 = boxes[i,1]
        tx2 = boxes[i,2]
        ty2 = boxes[i,3]
        ts = boxes[i,4]

        pos = i + 1
    # NMS iterations, note that N changes if detection boxes fall below threshold
        while pos < N:
            x1 = boxes[pos, 0]
            y1 = boxes[pos, 1]
            x2 = boxes[pos, 2]
            y2 = boxes[pos, 3]
            s = boxes[pos, 4]

            area = (x2 - x1 + 1) * (y2 - y1 + 1)
            iw = (min(tx2, x2) - max(tx1, x1) + 1)
            if iw > 0:
                ih = (min(ty2, y2) - max(ty1, y1) + 1)
                if ih > 0:
                    ua = float((tx2 - tx1 + 1) * (ty2 - ty1 + 1) + area - iw * ih)
                    ov = iw * ih / ua #iou between max box and detection box

                    if method == 1: # linear
                        if ov > Nt:
                            weight = 1 - ov
                        else:
                            weight = 1
                    elif method == 2: # gaussian
                        weight = np.exp(-(ov * ov)/sigma)
                    else: # original NMS
                        if ov > Nt:
                            weight = 0
                        else:
                            weight = 1

                    boxes[pos, 4] = weight*boxes[pos, 4]
                    print(boxes[:, 4])

            # if box score falls below threshold, discard the box by swapping with last box
            # update N
                    if boxes[pos, 4] < threshold:
                        boxes[pos,0] = boxes[N-1, 0]
                        boxes[pos,1] = boxes[N-1, 1]
                        boxes[pos,2] = boxes[N-1, 2]
                        boxes[pos,3] = boxes[N-1, 3]
                        boxes[pos,4] = boxes[N-1, 4]
                        N = N - 1
                        pos = pos - 1

            pos = pos + 1
    keep = [i for i in range(N)]
    return keep
if __name__ == '__main__':
	boxes = np.array([[100, 100, 150, 168, 0.63],
	                             [166, 70, 312, 190, 0.55],
	                             [221, 250, 389, 500, 0.79],
	                             [12, 190, 300, 399, 0.9],
	                             [28, 130, 134, 302, 0.3]])
	keep = soft_nms(boxes)
	print(keep)