【论文笔记】PointWeb: Enhancing Local Neighborhood Features for Point Cloud Processing

1. 摘要部分：本文的贡献+实验结果

本文的贡献

1. This paper presents PointWeb, a new approach to extract contextual features from local neighborhood in a point cloud. Unlike previous work, we densely connect each point with every other in a local neighborhood, aiming to specify feature of each point based on the local region characteristics for better representing the region.
2. A novel module, namely Adaptive Feature Adjustment (AFA) module, is presented to find the interaction between points. For each local region, an impact map carrying element-wise impact between point pairs is applied to the feature difference map.
   Each feature is then pulled or pushed by other features in the same region according to the adaptively learned impact indicators. The adjusted features are well encoded with region information, and thus benefit the point cloud recognition tasks, such as point cloud segmentation and classification.

实验结果

Experimental results show that our model outperforms the state-of-the-arts on both semantic segmentation and shape classification datasets.

2. 论文采用的AFA模块方法

AFA模块通过利用每个点对之间关系来加强训练效果，两句话总结AFA干了什么：

1. AFA就是在原point features上+局部的上下文信息
2. 这个局部的上下文信息是Impact Function 和Relation Function 的乘积求和。Impact Function代表将两个point features间关系通过MLP求出的weight，Relation Function代表两个point features间的关系，这两个point features间关系实际上的实现都是两point feature求差。
   

本文的总体目标：

Exploring the relationship among points in a local region is the focus of this paper. Particularly, after we extract
pointwise features (or point features, for short) using a neural network, further aggregating these local features helps
improve the point cloud recognition quality for the tasks of semantic segmentation and classification.

Adaptive Feature Adjustment (AFA) Module

Given region R and its feature set

we first formulate the adaptive feature adjustment (AFA) module to enhance the point features in F by learning the contextual information in local neighborhood as

Impact Function fimp

We use MLP to calculate the impact function fimp, as illustrated in Fig. 3. It is formulated as

Relation Function frel

3.AFA部分关键代码段

class _AFAModule(nn.Module):
    def __init__(self, mlp, use_softmax=False):
        r"""
        :param mlp: mlp for learning weight
               mode: transformation or aggregation
        """
        super().__init__()
        self.mlp = mlp
        self.use_softmax = use_softmax

    def forward(self, feature: torch.Tensor) -> torch.Tensor:
        r"""
        Parameters
        ----------
        features : torch.Tensor
            (B, C, N, M) or (B, C, N)
        Returns
        -------
        new_features : torch.Tensor
            transformation: (B, C, N, M) or (B, C, N)
            aggregation: (B, C, N) or (B, C)
        """
        B, C, N, M = feature.size() #batch channel numberpoints n_points_local_region 先假设channel=3 只有xyz,那这里feature就是grouped_xyz
        feature = feature.transpose(1, 2).contiguous().view(B * N, C, M, 1).repeat(1, 1, 1, M)  # (BN, C, M, M)
        #difference map
        feature = feature - feature.transpose(2, 3).contiguous() + torch.mul(feature, torch.eye(M).view(1, 1, M, M).cuda())  # (BN, C, M, M)
        #impact map
        weight = self.mlp(feature)
        if self.use_softmax:
            weight = F.softmax(weight, -1)
        feature = (feature * weight).sum(-1).view(B, N, C, M).transpose(1, 2).contiguous()  # (B, C, N, M)
        return feature

源代码中impact function部分用到的MLP是：

class SharedMLP(nn.Sequential):

    def __init__(
            self,
            args: List[int],
            *,
            bn: bool = False,
            activation=nn.ReLU(inplace=True),
            preact: bool = False,
            first: bool = False,
            name: str = ""
    ):
        super().__init__()

        for i in range(len(args) - 1):
            self.add_module(
                name + 'layer{}'.format(i),
                Conv2d(
                    args[i],
                    args[i + 1],
                    bn=(not first or not preact or (i != 0)) and bn,
                    activation=activation
                    if (not first or not preact or (i != 0)) else None,
                    preact=preact
                )
            )
            
class _ConvBase(nn.Sequential):

    def __init__(
            self,
            in_size,
            out_size,
            kernel_size,
            stride,
            padding,
            activation,
            bn,
            init,
            conv=None,
            batch_norm=None,
            bias=True,
            preact=False,
            name=""
    ):
        super().__init__()

        bias = bias and (not bn)
        conv_unit = conv(
            in_size,
            out_size,
            kernel_size=kernel_size,
            stride=stride,
            padding=padding,
            bias=bias
        )
        init(conv_unit.weight)
        if bias:
            nn.init.constant_(conv_unit.bias, 0)

        if bn:
            if not preact:
                bn_unit = batch_norm(out_size)
            else:
                bn_unit = batch_norm(in_size)

        if preact:
            if bn:
                self.add_module(name + 'bn', bn_unit)

            if activation is not None:
                self.add_module(name + 'activation', activation)

        self.add_module(name + 'conv', conv_unit)

        if not preact:
            if bn:
                self.add_module(name + 'bn', bn_unit)

            if activation is not None:
                self.add_module(name + 'activation', activation)
                
class Conv2d(_ConvBase):

    def __init__(
            self,
            in_size: int,
            out_size: int,
            *,
            kernel_size: Tuple[int, int] = (1, 1),
            stride: Tuple[int, int] = (1, 1),
            padding: Tuple[int, int] = (0, 0),
            activation=nn.ReLU(inplace=True),
            bn: bool = False,
            init=nn.init.kaiming_normal_,
            bias: bool = True,
            preact: bool = False,
            name: str = ""
    ):
        super().__init__(
            in_size,
            out_size,
            kernel_size,
            stride,
            padding,
            activation,
            bn,
            init,
            conv=nn.Conv2d,
            batch_norm=BatchNorm2d,
            bias=bias,
            preact=preact,
            name=name
        )                
            
self.mlp=[16,16] ##参数16来源于源代码示例

mlp_tmp = pt_util.SharedMLP(c + self.mlp, bn=True) #c代表输入特征通道
mlp_tmp.add_module('weight', (pt_util.SharedMLP([self.mlp[-1], c], bn=False, activation=None)))

即，pointweb源代码示例中impact function部分的MLP组成如下：

conv2d(c,16,bn=True,activation=nn.ReLU(inplace=True),preact=False)
conv2d(16,16,bn=True,activation=nn.ReLU(inplace=True),preact=False)
conv2d(16,c,bn=False,activation=None,preact=False)

这里有一个问题：如果想把AFA用到其他模块，MLP部分需要自己调试

本文中学到的、可能用到的词、搭配、表述方法

conduct experiments
hierarchical network 层级网络
composed of 由……组成