PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

1. Related Work

1.1 point cloud

PPT

1.2 Deep learning on 3D Data

• Volumetric CNNs: applying 3D convolutional neural networks on voxelized shapes.

However, volumetric representation is constrained by its resolution due to data sparsity and computation cost of 3D convolution.

• Multiview CNNs: have tried to render 3D point cloud or shapes into 2D images and then apply 2D conv nets to classify them.

However, it’s nontrivial to extend them to scene understanding or other 3D tasks such as point classification and shape completion.

• Feature-based DNNs: firstly convert the 3D data into a vector, by extracting traditional shape features and then use a fully connected net to classify the shape.

backwards：We think they are constrained by the representation power of the features extracted.

2.PointNet

• other methor：most researchers transform such data to regular 3D voxel grids or collections of images.

This, however, renders data unnecessarily voluminous and causes issues.

• paper methor:PPT

3.Experiment

Evaluation metric is mIoU on points. For each shape S of category C, to calculate the shape’s mIoU: For each part type in category C, compute IoU between groundtruth and prediction. If the union of groundtruth and prediction points is empty, then count part IoU as 1. Then we average IoUs for all part types in category C to get mIoU for that shape. To calculate mIoU for the category, we take average of mIoUs for all shapes in that category.

3.1 3D Object Classification

backwards：There is still a small gap between our method and multi-view based method (MVCNN [23]), which we think is due to the loss of fine geometry details that can be captured by rendered images.

3.2 3D Object Part Segmentation

Robust test：We also perform experiments on simulated Kinect scans to test the robustness of these methods. For every CAD model in the ShapeNet part data set, we use Blensor Kinect Simulator [7] to generate incomplete point clouds from six random viewpoints. We train our PointNet on the complete shapes and partial scans with the same network architecture and training setting. Results show that we lose only 5.3% mean IoU.

3.3 Semantic Segmentation in Scenes

idea： Our network on part segmentation can be easily extended to semantic scene segmentation, where point labels become semantic object classes instead of object part labels. We train our segmentation version of PointNet to predict per point class in each block.

other work： Based on the semantic segmentation output from our
network, we further build a 3D object detection system using connected component for object proposal

3.3 Robustness Test

PPT

4.Conclusion

In this work, we propose a novel deep neural network PointNet that directly consumes point cloud. Our network provides a unified approach to a number of 3D recognition tasks including object classification, part segmentation and semantic segmentation, while obtaining on par or better results than state of the arts on standard benchmarks. We also provide theoretical analysis and visualizations towards understanding of our network.

总结

• 论文看得不够透彻，很多地方还没懂，要结合Marginnote再仔细做笔记
• 汇报稿子没准备好，汇报结结巴巴，不顺畅
• 下一步，结合pointnet++做实验
• 附上在CPU上跑了一天半的pointnet