文章目录
Li Y, Ouyang W, Zhou B等. Scene Graph Generation from Objects, Phrases and Region Captions[J]. Proceedings of the IEEE International Conference on Computer Vision, 2017, 2017-October: 1270–1279.
摘要
- 目标检测 、 场景图谱生成 和 region captioning 是在不同语义等级上的图像理解任务
它们通常是被绑定在一起的。
本文为了利用跨越不同语义等级的相互联系,本文提出了一种新的神经网络模型,Multi-level Scene Description Network(MSDN).
-
一种端到端的方式解决这三个联合在一起的问题。
-
方法简介
- 对象、文本和regions 首先在一个动态图中表示
- 采用特征细化结构,在三个语义层次上传递消息
- 在三个基准上测试了本方法。SOTA 3% margin
- 代码地址: Scene Graph Generation from Objects, Phrases and Region Captions
引言
背景
- 图像理解任务
object detections
[33] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. arXiv preprint arXiv:1506.02640, 2015. 1, 2
[34] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015. 1, 2, 3, 7, 8
[30] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed. Ssd: Single shot multibox detector. arXiv preprint arXiv:1512.02325, 2015. 1, 2
scene graph genetation
[31] C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei. Visual rela- tionship detection with language priors. In ECCV, 2016. 1, 2, 6, 7
[40] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene graph
generation by iterative message passing. arXiv preprint arXiv:1701.02426, 2017. 1, 2, 6, 7
[28] Y. Li, W. Ouyang, X. Wang, and X. Tang. Vip-cnn: Visual
phrase guided convolutional neural network. CVPR, 2017.1, 2
[45] H. Zhang, Z. Kyaw, J. Yu, and S.-F. Chang. Ppr-fcn: Weakly
supervised visual relation detection via parallel pairwise rfcn. In ICCV, 2017. 1
image/region caption
[41] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2015. 1, 2
[10] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Dar- rell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015. 1
[22] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.1, 2, 5
这三个子任务分别代表了图像理解的三个不同的语义水平。
- object detection 关注单个对象
- scene graph 代表不同对象之间的关系
- region caption 代表一个自由句,用于描述图像
问题1
- 如何协同训练同时解决三个问题的模型。
关键点
- 利用视觉特征的空间和语义关系
方法简介
- end-to-end Multi-level Scene Description Network(MSDN)
contribution
- 同时利用三个语义水平的特征,解决三个任务
- 多图共同构建动态graph
- 利用特征细化结构在三个语义水平的特征上进行信息传递
数据集
- Visual Genome dataset [23]
[23] R. Krishna. Y. Zhu O. Groth J. Johnson. K. Hata. J. Kravitz S. Chen. Y. Kalantidis. L -J. Li. D. A. Shamma. et al Visual genome:Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:l602.07332.2016.2.5,7
SOTA
正文部分
敬请期待
Conclusionn
- 图像理解任务
- 同时对object detection , visual relationship , region captioning 三个任务进行建模
- 端到端 : MSDN
- 多图动态维护一张表
- SOTA