Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

Haonan Yu,  Jiang Wang,  Zhiheng Huang,  Yi Yang,  Wei Xu

(Submitted on 26 Oct 2015 ( v1), last revised 6 Apr 2016 (this version, v2))

We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a realistic video. Our hierarchical framework contains a sentence generator and a paragraph generator. The sentence generator produces one simple short sentence that describes a specific short video interval. It exploits both temporal- and spatial-attention mechanisms to selectively focus on visual elements during generation. The paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator. We evaluate our approach on two large-scale benchmark datasets: YouTubeClips and TACoS-MultiLevel. The experiments demonstrate that our approach significantly outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and 0.305 respectively.

Comments:	In CVPR2016
Subjects:	Computer Vision and Pattern Recognition (cs.CV)
Cite as:	arXiv:1510.07712 [cs.CV]
	(or arXiv:1510.07712v2 [cs.CV] for this version)

Submission history

From: Haonan Yu [ view email] 
[v1] Mon, 26 Oct 2015 22:47:00 GMT (4584kb,D)
[v2] Wed, 6 Apr 2016 02:24:35 GMT (2630kb,D)