Survey on Improving sequence-to-sequence constituency parsing

Improving Sequence-to-Sequence ConstituencyParsing

April 9, 2018

. . . ................. . .. . . .

	Outline
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	1 BackgroundParsing Task Sequence-to-Sequence Constituency Parsing 2 Improvment of Sequence-to-Sequence Constituency ParsingImprovment 1 : Supervised Attention Improvment 2 : Bottom-Up Deterministic AttentionImprovment 3 : Top-down Deterministic Attention ............... . ..................... ...

	Parsing Task
ImprovingSequence-to-SequenceConstituencyParsing Background Parsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Definition: Convert the sentence to tree structure. ............... . ..................... ...

	Grammar as Foreign Language
ImprovingSequence-to-SequenceConstituencyParsing Background Parsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Method: Top-down linearizatoin a parse tree + Seqs-RNN +Attention Mechanism ............... . ..................... ...

	Supervised Attention: Motivation
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Motivation: The attention of model proposed by Vinyals etal. (2015) is learned in an unsupervised manner, the alignmentquality might not be optimal. ............... . ..................... ...

	Supervised Attention: Method
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Method: Define several linguistically-motivatedannotations between surface words and nonterminals asgold standard alignments ) by supervised attention. ............... . ..................... ...

	Deterministic Attention: Motivation
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Motivation : The probabilistic attention mechanism ignoresthe inherent characteristics of constituent parsing: theboundary and head information that has proven to be criticalwhen identifying phrase structure and predicting its label. ............... . ..................... ...

	Deterministic Attention: Method
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Original Probabilistic Method: Deterministic Attention method: Di : indices of the words that should be paid attention to.Am: deterministic alignment matrix, dimension: dim(c)*dim(h) ............... . ..................... ...

	Deterministic Attention: Deterministic AttentionScheme
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	With Bottem-Up linearization: With a stack and buffer: Scheme 1 (datt-bound)Di = (0,r,s,t,T) Scheme 2 (datt-bound-head): Di = (0,r,t,hrs,hst,T) ............... . ..................... ...

	Deterministic Attention: Example
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Example: with scheme "datt-bound", at "4-5" time steps,Di = (0,1,2,3,5) ............... . ..................... ...

	Deterministic Attention: Case Study
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	............... . ..................... ...

	Top-down Deterministic Attention
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Motivation: Explore the deterministic attention directly ontop of top-down linearization, with the expectation to improvethe sequence-to-sequence constituency parsing ............... . ..................... ...

	Top-down Deterministic Attention: Design
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Alignment heuristics ............... . ..................... ...

	Top-down Deterministic Attention: Design
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Alignment heuristics ............... . ..................... ...

	Top-down Deterministic Attention: Design
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Alignment heuristics ............... . ..................... ...

	Top-down Deterministic Attention: Method
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Compare ............... . ..................... ...

	Top-down Deterministic Attention: Method
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	Define the context ct for yt based on the deterministicattention as following: simplifize: ct = φ(P(bt,st),P(st,?),st,Ex) φ = θc · [E x ; E x ; E x ] reduce: reduce: φ = θc · [E x ; E x ]bt st φ=θc ·Exst bt st−1 st ............... . ..................... ...

	Top-down Deterministic Attention: Contribution
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention	............... . ..................... ...

Reference

ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention

Vinyals, O.; Kaiser, .; Koo, T.; Petrov, S.; Sutskever, I.; and Hinton, G. 2015. Grammar as a foreignlanguage. In Proceedings of NIPS. Ma, C.; Liu, L.; Tamura, A.; Zhao, T.; and Sumita, E. 2017. Deterministic attention forsequence-to-sequence constituent parsing. In Proceedings of the AAAI. Kamigaito, H., Hayashi, K. (2017). Supervised Attention for Sequence-to-Sequence ConstituencyParsing. Ijcnlp2017, (2015), 712. Liu, L., Zhu, M., Shi, S. (2018). Improving Sequence-to-Sequence Constituency Parsing. ............... . ..................... ...