Improving Sequence-to-Sequence ConstituencyParsing
April 9, 2018
. . . ................. . .. . . .
Outline | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
1 BackgroundParsing Task Sequence-to-Sequence Constituency Parsing 2 Improvment of Sequence-to-Sequence Constituency ParsingImprovment 1 : Supervised Attention ............... . ..................... ... |
Parsing Task | |
ImprovingSequence-to-SequenceConstituencyParsing Background Parsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Definition: Convert the sentence to tree structure. ............... . ..................... ... |
Grammar as Foreign Language | |
ImprovingSequence-to-SequenceConstituencyParsing Background Parsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Method: Top-down linearizatoin a parse tree + Seqs-RNN +Attention Mechanism ............... . ..................... ... |
Supervised Attention: Motivation | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Motivation: The attention of model proposed by Vinyals etal. (2015) is learned in an unsupervised manner, the alignmentquality might not be optimal. ............... . ..................... ... |
Supervised Attention: Method | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Method: Define several linguistically-motivatedannotations between surface words and nonterminals asgold standard alignments ) by supervised attention. ............... . ..................... ... |
Deterministic Attention: Motivation | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Motivation : The probabilistic attention mechanism ignoresthe inherent characteristics of constituent parsing: theboundary and head information that has proven to be criticalwhen identifying phrase structure and predicting its label. ............... . ..................... ... |
Deterministic Attention: Method | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Original Probabilistic Method: Deterministic Attention method: Di : indices of the words that should be paid attention to.Am: deterministic alignment matrix, dimension: dim(c)*dim(h) ............... . ..................... ... |
Deterministic Attention: Deterministic AttentionScheme | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
With Bottem-Up linearization: With a stack and buffer: ............... . ..................... ... |
Deterministic Attention: Example | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Example: with scheme "datt-bound", at "4-5" time steps,Di = (0,1,2,3,5) ............... . ..................... ... |
Deterministic Attention: Case Study | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
............... . ..................... ... |
Top-down Deterministic Attention | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Motivation: Explore the deterministic attention directly ontop of top-down linearization, with the expectation to improvethe sequence-to-sequence constituency parsing ............... . ..................... ... |
Top-down Deterministic Attention: Design | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Alignment heuristics ............... . ..................... ... |
Top-down Deterministic Attention: Design | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Alignment heuristics ............... . ..................... ... |
Top-down Deterministic Attention: Design | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Alignment heuristics ............... . ..................... ... |
Top-down Deterministic Attention: Method | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Compare ............... . ..................... ... |
Top-down Deterministic Attention: Method | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Define the context ct for yt based on the deterministicattention as following: simplifize: ct = φ(P(bt,st),P(st,?),st,Ex) φ = θc · [E x ; E x ; E x ] reduce: reduce: φ = θc · [E x ; E x ]bt st φ=θc ·Exst bt st−1 st ............... . ..................... ... |
Top-down Deterministic Attention: Contribution | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
............... . ..................... ... |
Reference | |
ImprovingSequence-to-SequenceConstituencyParsing BackgroundParsing Task Sequence-to-SequenceConstituency Parsing Improvment ofSequence-to-SequenceConstituencyParsing Improvment 1 :Supervised Attention Improvment 2 :Bottom-UpDeterministicAttention Improvment 3 :Top-downDeterministicAttention |
Vinyals, O.; Kaiser, .; Koo, T.; Petrov, S.; Sutskever, I.; and Hinton, G. 2015. Grammar as a foreignlanguage. In Proceedings of NIPS. Ma, C.; Liu, L.; Tamura, A.; Zhao, T.; and Sumita, E. 2017. Deterministic attention forsequence-to-sequence constituent parsing. In Proceedings of the AAAI. Kamigaito, H., Hayashi, K. (2017). Supervised Attention for Sequence-to-Sequence ConstituencyParsing. Ijcnlp2017, (2015), 712. Liu, L., Zhu, M., Shi, S. (2018). Improving Sequence-to-Sequence Constituency Parsing. ............... . ..................... ... |