Survey on Improving sequence-to-sequence constituency parsing

Improving Sequence-to-Sequence ConstituencyParsing

April 9, 2018

. . . ................. . .. . . .

 

Outline

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

1 BackgroundParsing Task

Sequence-to-Sequence Constituency Parsing

2 Improvment of Sequence-to-Sequence Constituency ParsingImprovment 1 : Supervised Attention
Improvment 2 : Bottom-Up Deterministic AttentionImprovment 3 : Top-down Deterministic Attention

............... . ..................... ...

 

Parsing Task

ImprovingSequence-to-SequenceConstituencyParsing

Background

Parsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Definition: Convert the sentence to tree structure.

............... . ..................... ...

 

Grammar as Foreign Language

ImprovingSequence-to-SequenceConstituencyParsing

Background

Parsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Method: Top-down linearizatoin a parse tree + Seqs-RNN +Attention Mechanism

............... . ..................... ...

 

Supervised Attention: Motivation

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Motivation: The attention of model proposed by Vinyals etal. (2015) is learned in an unsupervised manner, the alignmentquality might not be optimal.

............... . ..................... ...

 

Supervised Attention: Method

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Method: Define several linguistically-motivatedannotations between surface words and nonterminals asgold standard alignments ) by supervised attention.

............... . ..................... ...

 

Deterministic Attention: Motivation

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Motivation : The probabilistic attention mechanism ignoresthe inherent characteristics of constituent parsing: theboundary and head information that has proven to be criticalwhen identifying phrase structure and predicting its label.

............... . ..................... ...

 

Deterministic Attention: Method

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Original Probabilistic Method:

Deterministic Attention method:

Di : indices of the words that should be paid attention to.Am: deterministic alignment matrix, dimension: dim(c)*dim(h)

............... . ..................... ...

 

Deterministic Attention: Deterministic AttentionScheme

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

With Bottem-Up linearization:

With a stack and buffer:
Scheme 1 (datt-bound)
Di = (0,r,s,t,T)
Scheme 2 (datt-bound-head): Di = (0,r,t,hrs,hst,T)

............... . ..................... ...

 

Deterministic Attention: Example

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Example:

with scheme "datt-bound", at "4-5" time steps,Di = (0,1,2,3,5)

............... . ..................... ...

 

Deterministic Attention: Case Study

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

............... . ..................... ...

 

Top-down Deterministic Attention

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Motivation: Explore the deterministic attention directly ontop of top-down linearization, with the expectation to improvethe sequence-to-sequence constituency parsing

............... . ..................... ...

 

Top-down Deterministic Attention: Design

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Alignment heuristics

............... . ..................... ...

 

Top-down Deterministic Attention: Design

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Alignment heuristics

............... . ..................... ...

 

Top-down Deterministic Attention: Design

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Alignment heuristics

............... . ..................... ...

 

Top-down Deterministic Attention: Method

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Compare

............... . ..................... ...

 

Top-down Deterministic Attention: Method

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Define the context ct for yt based on the deterministicattention as following:

simplifize:

ct = φ(P(bt,st),P(st,?),st,Ex)

φ = θc · [E x ; E x ; E x ]

reduce:

reduce:

φ = θc · [E x ; E x ]bt st

φ=θc ·Exst

bt st1

st

............... . ..................... ...

 

Top-down Deterministic Attention: Contribution

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

............... . ..................... ...

 

Reference

ImprovingSequence-to-SequenceConstituencyParsing

BackgroundParsing Task

Sequence-to-SequenceConstituency Parsing

Improvment ofSequence-to-SequenceConstituencyParsing

Improvment 1 :Supervised Attention

Improvment 2 :Bottom-UpDeterministicAttention

Improvment 3 :Top-downDeterministicAttention

Vinyals, O.; Kaiser, .; Koo, T.; Petrov, S.; Sutskever, I.; and Hinton, G. 2015. Grammar as a foreignlanguage. In Proceedings of NIPS.

Ma, C.; Liu, L.; Tamura, A.; Zhao, T.; and Sumita, E. 2017. Deterministic attention forsequence-to-sequence constituent parsing. In Proceedings of the AAAI.

Kamigaito, H., Hayashi, K. (2017). Supervised Attention for Sequence-to-Sequence ConstituencyParsing. Ijcnlp2017, (2015), 712.

Liu, L., Zhu, M., Shi, S. (2018). Improving Sequence-to-Sequence Constituency Parsing.

............... . ..................... ...


  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值