2016.9-Jiwei Li-Deep reinforcement learning for dialogue generation-Stanford 阅读笔记

2016.9-Jiwei Li-Deep reinforcement learning for dialogue generation-arXiv-Stanford

Abstract

• Recent neural models
  • Short-sighted
    • predicting utterances one at a time
    • ignoring their influence on future outcomes
• This paper: combine Seq2Seq and RL paradigms
  • back bone: encoder-decoder
  • Apply deep RL: model future reward
    • can optimize long-term rewards designed by sys. dvlprs.
    • Policy graident method
    • 3 conversation properties
    • informativity, coherence, ease of answering
• Result (compared with standard Seq2Seq models with MLE objective)
  • more interactive responses
  • manages to foster a more sustained conversation

Method

• Action $a$
  • the dialogue utterance to generate.
  • the action space is infinite.
• State $[p_i,q_i]$ (previous)
  • dialogue history is transformed to a vec. repr. by feeding state to the LSTM encoder-decoder.
• Policy
  • the form of an LSTM encoder-decoder $p_{RL}(p_{i+1}|p_i,q_i)$
• Reward
  • Ease of answering: negative log likelihood of responding with a dull reponse.
  • Information flow: negative log of the cosine similarity between two consecutive turns $p_i$ and $p_{i+1}$
  • Demantic Coherence
• Initialize the Policy Model using Mutual Informaiton Model
  • Training steps:
    • Use pre-trained $p_{Seq2Seq}$ and $p_{Seq2Seq}^{backward}$ as the initial model (supervised Seq2Seq model)
    • Generate a list of candidates $\hat{a}\sim p_{RL}$ (注意此处不是 $p_{Seq2Seq}$)
    • Obtain the mutual infromation score of each $(\hat{a},[p_i,q_i]$ from the $p_{Seq2Seq}$ and $p_{Seq2Seq}^{backward}$
    • Use the mutual infromation score as rewards to tailor the $p_{Seq2Seq}$ and $p_{Seq2Seq}^{backward}$ to generate sequences with higher rewards.
• Optimize the Policy Model
  • use policy gradient methods
• Automatic Evaluation
  • Length of the Dialogue.
  • Diversity
  • Human Evaluation