【论文阅读】Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of

前言

• 本文是2018年的文章，我们着重理解idea与模型
• 本文idea：通过判断一个问题的答案是否有用来对问题进行排序
• 原文地址：Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information

Model

We build a neural network model inspired by the theory of expected value of perfect information (EVPI).

The value of this question $q_i$ is the expected utility, over all possible answers:

• $p$ is the post
• $q_i$ is a potential question from a set of candidate questions $Q$ and $a_j$ is a potential answer from a set of candidate answers $A$
• $p[a_j|p,q_i]$ measures the probability of getting an answer $a_j$ given an initial post $p$ and a clarifying question $q_i$,
• $U(p+a_j)$ is a utility function that measures how much more complete $p$ would be if it were augmented with answer $a_j$ .

The modeling question then is how to model:

• The probability distribution $P[a_j|p,q_i]$
• The utility function $U(p+a_j)$.

We train the parameters of the two models jointly to minimize a joint loss defined such that an answer that has a higher potential of increasing the utility of a post gets a higher probability

Question&answer candidate generator

Given a post p, we find similar posts(use Lucene) and their questions.

• we find the top 10 posts(contain the original post itself) most similar posts
• consider the questions asked to these 10 posts as our set of question candidates $Q$ and the edits made to the posts in response to the questions as our set of answer candidates $A$

Joint training

The training loss function is:

• use GloVe to get word embeddings
• $\hat{q},\hat{a}$ are obtained by averaging the GloVe word embeddings for all words in the question and the answer respectively
• $\bar{p},\bar{q},\bar{a}$ are obtained by post LSTM, question LSTM and answer LSTM
  • The input layer consists of word embeddings of the words which is fed into a single hidden layer. The output of each of the hidden states is averaged together to get our neural representation
• $F_{ans}$ is a feedforward neural network with five hidden layers on the inputs $\bar{p}$ and $\bar{q}$
• $F_{util}$ is a feedforward neural network with five hidden layers on the inputs $\bar{p}$ and $\bar{q}$ and $\bar{a}$

We train the three LSTMs and two feedforward networks jointly to minimize the sum of the loss of the answer model and the utility calculator over the entire dataset:

Answer modeling

The likelihood of an answer candidate $a_j$ being the answer to a question $q_i$ on post $p$ is

where $F_{ans}(p,q_i)$ is the answer representation and

We model our answer generator using the following intuition

• a question can be asked in several different ways
• for a given post and a question, there can be several different answers to that question

The loss function is:

The answer representation is not only close to its original answer but also to one of its candidate answers if the candidate question is close to the original question

Utility calculator

This utility function measures how useful it would be if a given post $p$ were augmented with an answer $a_j$ paired with a different question $q_j$ in the candidate set. A

We train our utility calculator using our dataset of (p, q, a) triples

• For each $a_j\in A_i$, where $A_i$ is the set of answer candidates for post $p_i$, we label the pair $(p_i,q_j,a_j)$ with label y = 0, except for when $a_j=a_i$(label = 1)
• Thus, for each post $p_i$ in our triples dataset, we have one positive sample and nine negative samples

We want this utility to be close to 1 for all the positively labelled (p, q, a) triples and close to 0 for all the negatively labelled (p, q, a) triples.