KBQA 常用的问答数据集之 WebQSP

目录

1. 论文相关

2. 数据集概述

     2.1 内容介绍

     2.2  数据统计

      2.3  文件介绍 

3. 模型性能比较

---

1. 论文相关

 WebQuestionsSP [Yih et al., 2016]

源自论文：The value of semantic parse labeling for knowledge base question answering

数据集： http://aka.ms/WebQSP                                                          

2. 数据集概述

    2.1 内容介绍

     WebQSP(WebQuestionsSP) 目标知识库是Freebase。该数据集可用在Question Answering，Semantic Parsing和Entity Linking任务。

     2.2  数据统计

WebQSP 数据集划分

Total	4737
Training	3098
Testing	1,639

      2.3  文件介绍 

         该数据集包含四个文件：WebQSP.[test|train].[partial|_].json. 其中，train/test 划分跟WebQuestions中的一样。训练集和测试集被另外分割成主数据集(WebQSP.train.json, WebQSP.test.json)和额外的“部分”问题(即有效的解析无法表示，或问题本身是坏的或需要一个描述性的回答)。

         eval/eval.py ：用python语言写的评估脚本 。

评估脚本的使用方法：

python eval.py goldData predAnswers

        eval/Pred.sem.json ：使用完整语义解析训练的STAGG系统的输出。

3. 模型性能比较

各模型在数据集WebQSP上的表现

模型(年份)	Hit@1	F1	Accuracy	论文	代码链接
NSM(2017)		69.0	59.5	Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision	Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision | Papers With Code
HR-BiLSTM (2017)			82.53	Improved Neural Relation Detection for Knowledge Base Question Answering	https://github.com/StevenWD/HR-BiLSTM
GRAFT-Net(2018)	67.8	62.8		Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text	https://github.com/haitian-sun/GraftNet
PullNet(2019)	68.1			PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text
TextRay(2019)		60.3		Learning to Answer Complex Questions over Knowledge Bases with Query Composition	GitHub - umich-dbgroup/TextRay-Release at master
SSRP(2019)		72.61		Neural Program Induction for KBQA Without Gold Programs or Query Annotations	https://github.com/CIPITR/SSRP
FullModel(2019)	68.2	67.9		Knowledge Base Question Answering with Topic Units
*UHop(2019) (通过对比显示有用)				UHop: An Unrestricted-Hop Relation Extraction Framework for Knowledge-Based Question Answering	GitHub - zychen423/UHop: UHop: An Unrestricted-Hop Relation Extraction Framework for Knowledge-Based Question Answering
Tree2Seq(2019)		63.7		Knowledge-based question answering by tree-to-sequence learning
QGG(2020)		74.0		Query Graph Generation for Answering Multi-hop Complex Questions from Knowledge Bases	GitHub - lanyunshi/Multi-hopComplexKBQA
ReifKB(2020)	52.7			SCALABLE NEURAL METHODS FOR REASONING WITH A SYMBOLIC KNOWLEDGE BASE
HGCN(2020)	66.9	60.1		Open Domain Question Answering based on  Text Enhanced Knowledge Graph with Hyperedge Infusion
MULTIQUE(2020)		61.16		Answering Complex Questions by Combining Information from Curated and Extracted Knowledge Bases
Rigel-Intersect(2021)	73.3			Expanding End-to-End Question Answering on Differentiable Knowledge Graphs with Intersection
Rigel-E2E(2021)	45.0			End-to-End Entity Resolution and Question Answering Using Differentiable Knowledge Graphs
NMSER(2021)	71.6	71.0		ReTraCk: A Flexible and Efficient Framework for Knowledge Base Question Answering
BERT-based KBQA (2021)	71.2	63.4		Large-Scale Relation Learning for Question Answering over Knowledge Bases with Pre-trained Language Models
TransferNet(2021)	71.4			TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph	GitHub - shijx12/TransferNet: Pytorch implementation of EMNLP 2021 paper "TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph "
EmbedKGQA(2021)	66.6			Improving multi-hop question answering over knowledge graphs using knowledge base embeddings	GitHub - malllabiisc/EmbedKGQA: ACL 2020: Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings
NSM(2021)	68.7			Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals	https://github.com/​​​​RichardHGL/WSDM2021_NSM
BART-large(2021)	74.1	74.6		Unseen Entity Handling in Complex Question Answering over Knowledge Base via Language Generation
RNG-KBQA(2021)		75.6		RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering	GitHub - salesforce/rng-kbqa
shrink KB(2021)		66.0		Improving Query Graph Generation for Complex Question Answering over Knowledge Base
RecHyperNet(2021)	68.4			Knowledge Base Question Answering through Recursive Hypergraphs

内容将持续更新，欢迎大家评论和补充~