SemEval-2010 Task-8

关系抽取 (Relation Extraction) 论文中常用的数据集：SemEval-2010 Task-8
Multi-Way Classification of Semantic Relations Between Pairs of Nominals

官方网址： http://semeval2.fbk.eu/semeval2.php
文档： https://www.aclweb.org/anthology/S10-1006/
数据： https://github.com/CrazilyCode/SemEval2010-Task8

BibTeX:

@inproceedings{hendrickx-etal-2010-semeval,
    title = "{S}em{E}val-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals",
    author = "Hendrickx, Iris  and
      Kim, Su Nam  and
      Kozareva, Zornitsa  and
      Nakov, Preslav  and
      {\'O} S{\'e}aghdha, Diarmuid  and
      Pad{\'o}, Sebastian  and
      Pennacchiotti, Marco  and
      Romano, Lorenza  and
      Szpakowicz, Stan",
    booktitle = "Proceedings of the 5th International Workshop on Semantic Evaluation",
    month = jul,
    year = "2010",
    address = "Uppsala, Sweden",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/S10-1006",
    pages = "33--38",
}

示例数据：
句子：The <e1>fire</e1> inside WTC was caused by exploding <e2>fuel</e2>.
关系：Cause-Effect(e2,e1)

训练集：共含8000条数据

type	number	rate
Other	1410	17.63%
Cause-Effect	1003	12.54%
Component-Whole	941	11.76%
Entity-Destination	845	10.56%
Product-Producer	717	8.96%
Entity-Origin	716	8.95%
Member-Collection	690	8.63%
Message-Topic	634	7.92%
Content-Container	540	6.75%
Instrument-Agency	504	6.30%

测试集：共含2717条数据

type	number	rate
Other	454	16.71%
Cause-Effect	328	12.07%
Component-Whole	312	11.48%
Entity-Destination	292	10.75%
Message-Topic	261	9.61%
Entity-Origin	258	9.50%
Member-Collection	233	8.58%
Product-Producer	231	8.50%
Content-Container	192	7.07%
Instrument-Agency	156	5.74%

测评：官方给出了perl语言的测评程序
The scorer calculates and outputs the following statistics:
(1) confusion matrix, which shows
- the sums for each row/column: -SUM-
- the number of skipped examples: skip
- the number of examples with correct relation, but wrong directionality: xDIRx
- the number of examples in the answer key file: ACTUAL ( = -SUM- + skip + xDIRx )
(2) accuracy and coverage
(3) precision P, recall R, and F1-score for each relation
(4) micro-averaged P, R, F1, where the calculations ignore the Other category.
(5) macro-averaged P, R, F1, where the calculations ignore the Other category.
Note that in scores (4) and (5), skipped examples are equivalent to those classified as Other.
So are examples classified as relations that do not exist in the key file (which is probably not optimal).
The scoring is done three times:
(i) as a (2*9+1)-way classification
(ii) as a (9+1)-way classification, with directionality ignored
(iii) as a (9+1)-way classification, with directionality taken into account.
The official score is the macro-averaged F1-score for (iii).

相关论文：

论文	结果	备注
(2014COLING) Relation Classification via Convolutional Deep Neural Network	82.7%	预训练好的词向量表示+WordNet
(2015ACL) Classifying Relations by Ranking with Convolutional Neural Networks	84.1%	预训练好的词向量表示

持续更新中。。。