record

恢复实验

---

Paper: How a General-Purpose Commonsense Ontology can Improve Performance of Learning-Based Image Retrieval.
code: https://bitbucket.org/RToroIcarte/cn-detectors.

Abstract

The knowledge representation community has built general-purpose ontologies which contain large amounts of commonsense knowledge over relevant aspects of the world, including useful visual information, e.g.: “a ball is used by a football player”, “a tennis player is located at a tennis court”. Current state-of-the-art approaches for visual recognition do not exploit these rule-based knowledge sources. Instead, they learn recognition models directly from training examples. In this paper, we study how general-purpose ontologies—specifically, MIT’s ConceptNet ontology—can improve the performance of state-of-the-art vision systems. As a testbed, we tackle the problem of sentence-based image retrieval. Our retrieval approach incorporates knowledge from ConceptNet on top of a large pool of object detectors derived from a deep learning technique. In our experiments, we show that ConceptNet can improve performance on a common benchmark dataset. Key to our performance is the use of the ESPGAME
dataset to select visually relevant relations from ConceptNet. Consequently, a main conclusion of this work is that general-purpose commonsense ontologies improve performance on visual reasoning tasks when properly filtered to select meaningful visual relations.

About this project

这个project研究了通识实体，例如MIT’s ConceptNet ontology，如何可以提高最新的视觉系统的性能。第一步，我们解决基于句子的图像检索的问题。可以通过下面来引用来引用该论文。

@inproceedings{toro2017ontology,  
　   title={How a General-Purpose Commonsense Ontology can Improve Performance of Learning-Based Image Retrieval},  
 　  author={Toro Icarte, Rodrigo and Baier, Jorge A. and Ruz, Cristian and Soto, Alvaro},  
  　 booktitle={26th International Joint Conference on Artificial Intelligence},  
 　  year={2017}   
 }

这份代码已经更新并且clean过了，主要的不同就是这份的用的是最新版本的ConceptNet和一个不同的POS tagger和 word stemming algorithm。最终，性能略优于论文中提到的结果，我们得到一个新的评分优于论文中的CN-MAX【微笑】。

Installation instructions

Prerequisites

代码需要用到Python2.7里的三个库：numpy，nltk and enum34。
我的Ubuntu自带Python2.7，无需再下载，下载后面三个库。
再Ubuntu下安装Python模块通常可用apt-get和pip命令。apt-get是Ubuntu自带地包管理命令，pip则是Python安装扩展模块的工具。通常pip会下载扩展模块的源代码并编译安装。所以先安装开发包和编译器。
　sudo apt-get install python python-dev python-pip

• 下载numpy
  sudo pip install numpy

• 检查numpy是否可用
  命令行输入 python>>>import numpy>>>from numpy import mean>>>mean([1,2,3])

• 下载nltk
  sudo pip install -U nltk

• 下载nltk的数据
  命令行输入 python>>>import nltk>>>nltk.download()
  会给出列表，选择下载maxent_treebank_pos_tagger和punkt这两个包

• 下载enum34
  sudo pip install enum34

Cloning this repository and getting external resources

网页所给的下载命令是
git clone git@bitbucket.org:RToroIcarte/cn-detectors.git
cd cn-detectors
我的下载不了，用下面命令可下载
git clone https://bitbucket.org/RToroIcarte/cn-detectors
然后需要跑“install.sh”脚本来结束下载
cd cn-detectors
./install.sh
要等超级久！！！“install.sh”文件中可以看到是从几个网页下载的的文件，如果慢的话可以复制网页直接下载。
ConceptNet https://s3.amazonaws.com/conceptnet/precomputed-data/2016/assertions/conceptnet-assertions-5.5.0.csv.gz
ESPGAMES http://www.cs.toronto.edu/~rntoro/resources/ESPGame100k.tar.gz
MS-COCO http://msvocds.blob.core.windows.net/annotations-1-0-3/captions_train-val2014.zip
detectors http://www.cs.toronto.edu/~rntoro/resources/detections.zip
根据.sh文件输入命令行也是一样的。
这里下载了ConceptNet5.5,ESPGAMES,MSCOCO annotations和detections.json文件。为了省时间省内存，就没有下载MSCOCO的图片了。如果需要训练和重跑检测器的话，可以下载MSCOCO和visual-concept detectors

Running examples

python image_retrieval.py --num_cores=8 --test_file="./tests/coco-1k.txt" --score="./config/mil.json"
python image_retrieval.py --num_cores=8 --test_file="./tests/coco-1k.txt" --score="./config/milstem.json"
python image_retrieval.py --num_cores=8 --test_file="./tests/coco-1k.txt" --score="./config/espgame_max.json"
python image_retrieval.py --num_cores=8 --test_file="./tests/coco-1k.txt" --score="./config/cn_max.json"
python image_retrieval.py --num_cores=8 --test_file="./tests/coco-1k.txt" --score="./config/cn_gwa.json"

但是我的电脑跑着跑着跑了很久就MemoryErrorr了。。。
所以未完成，完成了我再更新好了。