datahub文档_DataHub的设计和开发.doc

DataHub的设计和开发.doc

DataHub 的设计和开发

周强 孔贝贝 谢靖 吴振新

(中国科学院文献情报中心，北京，100190)

[摘要]

本文介绍了OAI元数据获取协议(OAI-PMH)，介绍了Solr，描述了DataHub 的体系结构。首先详细说明了数据导入后台部分和前台部分的策略和设计思路，后台部分对于各个数据源分别创建数据连接器，由定时器调度来执行；前台部分通过Web界面，查看当前任务、所有任务、总数据量、每批次数据量。接着详细说明了创建索引的策略和设计思路，现在有两套Solr索引，一套是正式提供对外服务的服务，一套是备份索引来创建索引，正式索引和备份索引是轮流切换的，在创建索引时，要指定创建索引的目标地址。现在有两个配置界面，一个配置界面是指定创建索引的目标地址，另一个配置界面是每个数据类型，要指定本次索引的最小ID值、最大ID值。现在DataHub已经正常运行了，进行日常数据导入和创建索引，数据导入和索引创建及时正确。

[关键词] OAI-PMH; 数据导入; Solr; 创建索引

DataHub design and development

Zhou Qiang KongBeibei Xie Jing Wu ZhenXin

(National Science Library, Chinese Academy of Sciences, Beijing 100195, China)

[Abstract]

This article describes the OAI metadata access protocol (OAI-PMH), introduces the Solr, describes the DataHub system. First detailed import data back part and the front part of the strategy and design, the background section for each data source to create a data connection, executed by the timer schedule to; the front part through the Web interface, see the current tasks, all tasks, the total amount of data, each batch of data. Then details the strategy and design to create an index, there are two sets of Solr indexing, one official providing external services services, a set of backup index to create indexes, indexes and backup index is a rotating switch, when you create an index, to specify the destination address of the index created. There are two configuration interface, created a configuration interface is specified the destination address of the index, is another configuration interface for each data type, to specify the index of the smallest ID value, the maximum ID value. DataHub is running now, and daily data import and index creation, data import and index creation time is correct.

[Keyword] OAI-PMH；data import; Solr; create the index

一、引言

可视化分面集成检索平台集成的本地数据，包括订购图书期刊、电子期刊论文、馆藏信息、二次文献库资源、其它开放采集资源等[1]，来源比较多，有用OAI元数据获取协议收割的数据，有从文本文件中导入的数据，有从 E