【Apache-Nutch 简要介绍】

lijieshare

于 2016-07-24 15:44:23 发布

阅读量664

点赞数

分类专栏： Apache Nutch 文章标签： Apache-Nutch 简要介绍

Apache Nutch 是一个高度可扩展和可伸缩的开源网络爬虫软件，源自 Apache Lucene。它包含 Nutch 1.x 和 2.x 两个版本，分别针对不同需求。Nutch 提供了可插拔的接口，如 Parse, Index 和 ScoringFilter，支持 Apache Solr 和 Elastic Search 等。Nutch 的工作流程包括创建 WebDB、抓取更新循环、生成索引等步骤，遵循 Robots Exclusion Protocol，并能处理重复 URL。Nutch 是基于 Lucene，适用于需要抓取和搜索分散数据源的场景。" 54020277,5808241,解决Wampserver启动报MSVCR110.dll和VCRUNTIME140.dll缺失问题,"['Windows', '开发环境', '软件安装', '问题解决']

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、Nutch介绍

What is Apache Nutch?

Apache Nutch is a highly extensible and scalable open source web crawler software project. Stemming from Apache Lucene, the project has diversified and now comprises two codebases, namely:

Nutch 1.x: A well matured, production ready crawler. 1.x enables fine grained configuration, relying on Apache Hadoop data structures, which are great for batch processing.
Nutch 2.x: An emerging alternative taking direct inspiration from 1.x, but which differs in one key area; storage is abstracted away from any specific underlying data store by using Apache Gora for handling object to pe

最低0.47元/天解锁文章