Nutch 使用metadata plugin捕获页面中的meta标签数据

Nutch 使用metadata plugin捕获页面中的meta标签数据

添加index-metadata和parse-metatags

要使用Nutch的插件，就需要在conf/nutch-site.xml中指定要使用的插件，插件的选取可以使用正则匹配去选取的，插件的nutch的根目录下的apache-nutch-x.x/plugins文件夹中。

index-(basic|anchor|metadata)这句包含了index-metadata，index-anchor，index-basic。

parse-(html|metatags)这句包含了parse-html，parse-metatags

<!--具体需要什么插件,参考官方提供的plugins文件夹,不同版本包含的插件不同-->
<property>
  	<name>plugin.includes</name>
  	<value>protocol-http|protocol-httpclient|urlfilter-regex|parse-(html|metatags)|index-(basic|anchor|metadata)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
	<description>my Nutch plugin</description>
</property>

指定index-metadata需要的参数

需要获取那些metatags的内容，可以配置下面这些参数。

<!-- Used only if plugin parse-metatags is enabled. -->
<property>
<name>metatags.names</name>
<value>description,keywords</value>
<description> Names of the metatags to extract, separated by ','.
  Use '*' to extract all metatags. Prefixes the names with 'metatag.'
  in the parse-metadata. For instance to index description and keywords,
  you need to activate the plugin index-metadata and set the value of the
  parameter 'index.parse.md' to 'metatag.description,metatag.keywords'.
</description>
</property>

1.x版本

处理要配置要获取的metatags的名字，还要配置metatags的解析字段名

<property>
  <name>index.parse.md</name>
  <value>metatag.description,metatag.keywords</value>
  <description>
  Comma-separated list of keys to be taken from the parse metadata to generate fields.
  Can be used e.g. for 'description' or 'keywords' provided that these values are generated
  by a parser (see parse-metatags plugin)
  </description>
</property>

2.x版本

在2.x中官方说明可以省略metatag.

<property>
  <name>index.metadata</name>
  <value>description,keywords</value>
  <description>
  Comma-separated list of keys to be taken from the metadata to generate fields.
  Can be used e.g. for 'description' or 'keywords' provided that these values are generated
  by a parser (see parse-metatags plugin), and property 'metatags.names'.
  </description>
</property>

添加metatags plugin需要fields

如果你需要在solr中建立索引，可以在solr的配置文件中添加以下字段

<!-- fields for the metatags plugin -->
<field name="metatag.description" type="text" stored="true" indexed="true"/>
<field name="metatag.keywords" type="text" stored="true" indexed="true"/>

测试

可以用随便一条网页查看爬取解析的结果

#使用indexchecker检查爬取解析处理的index
./bin/nutch indexchecker https://blog.csdn.net/qq_43203949/article/details/108550960

#结果
fetching: https://blog.csdn.net/qq_43203949/article/details/108550960
parsing: https://blog.csdn.net/qq_43203949/article/details/108550960
contentType: text/html
tstamp :	Sun Jan 02 23:19:25 CST 2022
metatag.description :	解决Could not load dynamic library ‘cudnn64_7.dll’; dlerror: cudnn64_7.dll not found如果你使用的是tensorflow或
digest :	9f1bc7879417faaa87e82a7d65ca0cab
host :	blog.csdn.net
id :	https://blog.csdn.net/qq_43203949/article/details/108550960
title :	解决Could not load dynamic library ‘cudnn64_7.dll‘； dlerror cudnn64_7.dll not found_鸭梨的药丸哥的博客-CSDN博客_c
metatag.keywords :	cudnn64_7.dll
url :	https://blog.csdn.net/qq_43203949/article/details/108550960
content :	解决Could not load dynamic library ‘cudnn64_7.dll‘； dlerror cudnn64_7.dll not found_鸭梨的药丸哥的博客-CSDN博客_c