python从网页提取文本_使用Python进行网页正文提取

1. Goose Extractor

1.1 Python Goose介绍

Goose Extractor是一个Python的开源文章提取库。可以用它提取文章的文本内容、图片、视频、元信息和标签。Goose本来是由Gravity.com编写的Java库，最近转向了scala。

Goose Extractor网站是这么介绍的：

Goose Extractor完全用Python重写了。目标是给定任意资讯文章或者任意文章类的网页，不仅提取出文章的主体，同时提取出所有元信息以及图片等信息。

Goose Extractor基于NLTK和Beautiful Soup，分别是文本处理和HTML解析的领导者。用Python进行文章提取可以使用Python Goose。

Goose目前只支持Python2

1.2 安装Python Goose

pip install goose-extractor

直接使用Url链接抽取示例：

from goose import Goose

url = 'https://www.fireeye.com/blog/executive-perspective/2017/08/fireeye-provides-update-on-allegations-of-breach.html'

g = Goose()

article = g.extract(url=url)

print article.title

print article.meta_description

print article.cleaned_text[:150]

print article.top_image.src

使用Html文档抽取示例：

# -*- coding: utf-8 -*-

import goose,urllib2,sys

reload(sys)

sys.setdefaultencoding("utf-8")

#url = "https://www.fireeye.com/blog/executive-perspective/2017/08/anti-encryption-and-cyber-sovereignty.html"

url = "https://krebsonsecurity.com/2017/09/equifax-hackers-stole-200k-credit-card-accounts-in-one-fell-swoop/"

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())

response = opener.open(url)

raw_html = response.read()

g = goose.Goose()

article = g.extract(raw_html=raw_html)

print article.title.encode('gbk', 'ignore')

print article.meta_description.encode('gbk', 'ignore')

print article.cleaned_text.encode('gbk', 'ignore')

1.3 urllib2获取的HTML网页乱码问题

网页可能是压缩了，看里面是不是有 Content-Encoding:xxx

如果是压缩了，需要手动解压，urllib是不会帮你解压的

解决代码：

#-*- encoding: utf-8 -*-

import urllib2,gzip,StringIO

url = r'https://krebsonsecurity.com/2017/09/equifax-hackers-stole-200k-credit-card-accounts-in-one-fell-swoop/'

response = urllib2.urlopen(url)

stream = StringIO.StringIO(response.read())

with gzip.GzipFile(fileobj=stream) as f:

data = f.read()

print(data)

附一篇文章谈Python编码：也谈Python的中文编码处理

2. Boilerpipe

Github开源代码：Boilerpipe

在开源系统里Boilerpipe的precision和recall都好过Goose，甚至比收费的Alchemy API还要好。Boilerpipe是Java的，在Python里调用需要用python-boilerpipe这个包装，它底层用的是jpype。也可以用JCC来调。代码如下：

安装：

git clone https://github.com/misja/python-boilerpipe.git

cd python-boilerpipe

pip install -r requirements.txt

python setup.py install

使用：

from boilerpipe.extract import Extractor

url = "https://krebsonsecurity.com/2017/09/equifax-hackers-stole-200k-credit-card-accounts-in-one-fell-swoop/"

extractor = Extractor(extractor='ArticleExtractor', url=url)

print extractor.getText().encode('gbk', 'ignore')

或传入一个HTML文本作为参数：

extractor = Extractor(extractor='ArticleExtractor', html=myWebPage)

用getText() or getHTML() 拿回处理过的纯文本或加亮了正文的HTML

processed_plaintext = extractor.getText()

highlighted_html = extractor.getHTML()

也可以用JCC把Java的包编译成Python可以调用的包

wget http://boilerpipe.googlecode.com/files/boilerpipe-1.2.0-bin.tar.gz

tar xvzf boilerpipe-*.tar.gz

cd boilerpipe-1.2.0

sudo python -m jcc \ --jar boilerpipe-1.2.0.jar \ --classpath lib/nekohtml-1.9.13.jar \ --classpath lib/xerces-2.9.1.jar \ --package java.net \ java.net.URL \ --python boilerpipe --build --install

使用:

import boilerpipe

jars = ':'.join(('lib/nekohtml-1.9.13.jar', 'lib/xerces-2.9.1.jar'))

boilerpipe.initVM(boilerpipe.CLASSPATH+':'+jars)

extractor = boilerpipe.ArticleExtractor.getInstance()

url = boilerpipe.URL('http://readthedocs.org/docs/jcc')

extractor.getText(url)

3. 各种Python正文抽取工具比较

各种Python正文抽取工具比较