Python自然语言处理学习笔记(64)： 7.5 命名实体识别

最新推荐文章于 2022-07-27 20:25:26 发布

a13393665983

最新推荐文章于 2022-07-27 20:25:26 发布

阅读量200

点赞数

文章标签： python 人工智能

原文链接：http://www.cnblogs.com/lexus/archive/2012/05/16/2503802.html

版权

Python自然语言处理学习笔记(64)： 7.5 命名实体识别

7.5 Named Entity Recognition 命名实体识别

At the start of this chapter, we briefly introduced named entities (NEs). Named entities are definite（确定的） noun phrases that refer to specific types of individuals, such as organizations, persons, dates, and so on（命名实体是明确的名词短语，指的是个体的具体类型，例如组织，个人，日期等等）. Table 7.4 lists some of the more commonly used types of NEs. These should be self-explanatory（不言自明的）, except for "Facility": human-made artifacts（人工产品） in the domains of architecture and civil engineering（土木工程）; and "GPE": geo-political entities（地缘政治实体） such as city, state/province, and country.

NE Type	Examples
ORGANIZATION	Georgia-Pacific Corp., WHO
PERSON	Eddy Bonte, President Obama
LOCATION	Murray River, Mount Everest
DATE	June, 2008-06-29
TIME	two fifty a m, 1:30 p.m.
MONEY	175 million Canadian Dollars, GBP 10.40
PERCENT	twenty pct, 18.75 %
FACILITY	Washington Monument, Stonehenge
GPE	South East Asia, MidlothianTable 7.4: Commonly Used Types of Named Entity

The goal of a named entity recognition (NER) system is to identify all textual mentions of the named entities. This can be broken down into（分解成） two sub-tasks: identifying the boundaries of the NE, and identifying its type（识别NE的边界和它们的类型）. While named entity recognition is frequently a prelude（序曲） to identifying relations in Information Extraction, it can also contribute to other tasks. For example, in Question Answering (QA), we try to improve the precision of Information Retrieval by recovering not whole pages, but just those parts which contain an answer to the user's question. Most QA systems take the documents returned by standard Information Retrieval, and then attempt to isolate the minimal text snippet in the document containing the answer. Now suppose the question was Who was the first President of the US?, and one of the documents that was retrieved contained the following passage:

(5)

The Washington Monument is the most prominent structure in Washington, D.C. and one of the city's early attractions. It was built in honor of George Washington, who led the country to independence and then became its first President.

Analysis of the question leads us to expect that an answer should be of the form X was the first President of the US, where X is not only a noun phrase, but also refers to a named entity of type PER. This should allow us to ignore the first sentence in the passage. While it contains two occurrences of Washington, named entity recognition should tell us that neither of them has the correct type.

How do we go about identifying named entities? One option would be to look up each word in an appropriate list of names. For example, in the case of locations, we could use a gazetteer（地名词典）, or geographical dictionary, such as the Alexandria Gazetteer or the Getty Gazetteer. However, doing this blindly runs into problems, as shown in Figure 7.12.

Figure 7.12: Location Detection by Simple Lookup for a News Story: Looking up every word in a gazetteer is error-prone（易错配的）; case distinctions may help, but these are not always present.

Observe that the gazetteer has good coverage of locations in many countries, and incorrectly finds locations like Sanchez in the Dominican Republic and On in Vietnam. Of course we could omit such locations from the gazetteer, but then we won't be able to identify them when they do appear in a document.

It gets even harder in the case of names for people or organizations. Any list of such names will probably have poor coverage. New organizations come into existence every day, so if we are trying to deal with contemporary（当代的） newswire or blog entries, it is unlikely that we will be able to recognize many of the entities using gazetteer lookup.

Another major source of difficulty is caused by the fact that many named entity terms（措辞） are ambiguous. Thus May and North are likely to be parts of named entities for DATE and LOCATION, respectively, but could both be part of a PERSON; conversely Christian Dior looks like a PERSON but is more likely to be of type ORGANIZATION. A term like Yankee will be ordinary modifier in some contexts, but will be marked as an entity of type ORGANIZATION in the phrase Yankee infielders.

Further challenges are posed by multi-word names like Stanford University, and by names that contain other names such as Cecil H. Green Library andEscondido Village Conference Service Center. In named entity recognition, therefore, we need to be able to identify the beginning and end of multi-token sequences.

Named entity recognition is a task that is well-suited（适当的） to the type of classifier-based approach that we saw for noun phrase chunking. In particular, we can build a tagger that labels each word in a sentence using the IOB format, where chunks are labeled by their appropriate type. Here is part of the CONLL 2002 (conll2002) Dutch training data:

Eddy N B-PER

Bonte N I-PER

is V O

woordvoerder N O

van Prep O

diezelfde Pron O

Hogeschool N B-ORG

. Punc O

In this representation, there is one token per line, each with its part-of-speech tag and its named entity tag. Based on this training corpus, we can construct a tagger that can be used to label new sentences; and use the nltk.chunk.conlltags2tree() function to convert the tag sequences into a chunk tree.

NLTK provides a classifier that has already been trained to recognize named entities, accessed with the function nltk.ne_chunk(). If we set the parameter binary=True, then named entities are just tagged as NE; otherwise, the classifier adds category labels such as PERSON, ORGANIZATION, and GPE.

>>> sent = nltk.corpus.treebank.tagged_sents()[22]

>>> print nltk.ne_chunk(sent, binary=True)

The/DT

(NE U.S./NNP)

is/VBZ

one/CD

...

according/VBG

to/TO

(NE Brooke/NNP T./NNP Mossman/NNP)

...)

>>> print nltk.ne_chunk(sent)

The/DT

(GPE U.S./NNP)