NLP基础概览 + Spell Correction with Noisy Channel

往期文章链接目录

NLP = NLU + NLG

• NLU: Natural Language Understanding
• NLG: Natural Language Generation

NLG may be viewed as the opposite of NLU: whereas in NLU, the system needs to disambiguate the input sentence to produce the machine representation language, in NLG the system needs to make decisions about how to put a concept into human understandable words.

Classical applications in NLP

• Question Answering

• Sentiment Analysis

• Machine Translation

• Text Summarization: Text summarization refers to the technique of shortening long pieces of text. The intention is to create a coherent and fluent summary having only the main points outlined in the document. It involves both NLU and NLG. It requires the machine to first understand human text and overcome the long distance dependence problems (NLU) and then generate human understandable text (NLG).

  • Extraction-based summarization: The extractive text summarization technique involves pulling keyphrases from the source document and combining them to make a summary. The extraction is made according to the defined metric without making any changes to the texts. The grammar might not be right.
  • Abstraction-based summarization: The abstraction technique entails paraphrasing and shortening parts of the source document. The abstractive text summarization algorithms create new phrases and sentences that relay the most useful information from the original text — just like humans do.
  • Therefore, abstraction performs better than extraction. However, the text summarization algorithms required to do abstraction are more difficult to develop; that’s why the use of extraction is still popular.

• Information Extraction: Information extraction is the task of automatically extracting structured information from unstructured and/or semi-structured machine-readable documents and other electronically represented sources. QA uses information extraction a lot.

• Dialogue System

  • task-oriented dialogue system.

Text preprocessing

Tokenization

For Chinese, classical methods are forward max-matching and backward max-matching.

Shortcoming: Do not take semantic meaning into account.

Tokenization based on Language Modeling. Given an input, generate all possible way to split the sentence and then find the one with the highest possibility.

Unigram model:
$$P(s)=P(w_1)P(w_2)...P(w_k)$$

Bigram model:
$$P(s)=P(w_1)P(w_2|w_1)P(w_3|w_2)...P(w_k|w_{}$$