【编译原理】Lexical Analysis_Compiler_Week 2_Stanford University (词法分析)

课程源地址

第二周(上) 词法分析

Week 2.1: Lexical Analysis

标记类¹

Token Class(or class)

• In English:
  Noun, verb, adjective, …
• In a programming language:
  Identifier, keywords, ‘(’, ‘)’, numbers, …

标记类对应于字符串集

token classes correspond to sets of strings

• 标识符(Identifier):
  — 以字母开头的，字母或数字字符串
  strings of letters or digits, starting with a letter
• 整数(Integer):
  — 一个非空的字符串
  a non-empty string of digits
• 关键字(Keyword):
  — “else” 或 “if” 或 “begin” 或…
  “else” or “if” or “begin” or …
• 空白符(Whitespace)
  — 一个非空的空白序列，换行符和制表符
  a non-empty sequence of blanks, newlines, and tabs

根据标记类对程序子串进行分类

classify program substrings according to role

将标记传递给解析器

communicate tokens to the parser

词法分析例子

LA Example

• 目标是分割字符串。这是通过从左到右阅读来实现的，一次识别一个标记。
  The goal is to partition the string. This is implemented by reading left-to-right, recognizing one token at a time.
• “Lookahead” 可能需要决定一个标记的结束为止和下一个标记的开始位置。
  “Lookahead” may be required to decide where one token ends and the next token begins.

PL/1² 不保留关键字

PL/1 keywords are not reserved

Example I

Example II

Example III

正则语言

Regular Languages

词法结构 = 标记类

Lexical structure = token classes

我们必须说明标记类中有哪些字符串集

We must say what set of strings is in a token class

• Use regular languages

正则表达式

Regular Expressions

• Single character ′ c ′ = { " c " } 'c' = \left\{ "c" \right\} ′c′={"c"}
• Epsilon( the empty string, not empty ) ε = { " " } ε = \left\{ "" \right\} ε={""}
• Union A + B = { a ∣ a ∈ A } ∪ { b ∣ b ∈ B } A+B = \left\{ a| a∈A \right\} ∪ \left\{ b| b∈B \right\} A+B={a∣a∈A}∪{b∣b∈B}
• Concatenation A B = { a b ∣ a ∈ A ∧ b ∈ B } AB = \left\{ ab | a∈A ∧ b∈B \right\} AB={ab∣a∈A∧b∈B}
• Iteration $$A^{\ast }=\bigcup _{i\ge 1}{A}^i$$ $$A^i=\underset{itimes}{\underbrace{A\cdots A}}$$

例子: ∑ = { 0 , 1 } \sum = \left\{ 0, 1\right\} ∑={0,1}

• $1^{\ast }={\bigcup }_{i\ge 0}{1}^i=""+1+11+111+1111+\cdots =allstringsof1$
• ( 1 + 0 ) 1 = { a b ∣ a ∈ 1 + 0 ∧ b ∈ 1 } = { 11 , 01 } (1+0)1 = \left\{ ab| a \in 1+0 \land b\in1 \right\} = \left\{ 11, 01 \right\} (1+0)1={ab∣a∈1+0∧b∈1}={11,01}
• $0^{\ast }+1^{\ast }=\left\{0^i|i\ge 0\right\}\cup \left\{1^i|i\ge 0\right\}$
• $(0+1{)}^{\ast }={\bigcup }_{i\ge 0}(0+1{)}^i="",0+1,(0+1)(0+1),\cdots ,\underset{itimes}{\underbrace{(0+1)\cdots (0+1)}}=allstiringof0+1$
  There is more than one way to write down the same set, for example, $1^{\ast }=1^{\ast }+1$, $(1+0)1=11+10$

正则语言总结

正则表达式指定正则语言

Regular expressions specify regular languages.

五个结构

— Two base cases

• empty and 1-character strings

— Three compound expressions

• union, concatenation, iteration

形式语言

Formal Languages

定义

形式语言是一个字母表上的某些有限长字符串的集合。
Def. Let $\sum$ be a set of characters (an alphabet). A language over $\sum$ is a set of strings of characters drawn form $\sum$.

对比

— In English

• Alphabet = English characters
• Language = English sentences

— In rigorous formal languages

• Alphabet = ASCII
• Language = C programs

含义函数L将语法映射到语义

为什么使用Meaning function

• 明确什么是语法，什么是语义
  Makes clear what is syntax, what is semantics.
• 允许我们将符号视为一个单独的问题，例如阿拉伯数字比罗马数字在计算上更方便
  Allows us to consider notation as a separate issue.
• 因为表达和意义不是1-1
  Because expressions and meanings are not 1-1.

含义是多对一的

Meaning is many to one, never one to many.
That is many expressions correspond to one meaning, otherwise it will cause ambiguity.

词法规范

Keyword: “if” or “else” or “then” or …

${}^{\prime }{i}^{\prime \prime }{f}^{\prime }{+}^{\prime }{e}^{\prime \prime }{l}^{\prime \prime }{s}^{\prime \prime }{e}^{\prime }{+}^{\prime }{t}^{\prime \prime }{h}^{\prime \prime }{e}^{\prime \prime }{n}^{\prime }$
${}^{\prime }i{f}^{\prime }{+}^{\prime }els{e}^{\prime }{+}^{\prime }the{n}^{\prime }+\cdots$

Integer: a non-empty string of digits

$digit{=}^{\prime }{0}^{\prime }{+}^{\prime }{1}^{\prime }{+}^{\prime }{2}^{\prime }{+}^{\prime }{3}^{\prime }{+}^{\prime }{4}^{\prime }{+}^{\prime }{5}^{\prime }{+}^{\prime }{6}^{\prime }{+}^{\prime }{7}^{\prime }{+}^{\prime }{8}^{\prime }{+}^{\prime }{9}^{\prime }$
$digitdigi{t}^{\ast }ordigi{t}^{+}$

Identifier: strings of letters or digits, starting with a letter

$letter=[a-zA-Z]$
$letter(letter+digit{)}^{\ast }$

Whitespace: a non-empty sequence of blanks, newlines, and tabs

${(}^{\prime }{}^{\prime }{+}^{\prime }\backslash n^{\prime }{+}^{\prime }\backslash t^{\prime }{)}^{+}$

例子

Example I : $anyone@cs.stanford.edu$

$lette{r}^{+}{}^{\prime }{@}^{\prime }lette{r}^{+}{}^{\prime }{.}^{\prime }lette{r}^{+}{}^{\prime }{.}^{\prime }lette{r}^{+}$

Example II: Pascal

$digit{=}^{\prime }{0}^{\prime }{+}^{\prime }{1}^{\prime }{+}^{\prime }{2}^{\prime }{+}^{\prime }{3}^{\prime }{+}^{\prime }{4}^{\prime }{+}^{\prime }{5}^{\prime }{+}^{\prime }{6}^{\prime }{+}^{\prime }{7}^{\prime }{+}^{\prime }{8}^{\prime }{+}^{\prime }{9}^{\prime }$
$digits=digi{t}^{+}$
$opt\_fraction={(}^{\prime }{.}^{\prime }digits)+ϵ={(}^{\prime }{.}^{\prime }digits)?$
$opt\_exponent={(}^{\prime }{E}^{\prime }{(}^{\prime }{+}^{\prime }{+}^{\prime }{-}^{\prime }+ϵ)digits)+ϵ={(}^{\prime }{E}^{\prime }{(}^{\prime }{+}^{\prime }{+}^{\prime }{-}^{\prime })?digits)?$
$num=digitsopt\_fractionopt\_exponent$

正则表达式描述许多有用的语言

Regular expressions describe many useful languages

• email
• phone numbers
• file names

正则语言是语言规范

Regular languages are a language specification

• We still need an implementation

总结词法规范

• At lease one: $A^{+}\equiv A{A}^{\ast }$
• Union: $A|B\equiv A+B$
• Option: $A?\equiv A+ϵ$
• Range: ${}^{\prime }{a}^{\prime }{+}^{\prime }{b}^{\prime }+\cdots {+}^{\prime }{z}^{\prime }=[a-z]$
• Excluded range: $complementof[a-z]\equiv [\hat{}a-z]$

词法规范步骤

1. 给每个标记类的字根写一个正则表达式
   Write a rexp for the lexemes of each token class

• Number = $digi{t}^{+}$
• Keyword = ${}^{\prime }i{f}^{\prime }{+}^{\prime }els{e}^{\prime }+\cdots$
• Identifier = $letter(letter+digit{)}^{\ast }$
• OpenPar = ‘(’
• $\cdots$

2. 构造R, 匹配所有标记类的所有字根
   Construct R, matching all lexemes for all tokens
   $R=Keyword+Identifier+Number+\cdots$
   $=R1+R2+\cdots$

3. Let input be $x_1...x_n$
   For $1\le i\le n$ check
   $x_1...x_i\in L(R)$

4. If success, then we know that
   $x_1...x_n\in L(R_j)forsomej$

5. Remove $x_1...x_i$ from input and go to 3

FAQs(frequently asked questions)

I: 使用了多少输入?

How much input is used?
$x_1...x_i\in L(R)$
$x_1...x_j\in L(R)$
$i\ne j$
Answer: "Maximal munch"

II: 匹配了哪个标记类？

Which token is used?
$x_1....x_i\in L(R)R=R_1+\cdots +R_N$
$x_1...x_i\in L(R_j)$
$x_1...x_i\in L(R_k)$

For example, $if\in \{\begin{array}{llc}L(Keywords)& & Keywords{=}^{\prime }i{f}^{\prime }{+}^{\prime }els{e}^{\prime }+\cdots \\ L(Identifiers)& & Identifiers=letter(letter+digit{)}^{\ast }\end{array}$
Answer: Choose the one listed first

III: 如果没有规则匹配怎么办？

What if no rule matches?
$x_i...x_i\notin L(R)$

Answer: It is important for compilers to do good error handling. They cannot simply crash. The best solution for lexical analysis is to not do this so don’t let this ever happen. And so what we wanted to do instead is to write a category of error strings. Put it last in priority, then it will only match if no earlier regular expression match and only catch the error strings.
$ERROR=allstringsnotinthelexicalspecificationofthelanguage$

知识点梳理

正则表达式是字符串模式的简明表示法。

Regular expressions are a concise notation for string patterns.

在词法分析使用中需要小扩展

Use in lexical analysis requires small extensions

• To resolve ambiguities
  • matches as long as possible
  • highest priority match
    这只是通过在文件中按顺序列出它们而完成的，并且首先列出的那些优先于后面列出的那些。
    This has done just by listing them in order in a file and the ones listed first have higher priority over the ones listed later.
• To handle errors
  捕获所有正则表达式，它吸收所有可能的错误字符串并赋予它最低优先级，以便只有在没有有效的令牌类匹配某些输入时才会触发它。
  Catch all regular expression that soaks up all the possible erroneous strings and give it the lowest priority so that it only triggers if no valid token class matches some piece of the input.

注：个人英文水平有限，如有错误请指正，谢谢！

---

1. https://baike.baidu.com/item/TOKEN/2615248 ↩︎

2. https://baike.baidu.com/item/PL%2FI ↩︎