Tokens

The C Programming Language (K&R, second edition)
C Traps and Pitfalls

The word token refers to a part of a program that plays much the same role as a word in a sentence.

In some sentence it means the same thing every time it appears.

The same sequence of characters can belong to one token in one context and an entirely different token in another contex.

The part of a compiler that breaks a program up into tokens is often called a lexical analyzer.

There are six classes if tokens: identifier, keywords, constants, string literals, operators, and other seperators.

Blanks, horizontal and vertical tabs, newlines, formfeeds, and comments are ignored except as they separate tokens.

If the input stream has been separated into tokens up to a given character, the next token is the longest string of characters that could constitute a token.