ANTLR4 Common Lexical Structures

FROM: Page 78, The Definitive ANTLR 4 Reference, 2nd Edition

Token Category	Description and Examples
Punctuation	The easiest way to handle operators and punctuation is to directly reference them in parser rules. call : ID '(' exprList ')' ; Some programmers prefer to define token labels such as LP (left parenthesis) instead. call : ID LP exprList RP ; LP : '(' ; RP : ')' ;
Keywords	Keywords are reserved identifiers, and we can either reference them directly or define token types for them. returnStat : 'return' expr ';'
Identifiers	Identifiers look almost the same in every language, with some variation about what the first character can be and whether Unicode characters are allowed. ID : ID_LETTER (ID_LETTER | DIGIT)* ; // From C language fragment ID_LETTER : 'a'..'z'|'A'..'Z'|'_' ; fragment DIGIT : '0'..'9' ;
Numbers	These are definitions for integers and simple floating-point numbers. INT : DIGIT+ ; FLOAT : DIGIT+ '.' DIGIT* | '.' DIGIT+ ;
Strings	Match double-quoted strings. STRING : '"' ( ESC | . )*? '"' ; fragment ESC : '\\' [btnr"\\] ; // \b, \t, \n etc...
Comments	Match and discard comments. LINE_COMMENT : '//' .*? '\n' -> skip ; COMMENT : '/*' .*? '*/' -> skip ;
Whitespace	Match whitespace in the lexer and throw it out. WS : [ \t\n\r]+ -> skip ;