JavaCC SPECIAL_TOKEN option

SPECIAL_TOKEN are special tokens, that are bound to a token; they are referred from token.specialToken fields.

For example:

Grammar define comments as SPECIAL_TOKEN.

SKIP :
{
  " "
| "\t"
| "\n"
| "\r"
}

/* COMMENTS */
SPECIAL_TOKEN :
{
  <SINGLE_LINE_COMMENT: "--" (~["\n","\r"])* ("\n"|"\r"|"\r\n")>
| <MULTI_LINE_COMMENT: "/*" (~["*"])* "*" ("*" | (~["*","/"] (~["*"])* "*"))* "/">
}


SimpleNode Input() :
{}
{
 P()
 <EOF>
 { return jjtThis; }
}

void P() :
{}
{
  ( A() | B() | C() )+
}

So for this input:

-- line comment 1
A /* field comment 1 */ B
-- line comment 2
/* field comment 2 */
C
-- line comment 3

These inputs will generate 3 TOKEN, A, B, C

A.next = B;

B.next = C;

C.next = null;

And 5 comment special tokens; the comment tokens are bound to A, B, or C

A.specialToken = "--line comment 1"

  "--line comment 1".next = A

B.specialToken = "/* field comment 1 */"

  "/* field comment 1 */".next = B

C.specialToken = "/* field comment 2 */"

  "/* field comment 2 */".next = C

  "/* field comment 2 */".specialToken = "--line comment 2"

  "--line comment 2".next = "/* field comment 2 */"

But the problem is where is "--line comment 3", there is no Token after it ?

Fortunately, there is a <EOF> token at the end, <EOF>.next = "--line comment 3".

So the final token train is:

lc = line comment

fc = field comment