How do I access hidden tokens from parser?
Rules in the lexer can send tokens to the parser on different "channels". See How do I track whitespace, comments, and other hidden channels during AST construction? For example,
COMMENT : '/*' .* '*/' {$channel=HIDDEN;} ;
But, how do you access them from the parser? All of the tokens come to the parser via a TokenStream
, usually CommonTokenStream
. So, all you have to do is scan forwards and backwards from real tokens looking for off channel tokens. Use the get(i) method where i is the token index to get a token from the stream. Each token stores the token index, which means you know where it is in the stream.
There are a number of helper methods in CommonTokenStream
; E.g.,
/** Given a start and stop index, return a List of all tokens in * the token type BitSet. Return null if no tokens were found. This * method looks at both on and off channel tokens. */ public List getTokens(int start, int stop, BitSet types);
Here is a utility method within CommonTokenStream
that moves the input cursor to the next real token in the stream. You can cut and paste the loop in there, modifying it to get all off channel tokens that appear after a particular index:
/** Given a starting index, return the index of the first on-channel * token. */ protected int skipOffTokenChannels(int i) { int n = tokens.size(); while ( i<n && ((Token)tokens.get(i)).getChannel()!=channel ) { i++; } return i; }