David Holroyd wrote:
ANTLR 2 provides CommonHiddenStreamToken et al, which I try to use for layout-preserving code rewriting. I've just been looking at v3 to see if manipulating ASTs containing hidden tokens has become any easier, but I don't see equivalent classes. Is this because there's a radically different way to do this which I've not spotted, or because it's just not implemented yet?
ANTLR v3 in a sense sends all tokens to the parser, with the hidden tokens on different "channels". this is done with a simple channel number in each token object. The commonTokenStream objct filters for only those tokens of interest to the parser, but keeps all tokens around. At any point, you can walk forwards/backwards in the buffer looking for stuff of interest on other channels.
As for trees, I automatically track start/stop token indexes for all rule invocations and hence I know the precise range of tokens associated with each rule tree construction. This includes all hidden channels simply because they are in range. Now, it does not track whitespace etc... before/after the rule...those will presumably be encompassed by other rule's AST construction.