Motivation for complete rewrite

I must reimplement ANTLR v3 in v3, just like I did recently for ST (yielding ST v4). Besides being untidy, important projects like eclipse cannot include ANTLR at the moment due to license restrictions on it's v2 dependency Jim Idle has graciously build a proper ANTLR parser, AST builder, and AST tree grammar that I can use to get us to v3.

While I tried to do as much re-factoring as possible while developing ANTLR v3, most of it was tactical. At some point, strategic re-factoring (rewriting whole sections or all) becomes necessary. For example, I literally had to jam grammar composition into the tool, leaving it fragile. It's becoming hard to fix things and add new features. A lot of the current features have been added while writing the first and second books. Doing so simultaneously was valuable from a feature and functionality point of view, but not from a code cleanliness point of view.

I have some important new features such as the better expression grammar stuff that would be inconvenient to implement in the current code base.

Trying to jam a new ANTLR front end into the existing semantics engine and grammar analysis engine might be challenging. It would be hard to get all of the various pieces to hook up properly.

Code generation. I learned a lot while building ANTLR v3's re-targetable code generation system; as a result, it's not exactly the cleanest thing in the world. I also would like to restructure the generated parsers. E.g., I'm going back to exception handling for backtracking since it is much cleaner and likely faster, as long as I reuse the same exception object during backtracking. targets without exception handling like C can continue to use the existing "if (failed) return" concept. I'd also like to manage my own arguments and return values stack so that rules predicated upon parameters don't fail during code generation (when I move a parameter reference outside of the defining function). Even if we stay with the exact same code generator and templates, all of you target developers will have some changes to make to keep in sync regardless.

I have 8 months left on my sabbatical for full-time code development (i.e., no teaching duties). Sabbaticals come once every seven years. [Oh, feel free to start hating me now!] I suspect that I have about another year and a half or two years before I need to start writing again. First thing would probably be a v4 update for the definitive ANTLR reference guide.