ANTLR v3 C# Code generator and Runtime library

Johannes Luber (Maintainer)
jaluber AT gmx.de

Kunle Odutola
kunle UNDERSCORE odutola AT hotmail.com

Micheal Jordan

Contents

• #Architecture
• #Status
  • #Version 3.0.x
  • #Version 3.1.x
• #Target Platforms
  • #CSharp target (versions 3.0.x and 3.1.x only)
  • #CSharp2 target (version 3.1.x and later)
  • #Runtime Location
  • #Supported build tools
• #Performance
• #Usage
  • #Specify that C# code should be generated for a grammar
  • #List of generated C# source files
  • #Specify a C# namespace for your recognizer
  • #Members blocks vs C# v2 Partial Classes
  • #Syntactic differences from the Java target
  • #Debugging with ANTLRWorks

Architecture

The C# target consists of a set of code generation templates and a runtime library (written in C#) for the .NET/CLR platform. The C# code generation templates and the .NET/CLR runtime library are modeled on the Java version. As a consequence, the C# target supports features such as grammar development/prototyping and remote debugging with the superb ANTLRWorks integrated grammar development editor. Given ANTLRWorks popularity, this is very a important feature for ANTLR users.

The .NET/CLR runtime library currently consists of two assemblies named Antlr3.Runtime.dll and Antlr3.Utility.dll. All projects that include an ANTLR v3.x Lexer, Parser or TreeParser must include a reference to:

1. Antlr3.Runtime.dll - the ANTLR v3.x .NET/CLR runtime library
2. Antlr3.Utility.dll - OPTIONAL - only required if you use the DOTTreeGenerator class (correct as of May 2008)

No other references are required (except to the manadatory System assembly).

For projects that use the in-built integration with StringTemplate, the following assemblies must also be referenced:

1. StringTemplate.dll - the C# StringTemplate v3.x library (a.k.a ST# v3.x)
2. antlr.runtime.dll - the ANTLR v2.7.x .NET/CLR runtime library (ST# v3.x was developed with ANTLR v2.7.x)

Status

In general, development progress on the C# target proceeds sporadically.

Version 3.0.x

As of September 2007, the C# code generator and runtime are NOT in sync with the latest release and development versions of the ANTLR tool and Java language target. The latest release of the v3.0.x C# code generator and .NET/CLR runtime was developed for the ANTLR v3.0 release from July 2007. Nevertheless, no major problems have been reported by those using the C# codegen and .NET/CLR runtime with ANTLR v3.0.1 since that version was released in August 2007.

Version 3.1.x

As of October 2007, the ANTLR source depot contains an early pre-beta release of the C# codegen and .NET/CLR runtime for the upcoming ANTLR v3.1 release. Starting from the end of October, ANTLR daily builds of v3.1 have been available for those wishing to test the C# support with the new features of ANTLR v3.1.

As of May 2008, the C# target is in sync with the Java target of the upcoming ANTLR v3.1 beta release. This new version of the C# target breaks source compatibility with previous versions (including previous v3.1 builds). To a certain extent, regeneration of grammars does help, but certain fields have been renamed to follow .NET conventions, which means PascalCase (e.g. .tree is now .Tree). The exception is .st, which is now .ST. Additionally, a new target named CSharp2 has been introduced in addition to the existing CSharp target. The reason for this is three-fold:

1. Firstly, CSharp can retain it's compatibility with C# v1 and the .NET/CLR v1.1 platform. CSharp is restricted to C# v1 features and doesn't take advantage of any C# v2+ features in the code it generates. As a consequence the runtime sources will now require twice the number of files (one set each for C# v1 and v2+), effectively doubling the amount of maintenance.
2. Secondly, a certain bug fix requires a C# v2 feature or, a change to the templates for each occurrence of the bug (and this has to be done by the user). Further details of this issue can be found in the #Known Issues section below.
3. Thirdly, because maintaining the backwards compatibility sucks majorly, creating the new CSharp2 target allows the existing CSharp target to be deprecated without forcing the issue by simply abandoning C# v1 .NET/CLR v1.1 compatibility immediately.

Introducing the new target allowed changes to be made to it's distinct code generation templates without fear of breaking anything else. Further, as working on further enhancements to CSharp2 will at least break binary compatibility in the .NET/CLR runtime which is currently shared with CSharp, most of the changes will be done for a future ANTLR v3.2 realease. During the life-time of ANTLR v3.1, the public API of the C# target(s) will be frozen. If you wish to future-proof your grammar, you should change them to use the new CSharp2 target. The original CSharp target that uses only C# v1 and .NET/CLR v1.1 features is deprecated and, the current plan is to remove it for the ANTLR v3.3 release.

The C# code generation templates and the .NET/CLR runtime library are feature complete for the CSharp and CSharp2 targets. Both C# targets leverage the existing C# StringTemplate implementations to support the broadest range of the features that ANTLR provides. The long open issue of unit tests has finally been tackled with the adoption of MbUnit and the inclusion (in the v3.1 version) of a wide range of tests for the runtime library. As before, basic sanity checks will done by ensuring that the sample grammars in the examples-v3 archive function correctly. This is currently a work-in-progess for the v3.1 release.

Target Platforms

CSharp target (versions 3.0.x and 3.1.x only)

Microsoft .NET v1.1 and later
Mono v1.0 and later

CSharp2 target (version 3.1.x and later)

Microsoft .NET v2.0 and later
Mono v1.2 and later

Runtime Location

The compiled libraries are found in the distribution under the directory "runtime/csharp" (or "runtime/csharp/bin" in source distributions). Both targets use the same .NET/CLR runtime. Intermediary builds may not have the current version and can be compiled by using the build tools.

Supported build tools

Microsoft Visual Studio 2003 and 2005
Nant v0.85

Performance

The V3 target generates code that is easily faster than that generated by the V2 target (especially the lexers). We probably won't be able to match the bare-metal performance of the code generated by Jim Idle's C target or Ric Klaren's C++ target, but we expect to be very competitive with the other targets.

Usage

This section is NOT a tutorial on how to use either C# or ANTLR v3.x. It assumes that you are familiar with the concepts involved in developing ANTLR v3.x grammars and, in building and using C# programs and assemblies.

Specify that C# code should be generated for a grammar

To specify that the ANTLR tool should generate C# code (rather than the default of generating Java code) for a grammar, set the grammar-level option named language to the value CSharp2 as shown below:

grammar MyGrammar;

options
{
    language=CSharp2;
}

// rest of grammar follows
....

For the example grammar named MyGrammar above, the grammar file would typically be named MyGrammar.g. The grammar filename (excluding the extension) must match the grammar name as declared with the grammar directive in the file.

List of generated C# source files

For an example grammar named MyGrammar, the following table list the files that would be generated by ANTLR using the CSharp2 (and CSharp) target.

grammar MyGrammar;

options
{
    language=CSharp2;
}
// rest of grammar follows
....

lexer grammar MyGrammar;

options
{
    language=CSharp2;
}
// rest of grammar follows
....

parser grammar MyGrammar;

options
{
    language=CSharp2;
}
// rest of grammar follows
....

tree grammar MyGrammar;

options
{
    language=CSharp2;
}
// rest of grammar follows
....

Specify a C# namespace for your recognizer

You can specify that your generated recognizer should be declared within a specific namespace as shown below. By default all recognizers are generated as top-level types with no enclosing namespace.

grammar MyGrammar;

options
{
    language=CSharp2;
}

@parser::namespace { My.Custom.NameSpace.For.Parser.In.Combined.Grammar } // Or just @namespace { ... }

@lexer::namespace { My.Custom.NameSpace.For.Lexer.In.Combined.Grammar }

// rest of grammar follows
....

lexer grammar MyGrammar;

options
{
    language=CSharp2;
}

@namespace { My.Custom.NameSpace.For.Lexer }

// rest of grammar follows
....

parser grammar MyGrammar;

options
{
    language=CSharp2;
}

@namespace { My.Custom.NameSpace.For.Parser }

// rest of grammar follows
....

tree grammar MyGrammar;

options
{
    language=CSharp2;
}

@namespace { My.Custom.NameSpace.For.TreeParser }

// rest of grammar follows
....

Members blocks vs C# v2 Partial Classes

In addition to the the use of the @members block to define class members inline within the grammar file, the CSharp2 target also supports the use of the C# v2 partial classes feature. This has the additional advantage that you can use your favourite editor for C# to define class members since ANTLRworks doesn't support syntax highlighting for target languages.

Syntactic differences compared to grammars and the Java target

ANTLR grammars refer often to attributes of the tokens and rules. These attributes do not change because their use in a grammar targeted for C#. But when accessing those attributes from driver programs and other support files, the names used in the ANTLR grammar won't work. The C# target uses the official coding guidelines which are different from the general documentation.The following table details the status quo for ANTLR v3.1.

Known Issues

A problem, you may encounter while using the CSharp target, is that value types are initialized with null in the generated code (happens e.g. while using labels). The cause lies in the following definition of CSharp.stg:

csharpTypeInitMap ::= [
    "int":"0",
    "uint":"0",
    "long":"0",
    "ulong":"0",
    "float":"0.0",
    "double":"0.0",
    "bool":"false",
    "byte":"0",
    "sbyte":"0",
    "short":"0",
    "ushort":"0",
    "char":"char.MinValue",
    default:"null" // anything other than an atomic type
]

As you can see, only the in-built value types are supported (or can be reasonably supported). Since adding value types to this map is an open-ended task, the maintainer does not make any changes in that structure for all users. Any changes have to be done by the user locally (and repeatedly for each new version of ANTLR). It is recommend that users switch to the CSharp2 target (which requires C# v2+ and .NET v2.0 or higher) as the problem has been fixed there in an environment-independent manner.

Debugging with ANTLRWorks

The ANTLRWorks tool is written in Java and is only able to debug Java recognizers natively. Nevertheless, you can debug your C# recognizers with ANTLRWorks by using the Remote Debugging feature of ANTLRWorks. In ANTLRWorks, Remote Debugging works by connectiong to a running instance of a debug-instrumented recognizer (generated with the -debug switch to ANTLR) over the network.

To debug your C# recognizer with ANTLRWorks:

1. Generate a debuggable version of your recognizer by specifying the -debug option to ANTLR
2. Create a driver program that creates your recognizer and runs some test input through it (see the examples-v3 archive for sample driver programs)
3. Compile your driver and recognizer to produce your executable file(s)
4. Execute your driver program (it will launch your recognizer and appear to hang - it's just waiting for ANTLRWorks to connect)
5. Start ANTLRWorks (or switch to it if it is already running) and click the menu Debugger|Debug Remote...
6. Click Connect to accept the default host and port values (localhost and 49153 respectively)
7. ANTLRWorks should now start debugging your recognizer!

   Warning

   ANTLRWorks remote debugging has only been tested for C# Parsers. TreeParsers and Lexers should work but...