Building ANTLR with Maven

Sam Harwell - Dec. 7, 2012

Several phases of the Maven build exist for preparing proper releases. Each of the following variables must be configured or explicitly overridden/ignored.

Initial Setup

Before the first time Maven is used to build ANTLR 3 from source, the following command should be executed in the top-level directory of the source tree.

mvn -N install

Examples

Each of these examples is written as a "quick start" case, and assumes the following:

• You do not have GPG installed or configured
• You are building the project with Maven 3+ and JDK 6+
• You have a copy of Java 6 installed
• You may or may not have a copy of Java 5 installed

Windows

With Java 5 installed

mvn -Dgpg.skip=true -Duser.name="Your Name" -DskipTests=true "-Djava5.home=C:\Program Files (x86)\Java\jre5" "-Djava6.home=C:\Program Files (x86)\Java\jre6" install

Without Java 5 installed

mvn -Dgpg.skip=true -Duser.name="Your Name" -DskipTests=true "-Dbootclasspath.compile=C:\Program Files (x86)\Java\jre6\lib\rt.jar" "-Djava6.home=C:\Program Files (x86)\Java\jre6" install

Linux

With Java 5 installed

mvn -Dgpg.skip=true -Duser.name="Your Name" -DskipTests=true -Djava5.home=/path/to/jre5 -Djava6.home=/path/to/jre6 install

Without Java 5 installed

mvn -Dgpg.skip=true -Duser.name="Your Name" -DskipTests=true -Dbootclasspath.compile=/path/to/jre6/lib/rt.jar -Djava6.home=/path/to/jre6 install

Mac OSX

mvn -Dgpg.skip=true -Dbootclasspath.java5=/System/Library/Frameworks/JavaVM.framework/Versions/1.5/Classes/classes.jar -Dbootclasspath.java6=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Classes/classes.jar -DskipTests install

mvn -Dgpg.skip=true -Dbootclasspath.compile=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Classes/classes.jar -Dbootclasspath.java6=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Classes/classes.jar -DskipTests install

Artifact signing (GPG)

By default Maven will sign the build artifacts with GPG, and assumes you have your system configured according to Sonatype's Blog entry "How to Generate PGP Signatures with Maven". One of the following should be used each time Maven is used to build the project.

1. Skip the phase by passing -Dgpg.skip=true to Maven (required for users without GPG installed or configured).
2. Configure the GPG passphrase by passing -Dgpg.passphrase=passphrase to Maven (or simply -Dgpg.passphrase if your private key does not require a passphrase).
3. Run the Maven build without specifying one of the above, in which case signing will be enabled and Maven will ask you to enter the passphrase during the signing phase.

Bootstrap Classpath

The build is configured to pass the -bootstrap option to javac to ensure that the compiled artifacts will run with Java 5 and the test suites will run with Java 6, even when compiled with Java 7.

Compiled Artifacts: Java 5

Maven needs the path to a copy of rt.jar (Windows, Linux) or Classes.jar (OSX) from Java 5. This may be configured by any of the following methods:

1. (Windows, Linux) Set the JAVA5_HOME environment variable to the home folder of a Java 5 runtime installation. When using this option, the library rt.jar should be available in %JAVA5_HOME%\lib\rt.jar (Windows) or $JAVA5_HOME/lib/rt.jar (Linux).
2. (Windows, Linux) Pass the option -Djava5.home=path to Maven. The path should be in the same form as listed for the JAVA5_HOME enviroment variable, and takes precedence over the environment variable if both are specified.
3. (Windows, Linux, OSX) Pass the option -Dbootclasspath.java5=path to Maven. This should be the full path to rt.jar (Windows, Linux) or Classes.jar (OSX).
4. For users without a copy of Java 5 installed, the Java 5 validation may be skipped by passing the option -Dbootclasspath.compile=path to Maven. This should be the full path to rt.jar (Windows, Linux) or Classes.jar (OSX) for some version of Java newer than Java 5.

Compiled Test Suite: Java 6

Maven needs the path to a copy of rt.jar (Windows, Linux) or Classes.jar (OSX) from Java 6. This may be configured by any of the following methods:

1. (Windows, Linux) Set the JAVA6_HOME environment variable to the home folder of a Java 6 runtime installation. When using this option, the library rt.jar should be available in %JAVA6_HOME%\lib\rt.jar (Windows) or $JAVA6_HOME/lib/rt.jar (Linux).
2. (Windows, Linux) Pass the option -Djava6.home=path to Maven. The path should be in the same form as listed for the JAVA6_HOME enviroment variable, and takes precedence over the environment variable if both are specified.
3. (Windows, Linux, OSX) Pass the option -Dbootclasspath.java6=path to Maven. This should be the full path to rt.jar (Windows, Linux) or Classes.jar (OSX).
4. For users without a copy of Java 6 installed, the Java 6 validation may be skipped by passing the option -Dbootclasspath.testCompile=path to Maven. This should be the full path to rt.jar (Windows, Linux) or Classes.jar (OSX) for some version of Java newer than Java 6.

Additional Build Options

These options are not required for building ANTLR 3, but can be useful in certain scenarios. This is not a complete list of options which affect the build, but should include the most commonly used (or "interesting") options.

Jim Idle - March 2009

[out of date]

This file contains the build instructions for the ANTLR toolset as of version 3.1.3 and beyond.

The ANTLR toolset must be built using the Maven build system as this build system updates the version numbers and controls the whole build process. However, if you just want the latest build and do not care to learn anything about Maven, then visit the 'target' directories (for jars) under the depot mirror root here:

http://antlr.org/depot

If you are looking for the latest released version of ANTLR, then visit the downloads page on the main antlr.org website.

These instructions are mainly for the ANTLR development team, though you are free to build ANTLR yourself of course.

Source code Structure
-----------------------

The main development branch of ANTLR is stored within the Perforce SCM at:

//depot/code/antlr/main/...

release branches are stored in Perforce like so:

//depot/code/antlr/release-3.1.3/...

In this top level directory, you will find a master build file for Maven called pom.xml and you will also note that there are a number of subdirectories:

tool - The ANTLR tool itself
runtime/Java - The ANTLR Java runtime
runtime/X - The runtime for language target X
gunit - The grammar test tool
antlr3-maven-plugin - The plugin tool for Maven allowing Maven
projects to process ANTLR grammars.

Each of these sub-directories also contains a file pom.xml that controls the build of each sub-component (or module in Maven parlance).

Build Parameters
-----------------

Alongside each pom.xml (other than for the antlr3-maven-plugin), you will see that there is a file called antlr.config. This file is called a filter and should contain a set of key/value pairs in the same manner as Java properties files:

antlr.something="Some config thang!"

When the build of any component happens, any values in the antlr.config for the master build file and any values in the antlr.config file for each component are made available to the build. This is mainly used by the resource processor, which will filter any file it finds under: src/main/resources/** and replace any references such as ${antlr.something} with the actual value at the time of the build.

Building
--------

Building ANTLR is trivial, assuming that you have loaded Maven version 3.0.3 or better on to your build system and installed it as explained here:

http://maven.apache.org/download.html

Note that the ANTLR toolset will ONLY build with version 3.0.3 of Maven as of release 3.4.

If you are unfamiliar with Maven (and even if you are), the best resource for learning about it is The Definitive Guide:

http://www.sonatype.com/books/maven-book/reference/public-book.html

The instructions here assume that Maven is installed and working correctly.

If this is the first time you have built the ANTLR toolset, you will possibly need to install the master pom in your local repository (however the build may be able to locate this in the ANTLR snapshot or release repository). If you try to build sub-modules on their own (as in run the mvn command in the sub directory for that tool, such as runtime/Java), and you receive a message that maven cannot find the master pom, then execute this in the main (or release) directory:

mvn -N install

This command will install the master build pom in your local maven repository (it's ~/.m2 on UNIX) and individual builds of sub-modules will now work correctly.

To build then, simply cd into the master build directory (e.g. $P4ROOT//code/antlr/main) and type:

mvn -Dmaven.test.skip=true

Assuming that everything is correctly installed and synchronized, then ANTLR will build and skip any unit tests in the modules (the ANTLR tool tests can take a long time).

This command will build each of the tools in the correct order and will create the jar artifacts of all the components in your local development Maven repository (which takes precedence over remote repositories by default). At the end of the build you should see:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] ------------------------------------------------------------------------
[INFO] ANTLR Master build control POM ........................ SUCCESS [1.373s]
[INFO] Antlr 3 Runtime ....................................... SUCCESS [0.879s]
[INFO] ANTLR Grammar Tool .................................... SUCCESS [5.431s]
[INFO] Maven plugin for ANTLR V3 ............................. SUCCESS [1.277s]
[INFO] ANTLR gUnit ........................................... SUCCESS [1.566s]
[INFO] Maven plugin for gUnit ANTLR V3 ....................... SUCCESS [0.079s]
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11 seconds

However, unless you are using Maven exclusively in your projects, you will most likely want to build the ANTLR Uber Jar, which is an executable jar containing all the components that ANTLR needs to build and run parsers (note that at runtime, you need only the runtime components you use, such as the Java runtime and say stringtemplate).

Because the Uber jar is not something we want to deploy to Maven repositories it is built with a special invocation of Maven:

mvn -Dmaven.test.skip=true package assembly:assembly

Note that Maven will appear to build everything twice, which is a quirk of how it calculates the dependencies and makes sure it has everything packaged up so it can build the uber-jar assembly.

Somewhere in the build output (towards the end), you will find a line like this:

[INFO] Building jar: /home/jimi/antlrsrc/code/antlr/main/target/antlr-master-3.4-SNAPSHOT-completejar.jar

This is the executable jar that you need and you can either copy it somewhere or, like me, you can create this script (assuming UNIX) somewhere in your PATH:

#! /bin/bash
java -jar ~/antlrsrc/code/antlr/main/target/antlr-master-3.4-SNAPSHOT-completejar.jar $*

Version Numbering
-------------------

The first and Golden rule is that any pom files stored under the main branch of the toolset should never be modified to contain a release version number. They should always contain a.b.c-SNAPSHOT (e.g. 3.1.3-SNAPSHOT). Only release branches should have their pom version numbers set to a release version. You can release as many SNAPSHOTS as you like, but only one release version. However, release versions may be updated with a patch level: 3.1.3-1, 3.1.3-2 and so on.

Fortunately, Maven helps us with the version numbering in a number of ways. Firstly, the pom.xml files for the various modules do not specify a version of the artifacts themselves. They pick up their version number from the master build pom. However, there is a catch, because they need to know what version of the parent pom they inherit from and so they DO mention the version number. However, this does prevent accidentally releasing different versions of sub-modules than the master pom describes.

Fortunately once again, Maven has a neat way of helping us change the version. All you need do is check out all the pom.xml files from perforce, then modify the <version>a.b.c-SNAPSHOT</version> in the master pom. When the version number is correct in the master pom, you make sure your working directory is the location of the master pom and type:

mvn versions:update-child-modules

This command will then update the child pom.xml files to reflect the version number defined in the master pom.xml.

There is unfortunately one last catch here though and that is that the antlr3-maven-plugin and the gunit-maven-plugin are not able to use the parent pom. The reason for this is subtle but makes sense as doing so would create a circular dependency between the ANTLR tool (which uses the plugin to build its own grammar files), and the plugins (which uses the tool to build grammar files and gunit to test).

This catch-22 situation means that the pom.xml file in the antlr3-maven-plugin directory and the one in the gunit-maven-plugin directory MUST be updated manually (or we must write a script to do this).

Finally, we need to remember that because the tool is dependent on the antlr3-maven-plugin and the plugin is itself dependent on the tool, that we must manually update the versions of each that they reference. So, when we bump the version of the toolset to say 3.1.4-SNAPSHOT, we need to change the antlr3-maven-plugin pom.xml and the gunit-maven-plugin pom.xml to reference that version of the antlr tool. The tool itself is always built with the prior released version of the plugin, so when we release we must change the main branch of the plugin to use the newly released version of the plugin. This is covered in the release checklist.

Deploying
----------

Deploying the tools at the current version is relatively easy, but to deploy to the ANTLR repositories (snapshot or release) you must have been granted access to the Sonatype OSS repositories' ANTLR login. Few people will have this access of course.

Next, because we do not publish access information for antlr.org, you will need to configure the repository server names locally. You do this by creating (or adding to) the file:

~/.m2/settings.xml

Which should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<settings>
<servers>
<server>
<id>sonatype-nexus-snapshots</id>
<username>xxxxxxx</username>
<password>xxxxxxx</password>
</server>
<server>
<id>sonatype-nexus-staging</id>
<username>xxxxxxx</username>
<password>xxxxxxx</password>
</server>
</servers>
</settings>

When this configuration is in place, you will be able to deploy the components, either individually or from the master directory:

mvn -Dmaven.test.skip=true -Ddeplot deploy

You will then see lots of information about checking existing version information and so on, and the components will be deployed once you supply the ANTLR public key passphrase to sign the jars.

Note that so long as the artifacts are versioned with a.b.c-SNAPSHOT then deployment will always be to the development snapshot directory. When the artifacts are versioned with a release version then deployment will be to the release stahinh repository, which will then be mirrored around the world if closed and release. The sonatype documentation should be consulted.

Release Checklist
------------------

Here is the procedure to use to make a release of ANTLR. Note that we should really use the mvn release:release command, but the perforce plugin for Maven is not commercial quality and I want to rewrite it.

For this checklist, let's assume that the current development version of ANTLR is 3.1.3-SNAPSHOT. This means that it will probably (but not necessarily) become release version 3.1.3 and that the development version will bump to 3.1.4-SNAPSHOT.

0) Run a build of the main branch and check that it is builds and passes as many tests as you want it to.

1) First make a branch from main into the target release directory. Then submit this to perforce. You could change versions numbers before submitting, but doing that in separate stages will keep things sane;

--- Use main development branch from here ---

2) Before we deploy the release, we want to update the versions of the development branch, so we don't deploy what is now the new release as an older snapshot (this is not super important, but procedure is good right?).

Check out all the pom.xml files (and if you are using any antlr.config parameters that must change, then do that too).

3) Edit the master pom.xml in the main directory and change the version from 3.1.3-SNAPSHOT to 3.1.4-SNAPSHOT.

4) Edit the pom.xml file for antlr3-maven-plugin under the main directory and change the version from 3.1.3-SNAPSHOT to 3.1.4-SNAPSHOT. Do the same for the pom.xml in the gunit-maven-plugin directory.

Update the pom.xml for the archetype manually too.

5) Now (from the main directory), run the command:

mvn versions:update-child-modules

You should see:

[INFO] [versions:update-child-modules]
[INFO] Module: gunit
[INFO] Parent is org.antlr:antlr-master:3.1.4-SNAPSHOT
[INFO] Module: runtime/Java
[INFO] Parent is org.antlr:antlr-master:3.1.4-SNAPSHOT
[INFO] Module: tool
[INFO] Parent is org.antlr:antlr-master:3.1.4-SNAPSHOT

6) Run a build of the main branch:

mvn -Dmaven.test.skip=true

All should be good.

7) Submit the pom changes of the main branch to perforce.

8) Deploy the new snapshot as a placeholder for the next release. It will go to the snapshot repository of course:

mvn -N deploy
mvn -Dmaven.test.skip=true deploy

9) You are now finished with the main development branch and should change working directories to the release branch you made earlier.

--- Use release branch from here ---

10) Check out all the pom.xml files in the release branch (and if you are using any antlr.config parameters that must change, then do that too).

11) Edit the master pom.xml in the release-3.1.3 directory and change the version from 3.1.3-SNAPSHOT to 3.1.3.

12) Edit the pom.xml file for antlr3-maven-plugin under the release-3.1.3 directory and change the version from 3.1.3-SNAPSHOT to 3.1.3. Also change the version of the tool that the this pom.xml references from 3.1.3-SNAPSHOT to 3.1.3 as we are now releasing the plugin of course and it needs to reference the version we are about to release. You will find this reference in the dependencies section of the antlr3-maven-plugin pom.xml. Also change the version references in the pom for gunit-maven-plugin.

13) Now (from the release-3.1.3 directory), run the command:

mvn versions:update-child-modules

You should see:

[INFO] [versions:update-child-modules]
[INFO] Module: gunit
[INFO] Parent was org.antlr:antlr-master:3.1.3-SNAPSHOT,
now org.antlr:antlr-master:3.1.3
[INFO] Module: runtime/Java
[INFO] Parent was org.antlr:antlr-master:3.1.3-SNAPSHOT,
now org.antlr:antlr-master:3.1.3
[INFO] Module: tool
[INFO] Parent was org.antlr:antlr-master:3.1.3-SNAPSHOT,
now org.antlr:antlr-master:3.1.3

14) Run a build of the release-3.1.3 branch:

mvn # Note I am letting unit tests run here!

All should be good, or as good as it gets ;-)

15) Submit the pom changes of the release-3.1.3 branch to perforce.

16) Deploy the new release (this is it guys, make sure you are happy):

mvn -N deploy
mvn -Dmaven.test.skip=true deploy

Note that we must skip the tests as Maven will not let you deploy releases that fail any junit tests.

17) The final step is that we must update the main branch pom.xml for the tool to reference the newly release version of the antlr3-maven-plugin. This is because each release of ANTLR is built with the prior release of ANTLR, and we have just released a new version. Edit the pom.xml for the tool (main/tool/pom.xml) under the main (that's the MAIN branch, not the release branch) and find the dependency reference to the antlr plugin. If you just released say 3.1.3, then the tool should now reference version 3.1.3 of the plugin. Having done this, you should probably rebuild the main branch and let it run the junit tests. Later, I will automate this dependency update as mvn can do this for us.

18) Having deployed the release to maven, you will want to create the uber jar for the new release, to make it downloadable from the antlr.org website. This is a repeat of the earlier described step to build the uber jar:

mvn -Dmaven.test.skip=true package assembly:assembly

MAven will produce the uber jar in the target directory:

antlr-master-3.1.3-completejar.jar

And this is the complete jar that can be downloaded from the web site. You may wish to produce an md5 checksum to go with the jar:

md5sum target/antlr-master-3.1.3-completejar.jar
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx target/antlr-master-3.1.4-SNAPSHOT-completejar.jar

The command you just ran will also produce a second jar:

antlr-master-3.1.3-src.jar

This is the source code for everythign you just deployed and can be unjarred and built from scratch using the very procedures described here, which means you will now be reading this BUILD.txt file for ever.

19) Reward anyone around you with good beer.

Miscellany
-----------

It was a little tricky to get all the interdependencies correct because ANTLR builds itself using itself and the maven plugin references the ANTLR Tool as well. Hence the maven tool is not a child project of the master pom.xml file, even though it is built by it.

An observant person will note that when the assembly:assembly phase is run, that it invokes the build of the ANTLR tool using the version of the Maven plugin that it has just built, and this results in the plugin using the version of ANTLR tool that it has just built. This is safe because everything will already be up to date and so we package up the version of the tool that we expect, but the Maven plugin we deploy will use the correct version of ANTLR, even though there is technically a circular dependency.

The master pom.xml does give us a way to cause the build of the ANTLR tool to use itself to build itself. This is because in dependencyManagement in the master pom.xml, we can reference the current version of the Tool and the Maven plugin, even though in the pom.xml for the tool itself refers to the previous version of the plugin.

What happens is that if we first cd into the tool and maven directories and build ANTLR, it will build itself with the prior version and this will deploy locally (.m2). We can then clean build from the master pom and when ANTLR asks for the prior version of the tool, the master pom.xml will override it and build with the interim versions we just built manually.

However, strictly speaking, we need a third build where we rebuild the tool again with the version of the tool that was built with itself and not deploy the version that was built by the version of itself that was built by a prior version of itself. I decided that this was not particularly useful and complicates things too much. Building with a prior version of the tool is fine and if there was ever a need to, we could release twice in quick succession.

I have occasionally seen the MAven reactor screw up (or perhaps it is the ANTLR tool) when building. If this happens you will see an ANTLR Panic - cannot find en.stg message. If this happens to you, then just rerun the build and it will eventually work.