Perl ANTLR v3 Target
Author
Ron Blaschke, ron at rblasch.org
Status
Early prototyping phase. A simple lexer is Simple lexer and parser are working.
Progress
Here's a simple example. Note that everything is still subject to change.
Code Block |
---|
$ cat T.g
lexer grammar T;
options { language = Perl5; }
ZERO: '0';
ONE: '1';
|
Code Block |
---|
$ cat T.tokens Tokens=6 ZERO=4 ONE=5 |
...
Code Block |
---|
$ perl t.pl
type: 4
text: 0
type: 5
text: 1
type: 4
text: 0
|
2007-06-13
+ Escaped characters, like '\n', are now handled properly.
+ Added error handling.
Code Block |
---|
lexer grammar T2;
options { language = Perl5; }
ID : ('a'..'z'|'A'..'Z')+ ;
INT : '0'..'9'+ ;
NEWLINE:'\r'? '\n' ;
WS : (' '|'\t')+ ;
|
Code Block |
---|
INT=5
WS=7
Tokens=8
ID=4
NEWLINE=6
|
Code Block |
---|
#!/usr/bin/perl
use ANTLR::Runtime::ANTLRStringStream;
use T2Lexer;
use strict;
use warnings;
my $input = ANTLR::Runtime::ANTLRStringStream->new("Hello World!\n42\n");
my $lexer = T2Lexer->new($input);
while (1) {
my $token = $lexer->next_token();
last if $token->get_type() == $T2Lexer::EOF;
print "type: ", $token->get_type(), "\n";
print "text: ", $token->get_text(), "\n";
print "\n";
}
|
Code Block |
---|
type: 4 text: Hello type: 7 text: type: 4 text: World line 1:12 no viable alternative at character '!' type: 6 text: type: 5 text: 42 type: 6 text: |
Note the "no viable alternative" error message for the unrecognized '!'.
2007-06-15
+ Handle lexer actions
Here's another short example, similar to the one above. Note how whitespaces are put into the hidden channel (99) and newlines are skipped.
Code Block |
---|
lexer grammar T2; options { language = Perl5; } ID : ('a'..'z'\|'A'..'Z')\+ ; INT : '0'..'9'\+ ; NEWLINE:'\r'? '\n' { $self->skip(); } ; WS : (' '\|'\t')\+ { $channel = HIDDEN; } ; |
Code Block |
---|
$ perl t.pl text: Hello type: 4 pos: 1:0 channel: 0 token index: -1 text: type: 7 pos: 1:5 channel: 99 token index: -1 text: World type: 4 pos: 1:6 channel: 0 token index: -1 line 1:11 no viable alternative at character '!' text: 42 type: 5 pos: 2:0 channel: 0 token index: -1 |
2007-06-26
+ Simple Parser is working
Quick, what is 2 + 2? If you can't remember here's an easy way to find out. First we need a grammar.
Code Block |
---|
grammar MExpr; options { language = Perl5; } prog: stat+ ; stat: expr NEWLINE { print "$expr.value\n"; } | NEWLINE ; expr returns [value] : e=atom { $value = $e.value; } ( '+' e=atom { $value += $e.value; } | '-' e=atom { $value -= $e.value; } )* ; atom returns [value] : INT { $value = $INT.text; } | '(' expr ')' { $value = $expr.value; } ; ID : ('a'..'z'|'A'..'Z')+ ; INT : '0'..'9'+ ; NEWLINE:'\r'? '\n' ; WS : (' '|'\t')+ { $self->skip(); } ; |
And here's the test program.
Code Block |
---|
#!/usr/bin/perl
use strict;
use warnings;
use ANTLR::Runtime::ANTLRStringStream;
use ANTLR::Runtime::CommonTokenStream;
use MExprLexer;
use MExprParser;
while (<>) {
my $input = ANTLR::Runtime::ANTLRStringStream->new($_);
my $lexer = MExprLexer->new($input);
my $tokens = ANTLR::Runtime::CommonTokenStream->new({ token_source => $lexer });
my $parser = MExprParser->new($tokens);
$parser->prog();
}
|
Finally we're getting to the answer.
Code Block |
---|
$ perl t.pl
2 + 2
4
|
2007-08-08
+ Simple expression grammar
The grammar
Code Block |
---|
grammar Expr;
options {
language = Perl5;
}
@header {
}
@members {
my %memory;
}
prog: stat+ ;
stat: expr NEWLINE { print "$expr.value\n"; }
| ID '=' expr NEWLINE
{ $memory{$ID.text} = $expr.value; }
| NEWLINE
;
expr returns [value]
: e=multExpr { $value = $e.value; }
( '+' e=multExpr { $value += $e.value; }
| '-' e=multExpr { $value -= $e.value; }
)*
;
multExpr returns [value]
: e=atom { $value = $e.value; } ('*' e=atom { $value *= $e.value; })*
;
atom returns [value]
: INT { $value = $INT.text; }
| ID
{
my $v = $memory{$ID.text};
if (defined $v) {
$value = $v;
} else {
print STDERR "undefined variable $ID.text\n";
}
}
| '(' expr ')' { $value = $expr.value; }
;
ID : ('a'..'z'|'A'..'Z')+ ;
INT : '0'..'9'+ ;
NEWLINE:'\r'? '\n' ;
WS : (' '|'\t')+ { $self->skip(); } ;
|
Test program
Code Block |
---|
#!/usr/bin/perl
use strict;
use warnings;
use blib '../..';
use ANTLR::Runtime::ANTLRStringStream;
use ANTLR::Runtime::CommonTokenStream;
use ExprLexer;
use ExprParser;
my $in;
{
undef $/;
$in = <>;
}
my $input = ANTLR::Runtime::ANTLRStringStream->new($in);
my $lexer = ExprLexer->new($input);
my $tokens = ANTLR::Runtime::CommonTokenStream->new({ token_source => $lexer });
my $parser = ExprParser->new($tokens);
$parser->prog();
|
Test run
Code Block |
---|
$ perl t.pl
x=1
y=2
3*(x+y)
^Z
9
|
2008-02-23
Started real porting effort. The goal is to port one ANTLR runtime class at a time from Java to Perl, including full API coverage and documentation. First stop of the porting train: ANTLR::Runtime::BitSet.
2008-11-18
Got the first parser working: SimpleCalc, taken from the Five minute introduction to ANTLR 3.
Author
Ronald Blaschke (ron at rblasch org)