Lexer Class

Initializes a new instance of the Lexer class

Initializes a new instance of the Lexer class

Initializes a new instance of the Lexer class

What is the index of the current character of lookahead?

Return whether or not a backtracking attempt failed.

For debugging and other purposes, might want the grammar name. Have ANTLR generate an implementation for this method.

Get number of recognition errors (lexer, parser, tree parser). Each recognizer tracks its own number. So parser and lexer each have separate count. Does not count the spurious errors found between an error and next valid token match

Gets or sets the text matched so far for the current token or any text override.

Used to print out token names like ID during debugging and error reporting. The generated parsers implement a method that overrides this to point to their String[] tokenNames.

Has this rule already parsed input at the current index in the input stream? Return the stop token index or MEMO_RULE_UNKNOWN. If we attempted but failed to parse properly before, return MEMO_RULE_FAILED.

A hook to listen in on the token consumption during error recovery. The DebugParser subclasses this to fire events to the listenter.

Compute the context-sensitive FOLLOW set for current rule. This is set of token types that can follow a specific rule reference given a specific call chain. You get the set of viable tokens that can possibly come next (lookahead depth 1) given the current call chain. Contrast this with the definition of plain FOLLOW for rule r:

Consume tokens until one matches the given token set

The standard method called to automatically emit a token at the outermost lexical rule. The token object should point into the char buffer start..stop. If there is a text override in 'text', use that to set the token's text. Override this method to emit custom Token objects.

Currently does not support multiple emits per nextToken invocation for efficiency reasons. Subclass and override this method and nextToken (to push tokens into a list and pull from that list rather than a single variable as this implementation does).

Override this method to change where error messages go

Determines whether the specified object is equal to the current object.

Allows an object to try to free resources and perform other cleanup operations before it is reclaimed by garbage collection.

Match needs to return the current input symbol, which gets put into the label for the associated token ref; e.g., x=ID. Token and tree parsers need to return different objects. Rather than test for input stream type or change the IntStream interface, I use a simple method to ask the recognizer to tell me what the current input symbol is.

What is the error header, normally line/character position information?

Serves as the default hash function.

Conjure up a missing token during error recovery.

Return IListT of the rules in your parser instance leading up to a call to this method. You could override if you want more details such as the file/line info of where in the parser java code a rule is invoked.

Given a rule number and a start token index number, return MEMO_RULE_UNKNOWN if the rule has not parsed input starting from start index. If this rule has parsed input starting from the start index before, then return where the rule stopped parsing. It returns the index of the last token matched by the rule.

return how many rule/input-index pairs there are in total.

How should a token be displayed in an error message? The default is to display just the text, but during development you might want to have a lot of information spit out. Override in that case to use t.ToString() (which, for CommonToken, dumps everything about the token). This is better than forcing you to override a method in your token objects because you don't have to go modify your lexer so that it creates a new Java type.

Gets the Type of the current instance.

Match current input symbol against ttype. Attempt single token insertion or deletion error recovery. If that fails, throw MismatchedTokenException.

Match the wildcard: in a symbol

Creates a shallow copy of the current Object.

Record whether or not this rule parsed the input at this position successfully. Use a standard java hashtable for now.

This is the lexer entry point that sets instance var 'token'

Return a token from this source; i.e., match a token on the char stream.

Push a rule's follow set using our own hardcoded stack

Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out. You can instead use the rule invocation stack to do sophisticated error recovery if you are in a fragment rule.

Recover from an error found on the input stream. This is for NoViableAlt and mismatched symbol exceptions. If you enable single token insertion and deletion, this will usually not handle mismatched symbol exceptions but there could be a mismatched token that the match() routine could not recover from.

Attempt to recover from a single missing or extra token.

Instruct the lexer to skip creating a token for current lexer rule and look for another token. nextToken() knows to keep looking when a lexer rule finishes with token set to SKIP_TOKEN. Recall that if token==null at end of any token rule, it creates one for you and emits it.

Returns a string that represents the current object.

A convenience method for use most often with template rewrites. Convert a list of IToken to a list of String.

Where is the lexer drawing characters from?

State of a lexer, parser, or tree parser are collected into a state object so the state can be shared. This sharing is needed to have one grammar import others and share same error variables and other state variables. It's a kind of explicit multiple inheritance via delegation of methods and shared state.