std.experimental.lexer

Summary This module contains a range-based compile-time _lexer generator.

Overview The _lexer generator consists of a template mixin, Lexer, along with several helper templates for generating such things as token identifiers.

To write a _lexer using this API:

Create the string array constants for your language.

Create aliases for the various token and token identifier types
specific to your language.

Create a struct that mixes in the Lexer template mixin and
implements the necessary functions.
- Lexer

Examples

A _lexer for D is available here.
A _lexer for Lua is available here.
A _lexer for JSON is available here.

TemplateParameters Template Parameter Definitions $(DDOC_ANCHOR defaultTokenFunction) defaultTokenFunction $(DD A function that serves as the default token lexing function. For most languages this will be the identifier lexing function.)) $(DT $(DDOC_ANCHOR tokenSeparatingFunction) tokenSeparatingFunction) $(DD A function that is able to determine if an identifier/keyword has come to an end. This function must return bool and take a single size_t argument representing the number of bytes to skip over before looking for a separating character.) $(DT $(DDOC_ANCHOR staticTokens) staticTokens) $(DD A listing of the tokens whose exact value never changes and which cannot possibly be a token handled by the default token lexing function. The most common example of this kind of token is an operator such as $(D_STRING "*"), or $(D_STRING "-") in a programming language.) $(DT $(DDOC_ANCHOR dynamicTokens) dynamicTokens) $(DD A listing of tokens whose value is variable, such as whitespace, identifiers, number literals, and string literals.) $(DT $(DDOC_ANCHOR possibleDefaultTokens) possibleDefaultTokens) $(DD A listing of tokens that could posibly be one of the tokens handled by the default token handling function. An common example of this is a keyword such as $(D_STRING "for"), which looks like the beginning of the identifier $(D_STRING "fortunate"). tokenSeparatingFunction is called to determine if the character after the $(D_STRING 'r') separates the identifier, indicating that the token is $(D_STRING "for"), or if lexing should be turned over to the defaultTokenFunction.) $(DT $(DDOC_ANCHOR tokenHandlers) tokenHandlers) $(DD A mapping of prefixes to custom token handling function names. The generated _lexer will search for the even-index elements of this array, and then call the function whose name is the element immedately after the even-indexed element. This is used for lexing complex tokens whose prefix is fixed.)

Here are some example constants for a simple calculator _lexer:

// There are a near infinite number of valid number literals, so numbers are
// dynamic tokens.
enum string[] dynamicTokens = ["numberLiteral", "whitespace"];

// The operators are always the same, and cannot start a numberLiteral, so
// they are staticTokens
enum string[] staticTokens = ["-", "+", "*", "/"];

// In this simple example there are no keywords or other tokens that could
// look like dynamic tokens, so this is blank.
enum string[] possibleDefaultTokens = [];

// If any whitespace character or digit is encountered, pass lexing over to
// our custom handler functions. These will be demonstrated in an example
// later on.
enum string[] tokenHandlers = [
    "0", "lexNumber",
    "1", "lexNumber",
    "2", "lexNumber",
    "3", "lexNumber",
    "4", "lexNumber",
    "5", "lexNumber",
    "6", "lexNumber",
    "7", "lexNumber",
    "8", "lexNumber",
    "9", "lexNumber",
    " ", "lexWhitespace",
    "\n", "lexWhitespace",
    "\t", "lexWhitespace",
    "\r", "lexWhitespace"
];

Copyright

Brian Schott 2013

License

License 1.0

Authors

Brian Schott, with ideas shamelessly stolen from Andrei Alexandrescu

Source: std/experimental/_lexer.d

tmpl TokenId TokenIdType

struct LexerRange TokenStructure

fn tokenStringRepresentation

Types 2

structTokenStructure(IdType, string extraFields = "")

The token that is returned by the lexer.

Parameters

`IdType`	The D type of the "type" token type field.
`extraFields`	A string containing D code for any extra fields that should be included in the token structure body. This string is passed directly to a mixin statement.

Examples

// No extra struct fields are desired in this example, so leave it blank.
alias Token = TokenStructure!(IdType, "");
Token minusToken = Token(tok!"-");

Fields

string textThe text of the token for dynamic tokens. May be `null`, e.g. for static tokens.

size_t line1-based line number at which the start of this token occurs.

size_t column1-based column number at which the start of this token occurs. This is measured in bytes and may not be correct when tab characters are involved.

size_t index0-based byte offset from the beginning of the input at which the start of this token occurs.

IdType typeThe token type.

Methods

bool opEquals(ref const typeof(this) other) pure nothrow @safe @nogc const

bool opEquals(IdType type) pure nothrow @safe @nogc constReturns: true if the token has the given type, false otherwise.

Constructors

this(IdType type)Constructs a token from a token type. Params: type = the token type

this(IdType type, string text, size_t line, size_t column, size_t index)Constructs a token. Params: type = the token type text = the text of the token, which may be null line = the line number at which this token occurs, 1-based column = the column number at which this...

structLexerRange

Range structure that wraps the _lexer's input.

Fields

const(ubyte)[] bytesThe input bytes.

size_t indexThe range's current position.

size_t columnThe current column number.

size_t lineThe current line number.

Methods

size_t mark()() nothrow pure @safe @nogc constReturns: a mark at the current position that can then be used with slice.

void seek()(size_t m) nothrow pure @safe @nogcSets the range to the given position. Params: m = the position to seek to

const(ubyte)[] slice()(size_t m) nothrow pure @safe @nogc constReturns a slice of the input byte array between the given mark and the current position. Params m = the beginning index of the slice to return

bool empty()() nothrow pure @safe @nogc constImplements the range primitive empty.

ubyte front()() nothrow pure @safe @nogc constImplements the range primitive front.

const(ubyte)[] peek(size_t p) nothrow pure @safe @nogc constReturns: the current item as well as the items p items ahead.

bool startsWith(const(ubyte[]) needle) nothrow pure @safe @nogc constReturns: true if the range starts with the given byte sequence

ubyte peekAt()(size_t offset) nothrow pure @safe @nogc const

bool canPeek()(size_t p) nothrow pure @safe @nogc constReturns: true if it is possible to peek p bytes ahead.

void popFront()() nothrow pure @safe @nogcImplements the range primitive popFront.

void popFrontN()(size_t n) nothrow pure @safe @nogcImplements the algorithm popFrontN more efficiently. This function does not detect or handle newlines.

void incrementLine()(size_t i = 1) nothrow pure @safe @nogcIncrements the range's line number and resets the column counter.

Constructors

this(const(ubyte)[] bytes, size_t index = 0, size_t column = 1, size_t line = 1)Params: bytes = the lexer input index = the initial offset from the beginning of bytes column = the initial column number line = the initial line number

Functions 1

string tokenStringRepresentation(IdType,  alias  staticTokens,  alias  dynamicTokens,
     alias  possibleDefaultTokens)(IdType type)

pure nothrow @property @nogc @safeLooks up the string representation of the given token type.

Templates 2

tmplTokenIdType(alias staticTokens, alias dynamicTokens, alias possibleDefaultTokens)

Template for determining the type used for a token type.

Selects the smallest unsigned integral type that is able to hold the value staticTokens.length + dynamicTokens.length + possibleDefaultTokens.length. For example if there are 20 static tokens, 30 dynamic tokens, and 10 possible default tokens, this template will alias itself to ubyte, as 20 + 30 + 10 < ubyte.max.

Examples

// In our calculator example this means that IdType is an alias for ubyte.
alias IdType = TokenIdType!(staticTokens, dynamicTokens, possibleDefaultTokens);

tmplTokenId(IdType, alias staticTokens, alias dynamicTokens, alias possibleDefaultTokens, string symbol)

Generates the token type identifier for the given symbol.

There are two special cases:

If symbol is "", then the token identifier will be 0
If symbol is "\0", then the token identifier will be the maximum
valid token type identifier

In all cases this template will alias itself to a constant of type IdType. This template will fail at compile time if symbol is not one of the staticTokens, dynamicTokens, or possibleDefaultTokens.

Examples

template tok(string symbol)
{
    alias tok = TokenId!(IdType, staticTokens, dynamicTokens,
        possibleDefaultTokens, symbol);
}
// num and plus are of type ubyte.
IdType plus = tok!"+";
IdType num = tok!"numberLiteral";