class

TOML::Lexer

Inherits Reference < Object

Streaming lexer for TOML v1.0 documents.

Design notes

  • Trivia is emitted, not skipped. Whitespace, newlines and comments are returned as tokens because the AST keeps them attached to nodes for byte-identical round-trips.

  • Atoms are coarse-grained. Anything that is not a delimiter, string, or trivia is returned as a single BareKeyOrAtom token whose raw is the verbatim source slice. The parser decides whether it is a bare key, integer, float, datetime, boolean, inf, or nan based on context. Two consequences:

    • The dot character is always emitted as Dot. A float like 1.5 lexes as Atom("1") Dot Atom("5"); the parser reassembles it. This keeps the lexer context-free at the cost of a tiny bit of parser work.
    • The colon : is part of an atom (so a time value like 07:32:00 is a single Atom).

  • Brackets are always single-character. [ and ] are emitted one at a time even when adjacent. The parser disambiguates [[products]] (array-of-tables header) from [[1, 2]] (nested array) by position: a line that starts with two consecutive LBracket tokens is an AoT header, anywhere else they open two arrays.

  • Line endings. \n and \r\n are both valid newlines. A bare \r not followed by \n is not a TOML line ending and triggers a ParseError if it appears outside a string.

Constructors

new(source : String)
Source

Instance methods

eof?

Returns true if the lexer has consumed all input.

Source
next_token

Returns the next token. Once the end of input is reached, returns an EOF token at the current position; subsequent calls keep returning EOF.

Source