Token kinds emitted by the lexer.
The lexer is intentionally fine-grained for structural elements
(so the parser does not need to look at characters), and
coarse-grained for value atoms (Number, BareKeyOrAtom) — those
are disambiguated by the parser based on context (a 1979 after
= in a key/value pair could be an integer, a float, a date, or
part of a datetime; only the parser can decide).
Trivia tokens (Whitespace, Newline, Comment) are emitted rather
than skipped, because the AST keeps them attached to their nodes
for byte-identical round-trips.
Constants
BasicString = 10
Strings (delimiters included in the raw text).
MultilineBasicString = 11
MultilineLiteralString = 13
BareKeyOrAtom = 14
An atom that can be a bare key (a-z A-Z 0-9 _ -) OR a value
atom (integer, float, boolean, datetime, inf, nan, true,
false). The parser disambiguates by context. We keep the raw
text byte-for-byte so the serializer can re-emit it unchanged.
Instance methods
bare_key_or_atom?
Returns true if this enum value equals BareKeyOrAtom
basic_string?
Returns true if this enum value equals BasicString
comma?
Returns true if this enum value equals Comma
dot?
Returns true if this enum value equals Dot
eof?
Returns true if this enum value equals EOF
equal?
Returns true if this enum value equals Equal
l_brace?
Returns true if this enum value equals LBrace
l_bracket?
Returns true if this enum value equals LBracket
literal_string?
Returns true if this enum value equals LiteralString
multiline_basic_string?
Returns true if this enum value equals MultilineBasicString
multiline_literal_string?
Returns true if this enum value equals MultilineLiteralString
newline?
Returns true if this enum value equals Newline
r_brace?
Returns true if this enum value equals RBrace
r_bracket?
Returns true if this enum value equals RBracket
whitespace?
Returns true if this enum value equals Whitespace