Logos::Lexer(Token, Source, Extras, Error)
Inherits Iterator < Enumerable < Reference < Object
Lexer is the main struct that allows you to read through a
Source and produce tokens for enums implementing the Logos trait.
Constructors
Create a new Lexer with the provided Extras.
Due to type inference, it might be more ergonomic to construct
it by calling Token.lexer_with_extras on any Token with derived Logos.
Instance methods
Bumps the end of currently lexed token by n bytes.
Panics if adding n to current offset would place the Lexer beyond the last byte,
or in the middle of an UTF-8 code point (does not apply when lexing raw Slice(UInt8)).
Last callback return value (for tokens with associated data)
Get callback value as a specific type.
Turn this lexer into a lexer for a new token type.
The new lexer continues to point at the same span as the current lexer, and the current token becomes the error token of the new token type.
Return a typed callback payload when the result is the expected token variant.
Same as payload_for, but raises if the token doesn't match or payload type is missing.
Read bytes bytes starting at current position of the Lexer plus offset.
If end of the Source has been reached, this will return nil.
Read a single byte at current position of the Lexer plus offset.
If end of the Source has been reached, this will return nil.
Wrap the Lexer in an Iterator that produces tuples of (Token, Span).
tokens = Token.lexer("42 3.14 -5 f").spanned.to_a
# => [{Token::Integer(42), 0..2}, {Token::Float(3.14), 3..7}, ...]