class

Obelisk::Token

Inherits Reference < Object

Represents a single token with its type and value

PERFORMANCE: Stores a reference (source text + position) instead of copying the value string. This avoids allocating thousands of strings during tokenization. The value is lazily computed when accessed.

Constructors

from_ref(type : TokenType, source : String, pos : Int32, length : Int32) : Token

Create a token from a reference into source text

Source
new(type : TokenType, value : String)
Source

Instance methods

clone

Clone this token

Source
css_class

Get the CSS class for this token

Source
eof?

Check if this is an EOF token

Source
to_s(io : IO) : Nil

Appends a short String representation of this object which includes its class name and its object address.

class Person
  def initialize(@name : String, @age : Int32)
  end
end

Person.new("John", 32).to_s # => #<Person:0x10a199f20>
Source
type
Source
value

Get the token value Computes from source reference if needed, otherwise returns cached value

Source