class

Crodoc::Tokenizer

Inherits Reference / Object

Represents a Crystal syntaxt tokenizer based on Crystal::Lexer.

Example:

source = Crodoc::Source.new(code, path)
tokenizer = Crodoc::Tokenizer.new(source)
tokenizer.run do |token|
  puts token
end

Constructors

new(source : Source)

Instantiates a Tokenizer using source.

Example:

source = Crodoc::Source.new(code, path)
Crodoc::Tokenizer.new(source)
Source
new(lexer : Crystal::Lexer)

Instantiates a Tokenizer using a Crystal::Lexer.

Example:

lexer = Crystal::Lexer.new(code)
Crodoc::Tokenizer.new(lexer)
Source

Instance methods

run

Runs the tokenizer and yields each token as a block argument.

Example:

Crodoc::Tokenizer.new(source).run do |token|
  puts token
end
Source