class

Ameba::Tokenizer

Inherits Reference < Object

Represents Crystal syntax tokenizer based on Crystal::Lexer.

source = Ameba::Source.new code, path
tokenizer = Ameba::Tokenizer.new(source)
tokenizer.run do |token|
  puts token
end

Constructors

new(lexer : Crystal::Lexer)

Instantiates Tokenizer using a lexer.

lexer = Crystal::Lexer.new(code)
Ameba::Tokenizer.new(lexer)
Source
new(source)

Instantiates Tokenizer using a source.

source = Ameba::Source.new code, path
Ameba::Tokenizer.new(source)
Source

Instance methods

run

Runs the tokenizer and yields each token as a block argument.

Ameba::Tokenizer.new(source).run do |token|
  puts token
end
Source