class

TokenMemoizer

Inherits Reference < Object

Constructors

new(tokenizer : Tokenizer)
Source

Instance methods

freeze
Source
get_next_token
Source
peek_current_token
Source
peek_next_token
Source
populate_tokens_to_index(index)
Source
unfreeze(where : Int32)
Source