class

Obelisk::DocumentLexer

Inherits Obelisk::Lexer < Reference < Object

Multi-language document lexer

Constructors

new(architecture : EmbeddedLanguageArchitecture, base_language : String)
Source

Instance methods

analyze(text : String) : Float32

Analyze text to determine if this lexer can handle it Returns a score from 0.0 to 1.0

Source
config
Source
tokenize(text : String) : TokenIterator
Source