class

Obelisk::Lexer

Inherits Reference < Object

Base lexer interface

Instance methods

aliases

Get lexer aliases

Source
analyze(text : String) : Float32

Analyze text to determine if this lexer can handle it Returns a score from 0.0 to 1.0

Source
config
Source
filenames

Get supported filenames (patterns)

Source
matches_filename?(filename : String) : Bool

Check if lexer supports a filename

Source
matches_mime_type?(mime_type : String) : Bool

Check if lexer supports a MIME type

Source
mime_types

Get supported MIME types

Source
name

Get lexer name

Source
tokenize(text : String) : TokenIterator
Source
tokenize_unsafe(text : String) : TokenIterator

Create the raw token iterator without safety wrappers. This is used internally by SafeTokenIteratorAdapter to avoid infinite recursion. Default implementation just calls tokenize, but RegexLexer overrides this to return the raw RegexTokenIterator (which has memory corruption issues).

Source