module

Xerp::Tokenize

Constants

COMMON_KEYWORDS = Set {"if", "else", "elsif", "elif", "then", "unless", "case", "when", "switch", "default", "for", "while", "do", "loop", "until", "foreach", "break", "continue", "return", "yield", "next", "try", "catch", "finally", "rescue", "ensure", "raise", "throw", "def", "fn", "func", "function", "fun", "method", "class", "struct", "enum", "module", "interface", "trait", "type", "var", "let", "const", "val", "mut", "public", "private", "protected", "internal", "static", "abstract", "virtual", "override", "final", "true", "false", "nil", "null", "none", "undefined", "self", "this", "super", "int", "float", "string", "bool", "boolean", "void", "array", "hash", "map", "list", "set", "new", "delete", "import", "require", "include", "use", "from", "as", "in", "is", "not", "and", "or", "begin", "end", "with", "lambda", "proc"}

Common programming keywords across languages.

COMPOUND_PATTERNS = [/([a-zA-Z_][a-zA-Z0-9_]*)\.([a-zA-Z_][a-zA-Z0-9_]*)/, /([a-zA-Z_][a-zA-Z0-9_]*)::([a-zA-Z_][a-zA-Z0-9_]*)/, /([a-zA-Z_][a-zA-Z0-9_]*)\/(\d+)/]

Patterns for compound token detection in source code. These detect patterns like A.B, A::B, A/N (arity notation)

MAX_TOKEN_LEN = 128

Maximum allowed token length.

MIN_TOKEN_LEN = 1

Minimum token length to keep.

TOKEN_WEIGHTS = {TokenKind::Ident => 1.0, TokenKind::Compound => 0.9, TokenKind::Word => 0.7, TokenKind::Str => 0.3, TokenKind::Num => 0.2, TokenKind::Op => 0.1}

Default weights for scoring different token kinds. Higher weight = more significant for search ranking.

Class methods

add_compounds_to_result(result : TokenizeResult, lines : Array(String)) : TokenizeResult

Adds compound tokens to an existing TokenizeResult.

Source
common_keyword?(token : String) : Bool

Checks if a token looks like a common keyword that should be deprioritized.

Source
derive_compounds(lines : Array(String)) : Array(TokenOcc)

Derives compound tokens from source lines. Returns additional compound tokens to add to the token set.

Source
kind_from_s(s : String) : TokenKind

Parses a string back to TokenKind.

Source
kind_to_s(kind : TokenKind) : String

Returns the string representation for database storage.

Source
normalize_token(token : String, kind : TokenKind, max_len : Int32 = MAX_TOKEN_LEN) : String | Nil

Normalizes a token based on its kind. Returns nil if the token should be filtered out.

Source
split_identifier(ident : String) : Array(String)

Splits an identifier into components based on naming conventions. For example: "getUserName" -> ["get", "User", "Name"] "user_name" -> ["user", "name"] Returns the original token plus any split components.

Source
weight_for(kind : TokenKind) : Float64

Returns the weight for a given token kind.

Source

Nested types