module

Similar::Text

Class methods

ends_with_newline(str : String) : Bool

Checks if a string ends with a newline.

ends_with_newline(bytes : Bytes) : Bool

Checks if bytes end with a newline.

tokenize_chars(str : String) : Array(String)

Splits a string into characters (UTF-8 aware).

tokenize_chars(bytes : Bytes) : Array(BytesWrapper)

Splits bytes into single-byte slices.

tokenize_graphemes(str : String) : Array(String)

Basic grapheme tokenization (handles combining characters).

tokenize_lines(str : String) : Array(String)

Splits a string into lines with newlines attached.

tokenize_lines(bytes : Bytes) : Array(BytesWrapper)

Splits bytes into lines with newlines attached.

tokenize_unicode_words(str : String) : Array(String)

Basic Unicode word tokenization using character categories.

tokenize_words(str : String) : Array(String)

Splits a string into words and whitespace.

tokenize_words(bytes : Bytes) : Array(BytesWrapper)

Splits bytes into words and whitespace (ASCII whitespace only).

upper_seq_ratio(seq1 : Array(T), seq2 : Array(T)) : Float32 forall T

Quick and dirty way to get an upper sequence ratio.