module

LexisMinhash::Engine

Backward compatibility: allow Engine to accept Document interface

Constants

DEFAULT_WEIGHT = 1.0
MAX_SHINGLE_SIZE = 32
MIN_WORDS = 4
NUM_BANDS = 20
ROWS_PER_BAND = 5
SHINGLE_SIZE = 5
SIGNATURE_SIZE = 100

Default configuration constants

Class methods

bytes_to_signature(bytes : Bytes) : Array(UInt32)
Source
bytes_to_signature_slice(bytes : Bytes) : Slice(UInt32)
Source
compute_signature(text : String, weights : Hash(String, Float64) | Nil) : Array(UInt32)

Computes a MinHash signature with optional TF-IDF weights. Unknown shingles use the configured default weight (default: 1.0). Negative weights are clamped to 0 (excluded from signature).

Source
compute_signature(text : String, weights : Hash(UInt64, Float64)) : Array(UInt32)

Compute weighted signature using hashed shingle keys (UInt64).

Source
compute_signature(text : String) : Array(UInt32)

Compute signature using rolling hash + multiply-shift. Returns Array(UInt32) for backward compatibility.

Source
compute_signature(document : LexisMinhash::Document) : Array(UInt32)

Computes a MinHash signature from a Document

Provides backward compatibility for code using the Document interface. See LexisMinhash::Document for implementation details.

Source
compute_signature_from_hashes(hashes : Iterable(UInt64), weights : Iterable(Float64)) : Slice(UInt32)

Compute weighted signature from parallel iterables of hashes and weights.

Source
compute_signature_from_hashes(hashes : Iterable(UInt64)) : Slice(UInt32)

Compute signature directly from pre-hashed UInt64 values. The application handles String → UInt64 conversion.

Source
compute_signature_slice(text : String, weights : Hash(String, Float64) | Nil) : Slice(UInt32)

Compute signature slice with optional String->Float64 weights.

Source
compute_signature_slice(text : String) : Slice(UInt32)

Compute signature as Slice(UInt32) for performance-critical code.

Source
compute_signature_slice_weighted(text : String, weights : Hash(String, Float64)) : Slice(UInt32)

Compute a weighted signature from String-keyed weights. Pre-hashes the weights map once to avoid repeated String allocations.

Source
compute_signature_slice_weighted_hashed(text : String, weights_hashed : Hash(UInt64, Float64)) : Slice(UInt32)

Compute weighted signature where weights are keyed by the shingle's UInt64 rolling hash.

Source
compute_signature_weighted(text : String, weights : Hash(String, Float64)) : Array(UInt32)

Computes a weighted MinHash signature from String-keyed weights.

Source
compute_signature_with_config(cfg : Config, text : String) : Slice(UInt32)

Pure signature computation using an explicit Config. Returns a Slice(UInt32). Deterministic given the same config and text.

Source
compute_signature_with_prehashed_weights(text : String, weights : Hash(String, Float64)) : Array(UInt32)

Convenience: prehash String-keyed weights and compute signature.

Source
config

Return current engine configuration as a tuple for backward compatibility

Source
configure(signature_size : Int32 = SIGNATURE_SIZE, num_bands : Int32 = NUM_BANDS, shingle_size : Int32 = SHINGLE_SIZE, min_words : Int32 = MIN_WORDS, default_weight : Float64 = DEFAULT_WEIGHT, seed : Int64 | Nil = nil) : Nil

Configure the engine by creating a new default_config from supplied params.

Source
default_config

Return or generate the runtime default config. Thread-safe.

Source
default_weight
Source
detection_probability(similarity : Float64) : Float64

Estimate probability of detecting similar items.

Source
generate_bands(signature : Array(UInt32) | Slice(UInt32), bands : Int32 | Nil = nil) : Array(Tuple(Int32, UInt64))

Generate LSH bands from a signature (Array or Slice). Returns Array({Int32, UInt64}) with {band_index, band_hash} tuples.

Source
generate_config(signature_size : Int32 = SIGNATURE_SIZE, num_bands : Int32 = NUM_BANDS, shingle_size : Int32 = SHINGLE_SIZE, min_words : Int32 = MIN_WORDS, default_weight : Float64 = DEFAULT_WEIGHT, seed : Int64 | Nil = nil) : Config

Generate a Config instance. When seed is provided the coefficient arrays a and b are filled deterministically using splitmix64 so results are reproducible across runs. When seed is nil, uses Random::Secure as before.

Source
jaccard_similarity(text1 : String, text2 : String) : Float64

Compute true Jaccard similarity between two texts based on shingle sets.

Source
jaccard_similarity(doc1 : LexisMinhash::Document, doc2 : LexisMinhash::Document) : Float64

Compute true Jaccard similarity between two Documents.

Source
overlap_coefficient(a : Slice(UInt64), b : Slice(UInt64)) : Float64

Overlap coefficient for two sorted UInt64 slices.

Source
overlap_coefficient(a : Slice(UInt32), b : Slice(UInt32)) : Float64

Overlap coefficient for two sorted UInt32 slices.

Source
prehash_weights(weights : Hash(String, Float64)) : Hash(UInt64, Float64)

Convert String-keyed weights into UInt64-keyed weights using rolling shingle hash.

Source
shingle_hash_for(shingle : String) : UInt64

Compute the rolling UInt64 hash for a given shingle String.

Source
shingles_hashes(text : String, k : Int32, &)

Generate rolling shingle hashes (UInt64) for a text and window size k. Yields each rolling hash without allocating shingle strings.

Source
shingles_with_strings(text : String, k : Int32, &)

Generate rolling shingles with both hash and string representation. Yields (UInt64 hash, String shingle) for each shingle in the text.

Source
signature_to_bytes(signature : Array(UInt32) | Slice(UInt32)) : Bytes
Source
similarity(sig1 : Array(UInt32) | Slice(UInt32), sig2 : Array(UInt32) | Slice(UInt32)) : Float64

Compute similarity between two signatures (Array or Slice)

Source

Nested types