module

LexisMinhash::Similarity

Similarity measures for comparing documents

This module provides various similarity metrics for comparing document representations, including weighted overlap for TF-IDF or other weighted document representations. Similarity contains similarity and overlap measures useful for comparing MinHash signatures and weighted document vectors.

Class methods

fast_overlap(a : Slice(UInt64), b : Slice(UInt64)) : Float64

Fast overlap for UInt64 slices

Source

fast_overlap(a : Slice(UInt32), b : Slice(UInt32)) : Float64

Optimized overlap coefficient using two-pointer scan for sorted Slices (UInt32)

Input slices MUST be sorted in ascending order.

Source

jaccard(a : Set(T), b : Set(T)) : Float64 forall T

Computes the Jaccard similarity coefficient between two sets

The Jaccard similarity measures similarity between two sets as |A ∩ B| / |A ∪ B|. Returns a value between 0.0 (no overlap) and 1.0 (identical sets).

a = Set{1, 2, 3}
b = Set{2, 3, 4}
LexisMinhash::Similarity.jaccard(a, b) # => 0.5 (intersection: 2, union: 4)

Source

weighted_overlap(a : Hash(String, Float64), b : Hash(String, Float64)) : Float64

Computes the weighted overlap coefficient between two weighted document representations

The weighted overlap coefficient measures similarity as the sum of minimum weights for intersecting terms, normalized by the smaller total weight. This is useful for comparing weighted document representations like TF-IDF vectors.

doc_a = {"machine" => 0.8, "learning" => 0.9, "data" => 0.5}
doc_b = {"machine" => 0.8, "learning" => 0.6, "model" => 0.7}
LexisMinhash::Similarity.weighted_overlap(doc_a, doc_b) # => ~0.736

NOTE: Keys are case-sensitive; ensure both hashes use consistent casing.

Source