class

LexisMinhash::LSHIndex

Inherits Reference < Object

In-memory LSH index using Int32 doc IDs and linear probing storage LSHIndex is an in-memory locality-sensitive hashing index. It stores signatures per document and a set of per-band hash tables (LinearBucketTable) to quickly retrieve candidate document ids for a query.

Constructors

new(bands : Int32 = 20, expected_docs : Int32 = 1000, store_signatures : Bool = true)

Initialize with expected number of documents for capacity planning Table capacity is ~2x expected docs per band for good load factor Initialize index with bands and expected_docs for capacity planning. When store_signatures is false, signature storage is disabled to reduce memory usage, but query_with_scores() will raise NotImplementedError.

Source

Instance methods

add(doc_id : Int32, text : String) : Nil

Compute signature for text and insert into all band tables

Source
add_with_signature(doc_id : Int32, signature : Array(UInt32)) : Nil

Add a document using a precomputed signature

Source
add_with_weights(doc_id : Int32, text : String, weights : Hash(String, Float64)) : Nil

Add a document using TF-IDF style weights

Source
clear

Clear the index and all tables

Source
find_similar_pairs(threshold : Float64 = 0.75) : Set(Tuple(Int32, Int32))

Find all similar document pairs above threshold similarity

Source
get_signature(doc_id : Int32) : Array(UInt32) | Nil

Retrieve stored signature by doc id (returns nil if not present or storage disabled)

Source
load_factors

Returns load factors for each band's table Returns load factor per band table

Source
query(text : String) : Set(Int32)

Query by plain text (unweighted)

Source
query_by_signature(signature : Array(UInt32)) : Set(Int32)

Query by precomputed signature

Source
query_with_scores(text : String) : Array(Tuple(Int32, Float64))

Query and return candidates with similarity scores, sorted desc

Source
query_with_scores_by_signature(signature : Array(UInt32)) : Array(Tuple(Int32, Float64))

Query by signature and return scored results

Source
query_with_weights(text : String, weights : Hash(String, Float64)) : Set(Int32)

Query with weights (TF-IDF)

Source
query_with_weights_by_signature(signature : Array(UInt32), weights : Hash(String, Float64)) : Set(Int32)

Alias for query_by_signature - weights are not needed at query time since the signature already contains the weighted information

Source
size

Number of stored documents (approximate count based on first band table)

Source