class

Semtrace::Decomposer

Greedy residual decomposition of embedding vectors.

Given a target vector v, iteratively finds the nearest token embedding, subtracts it, and repeats until the residual norm falls below epsilon or max_steps is reached.

Constructors

new(store : EmbeddingStore)

Source

Instance methods

arithmetic(positive : Array(String), negative : Array(String) = [] of String, **opts) : TraceResult

Concept arithmetic: sum positive tokens, subtract negative tokens, decompose. Example: arithmetic(positive: [" king", " woman"], negative: [" man"]) = E(" king") - E(" man") + E(" woman")

Source

decompose(target : Array(Float32) | Slice(Float32), epsilon : Float32 = 0.01_f32, max_steps : Int32 = 20, lookahead : Int32 = 1) : TraceResult

Decomposes a target vector into a sequence of tokens. When lookahead > 1, evaluates the top-k candidates at each step and picks the one that minimizes the residual norm after subtraction.

Source

decompose_ip(target : Array(Float32) | Slice(Float32), epsilon : Float32 = 0.01_f32, max_steps : Int32 = 20, lookahead : Int32 = 1) : TraceResult

Decomposes using inner product search. Requires the store to have an inner product HNSW index.

Source

decompose_normalized(target : Array(Float32) | Slice(Float32), epsilon : Float32 = 0.01_f32, max_steps : Int32 = 20, lookahead : Int32 = 1) : TraceResult

Decomposes using L2-normalized token vectors. Requires the store to have a normalized HNSW index (build_norm_index: true). Searches the normalized index AND subtracts normalized vectors.

Source

midpoint(token_a : String, token_b : String, **opts) : TraceResult

Midpoint between two tokens.

Source

trace_sentence(token_ids : Array(Int32), **opts) : TraceResult

Decomposes a bag-of-words sentence embedding (sum of static token embeddings).

Source

trace_single(token_name : String, **opts) : TraceResult

Decomposes a single token's embedding (sanity check: should return itself).

Source

trace_tokens(token_names : Array(String), **opts) : TraceResult

Composes a vector by summing token embeddings, then decomposes it. This is the round-trip test: does trace(compose(tokens)) ≈ tokens?

Source