DataDungeon::SemanticIndex
Inherits DataDungeon::Searcher < DataDungeon::AssetIndexer < Reference < Object
Optional Memo index of what a type's registration says to embed.
An asset of a type registered with embed (DB#register_type) gets one
embedding of those parts: its description as written, then a statement
for each tag, by name ("Hair color is red."). A type can also declare
facets, each a group of tags embedded on its own as Memo source type
<asset_type>.<facet>, so a search can rank one part of an asset
without the rest diluting it. Types with neither aren't embedded.
A store has one embedding model, the first one used: vectors from two
models can't be compared, so opening with another raises, unless
switch_model: true drops the old vectors to embed everything again.
Stored assets enqueue work; one worker per store calls catch_up, for what other processes changed, and process_queue to embed it. Search does not touch DataDungeon usage counters or Memo's search counters.
Constants
Constructors
Class methods
The Memo source type holding one facet of a type's assets.
The text embedded for these parts of an asset: its description as written, then one statement per tag, by name ("Hair color is red."), which searched best of the forms measured (DESIGN-embeddings.md). The parts are a set: any order gives the same text, so the same vector. Empty parts are left out.
Instance methods
Keyset pagination keeps memory bounded for large existing catalogs. Returns the number of assets newly queued.
Queues what changed since the last call, in this process or another (DB's change log, asset_changes), so assets other programs store become searchable too. The first call, and one the log no longer reaches back to, checks every asset (backfill). Returns the number of assets queued. The worker calls it before process_queue.
The asset write has already committed when this runs. Embedding happens later, outside the asset write path.
Queues the asset's embedding and each facet's whose text changed, and drops sources it no longer has (a facet removed, or one left with no text). A retired asset is dropped, as a deleted one is, until it is reinstated. Returns true when anything was queued.
Run from a single background worker; Memo owns retries and the queue. Memo saves the index file on its own policy (after many unsaved changes or a few minutes) and on close; until then its journal covers a crash, so saving after every batch would only rewrite the file needlessly.
Returns up to limit hits, best first, each with its score. There is no
score cutoff unless min_score gives one: scores aren't comparable
across embedding models or query lengths, so the threshold belongs to
the caller's decision (offering choices vs. reusing an asset outright).
key ranks only the assets with one key; key_prefix (ending in ".",
e.g. from DB#group_key_for) only those whose leading traits match.
facet ranks one of a registered type's facets instead of the whole
asset's embedding, and needs asset_type.
Ranks assets of a registered type on several facets at once, each with
its own query ({"hair" => "long red braids", "body" => "hulking"}). An
asset must have every facet named, and scores its weakest one: a close
match on one part doesn't make up for a poor match on another. Each
facet is searched to a depth of depth assets, so combine within a
scope (key, key_prefix, tags) small enough to be ranked whole.