Durable state the context store keeps, expressed without reference to any particular database.
A storage adapter owns the authoritative records: documents, memory assertions, sessions, session commit bookkeeping, and the persisted job queue. Those records have to agree with each other, so the contract exposes whole operations rather than connections or statements: a workflow asks for a subtree to be removed, not for a transaction to be opened.
A vector index is different in kind. It is derived: every row in it can be
reproduced by work the store already persisted. That makes it safe to let the
index lag — it is regenerated, not recovered — and it means the guarantee
belongs in the read rather than in the write. search_vector/4 therefore
never returns a URI the node store cannot hold, so a row naming a removed
URI is inert until reclamation deletes it, and a re-write retires the URI's
previous vector so a live document is never ranked on content it no longer
has. Drift remains a cost, so it is bounded by cancel_jobs/1 and reclaimed
rather than silently accumulated.
Invariants an implementation must uphold
- A write is durable before it is acknowledged, and a cache in front of the adapter may never serve anything the adapter has not already committed.
- The authoritative stores change together.
remove_subtree/1removes a URI and everything beneath it from the node store, the memory store, the commit bookkeeping and the job queue in one step, andreplace_skill/2has the same two properties for a whole subtree. A rejected removal leaves all of that untouched. - A vector hit is only ever returned for a URI the node store still holds, so a removed URI is never surfaced even while a row naming it survives in the index. This is the guarantee that lets removal skip the index and reclamation pick up afterwards.
- A write retires the previous vector for the URI it writes, so a document is never ranked against content it no longer has. Retiring derived data only ever makes vector search return less, never something the store cannot hold.
- A result produced by work that started before a removal never brings the
removed URI back.
put_embedding_result/3andput_layer_result/4check that the node is still there before writing the result and report:discardedwhen it is not, which avoids most orphans outright; the read-path join is what holds for the rest.
Callbacks exchange core values and {:ok, _} / {:error, _} results, never
adapter-specific structs or exceptions.
Summary
Types
A vector search hit, scored by the adapter.
A unit of durable background work.
The layered summary a background worker produced.
A content line match: document URI, one-based line number, and bounded excerpt.
Two active memories at distinct URIs that may conflict.
One assertion ever made at a memory's URI.
One message of a session, in order.
A path-discovery match: URI, name, and node kind, without document content.
One file of a skill, at path below the skill's root.
A stored document or directory.
Callbacks
Appends a message to a session, preserving order.
Drops queued work for uri or its descendants, in any state.
Supervised children the adapter needs, given the resolved path to the database file.
The content hash recorded for a (session, destination) commit, if any.
Marks a job done.
How many jobs target uri or its descendants and are in one of statuses.
How many jobs of kind exist, in any status. A standing job has no URI to count by.
Creates a session and returns its id.
Puts a job back without spending one of its attempts, for work that is merely waiting.
Claims the next runnable job of one of kinds, advancing its attempt count.
How many documents are stored at prefix or beneath it.
Records durable background work.
Records a job failure, rescheduling it with backoff while attempts remain and marking it failed once they are exhausted.
Path-discovery matches for query within scope_uri or beneath it.
The node at uri, or {:ok, nil} when nothing is stored there.
The nodes at these URIs in one read, in the same order as the input (missing ones as nil).
Every message of a session, in order.
Line matches for query in full document content (L2) within scope_uri
or beneath it.
Whether the durable state can be reached at all.
Recursive document URIs under scope in deterministic order, paged.
Names of uri's direct children, sorted. {:error, :not_found} when the URI holds no directory.
Ids of every session, ordered, so an export can enumerate them.
Installed skill roots in URI order, paged. Each entry carries name,
owner, uri, files, enabled, and group_tag. filter supports
:substring, :owner, :include_disabled, and :group.
The stored embedding vectors of these URIs, keyed by URI as float32 blobs.
Records that the assertions with these ids were surfaced by a recall.
Pairs of active memories beneath prefix whose values are similar enough to
be possible conflicts. Reuses stored vectors and runs no inference.
{:error, :embeddings_unavailable} when conflicts cannot be evaluated.
Whether any assertion is recorded at uri, superseded and candidate ones included.
Counts of memories per type and per status, the number waiting for review, and the number never surfaced, computed by the store without materialising every memory.
Promotes the latest candidate at uri to the active assertion, superseding
any prior active one, and carries the promoted value through the read-through
document at uri. {:error, :no_candidate} when none is waiting.
Writes the committed document and its commit bookkeeping together.
Writes a document at uri, creating missing parent directories.
Stores a generated embedding for a job, unless its node has since been removed.
Stores a generated summary for a job, unless its node has since been removed.
Writes a memory's document, its assertion, and its required background work together, so a URI never holds a document no assertion backs, and a memory reported as recorded never lacks the job that makes it searchable.
How much work is outstanding and what went wrong.
Job counts by status.
Assertions at prefix or beneath it, most confident first.
Deletes every index row that names a URI the node store no longer holds.
Removes candidate assertions at uri without a trace. Active and superseded
history is untouched. {:error, :no_candidate} when none is waiting.
Removes uri and its whole subtree from every store keyed by URI.
Replaces the whole subtree at uri with files, in one atomic step.
Returns jobs left running by a previous run to pending.
Restores a session with a specific id and its messages in order.
Case-insensitive substring search over stored documents, optionally within
scope_prefix.
Nearest neighbours of a query embedding, optionally within scope_prefix.
Sets the enabled state for uri and its whole subtree.
Sets the operator group tag for uri and its whole subtree. Empty clears.
Tags are names only, never content.
The store's composition and footprint, read-only.
How many index rows name a URI the node store no longer holds.
Coverage of the vector index relative to what is stored.
Types
@type hit() :: %{ uri: String.t(), content: String.t() | nil, abstract: String.t() | nil, overview: String.t() | nil, score: float() }
A vector search hit, scored by the adapter.
@type job() :: %{ id: integer(), kind: atom(), payload: map(), attempts: integer(), max_attempts: integer() }
A unit of durable background work.
@type layer() :: :abstract | :overview
The layered summary a background worker produced.
@type line_match() :: %{ uri: String.t(), line_number: pos_integer(), excerpt: String.t() }
A content line match: document URI, one-based line number, and bounded excerpt.
Two active memories at distinct URIs that may conflict.
@type memory_row() :: %{ id: integer(), uri: String.t(), value: String.t(), confidence: float(), importance: float() | nil, source: String.t() | nil, status: :active | :superseded | :candidate, supersedes: integer() | nil, last_surfaced_at: integer() | nil, updated_at: integer(), truncated: boolean() | nil }
One assertion ever made at a memory's URI.
One message of a session, in order.
A path-discovery match: URI, name, and node kind, without document content.
One file of a skill, at path below the skill's root.
@type tree_node() :: %{ uri: String.t(), parent_uri: String.t() | nil, name: String.t(), kind: :doc | :dir, content: String.t() | nil, abstract: String.t() | nil, overview: String.t() | nil }
A stored document or directory.
Callbacks
Appends a message to a session, preserving order.
Drops queued work for uri or its descendants, in any state.
@callback child_specs(keyword()) :: [ Supervisor.child_spec() | module() | {module(), term()} ]
Supervised children the adapter needs, given the resolved path to the database file.
The content hash recorded for a (session, destination) commit, if any.
Marks a job done.
@callback count_jobs(String.t(), [String.t()]) :: non_neg_integer()
How many jobs target uri or its descendants and are in one of statuses.
@callback count_jobs_of_kind(atom()) :: non_neg_integer()
How many jobs of kind exist, in any status. A standing job has no URI to count by.
Creates a session and returns its id.
@callback defer_job(integer(), non_neg_integer()) :: :ok | {:error, term()}
Puts a job back without spending one of its attempts, for work that is merely waiting.
Claims the next runnable job of one of kinds, advancing its attempt count.
Filtered by kind so a worker never claims work it cannot run: a claimed job of another kind would be a job destroyed.
@callback document_count(String.t()) :: non_neg_integer()
How many documents are stored at prefix or beneath it.
Membership is exact-URI-or-descendant, the same rule find/4 and
grep_content/3 use, so a count of a subtree cannot include a sibling whose
name merely starts the same way. Directories are not counted.
Records durable background work.
Records a job failure, rescheduling it with backoff while attempts remain and marking it failed once they are exhausted.
reason is the worker's classified failure, persisted so a failed job can be
explained later without a log line. It is a classification -- never content,
a prompt, or a credential.
@callback find_paths(String.t(), String.t() | nil, pos_integer()) :: {:ok, [path_match()]} | {:error, term()}
Path-discovery matches for query within scope_uri or beneath it.
query is a validated literal substring (1 through 256 characters).
scope_uri is a validated exact URI or nil for the whole tree; scope
membership is exact-URI-or-descendant, so scope never matches a sibling
such as scope-old. Matching is case-insensitive over the URI path
(excluding the viking:// scheme), %, _, and \ match literally,
results are ordered by URI, and at most limit entries are returned
without document content.
The node at uri, or {:ok, nil} when nothing is stored there.
The nodes at these URIs in one read, in the same order as the input (missing ones as nil).
Every message of a session, in order.
@callback grep_content(String.t(), String.t() | nil, pos_integer()) :: {:ok, [line_match()]} | {:error, term()}
Line matches for query in full document content (L2) within scope_uri
or beneath it.
query is a validated literal substring (1 through 256 characters).
scope_uri is a validated exact URI or nil for the whole tree; scope
membership is exact-URI-or-descendant. Matching is case-insensitive over
L2 content only (abstracts and overviews never match), %, _, \,
and regular-expression metacharacters match literally, results are ordered
by URI then one-based line number, each excerpt contains the match and is
at most 280 characters, and at most limit entries are returned.
@callback healthy?() :: boolean()
Whether the durable state can be reached at all.
@callback list_all_documents(String.t(), pos_integer(), non_neg_integer(), map()) :: {:ok, {[map()], non_neg_integer()}} | {:error, term()}
Recursive document URIs under scope in deterministic order, paged.
Returns {rows, total} where rows carry uri, enabled, and group_tag
without blobs. filter supports :substring, :include_disabled, and
:group. limit/offset bound the page; total is the filtered count.
Names of uri's direct children, sorted. {:error, :not_found} when the URI holds no directory.
Ids of every session, ordered, so an export can enumerate them.
@callback list_skill_roots(pos_integer(), non_neg_integer(), map()) :: {:ok, {[map()], non_neg_integer()}} | {:error, term()}
Installed skill roots in URI order, paged. Each entry carries name,
owner, uri, files, enabled, and group_tag. filter supports
:substring, :owner, :include_disabled, and :group.
@callback load_memory_vectors([String.t()]) :: {:ok, %{required(String.t()) => binary()}} | {:error, term()}
The stored embedding vectors of these URIs, keyed by URI as float32 blobs.
Vectors come back from the active vector dimension only: a URI not yet embedded, or embedded under another dimensionality, is simply absent from the answer rather than an error. The read is storage-only -- it never waits on inference and never triggers model work -- because it exists so the term recall can rank from what is already on disk instead of re-embedding the candidate values it was handed.
Records that the assertions with these ids were surfaced by a recall.
@callback memory_conflict_pairs(String.t()) :: {:ok, [memory_conflict()]} | {:error, term()}
Pairs of active memories beneath prefix whose values are similar enough to
be possible conflicts. Reuses stored vectors and runs no inference.
{:error, :embeddings_unavailable} when conflicts cannot be evaluated.
Whether any assertion is recorded at uri, superseded and candidate ones included.
Counts of memories per type and per status, the number waiting for review, and the number never surfaced, computed by the store without materialising every memory.
Promotes the latest candidate at uri to the active assertion, superseding
any prior active one, and carries the promoted value through the read-through
document at uri. {:error, :no_candidate} when none is waiting.
@callback put_commit(String.t(), String.t(), String.t(), String.t()) :: {:ok, :ok} | {:error, term()}
Writes the committed document and its commit bookkeeping together.
Writes a document at uri, creating missing parent directories.
:abstract and :overview in opts are the caller's own layers; nil for
either keeps whatever is already stored, so a re-write does not discard a
layer it did not supply.
:jobs in opts is the required background work for the write, as a list
of {kind, payload} pairs. A provider persists the document and all jobs
as one storage outcome: success means both are durable, and a failure to
persist or enqueue any job leaves prior document, cache, and queue state
unchanged.
@callback put_embedding_result(integer(), String.t(), binary()) :: {:ok, :stored | :discarded} | {:error, term()}
Stores a generated embedding for a job, unless its node has since been removed.
Because the index is derived, an embedding for a URI the store no longer holds
is simply not wanted: the job is completed either way, and the result is
reported :discarded. A write at the URI meanwhile retires the previous
vector, so a re-write is never ranked on content it no longer has.
@callback put_layer_result(integer(), String.t(), layer(), String.t()) :: {:ok, :stored | :discarded} | {:error, term()}
Stores a generated summary for a job, unless its node has since been removed.
@callback put_memory(String.t(), String.t(), float(), String.t() | nil, keyword()) :: {:ok, :ok} | {:error, term()}
Writes a memory's document, its assertion, and its required background work together, so a URI never holds a document no assertion backs, and a memory reported as recorded never lacks the job that makes it searchable.
opts carries :importance (0.0..1.0, defaults when absent), :status
(:active by default, or :candidate for a record awaiting promotion), and
:jobs -- the required background work for the write, as a list of
{kind, payload} pairs, as put_document/3 takes. An active re-record of the
value the URI already asserts is a no-op: the URI keeps its one assertion. A
candidate recorded at a URI that already holds a document does not overwrite
that document: the active belief is what the document is read through, and
promote_memory/1 is what repairs it on promotion. A candidate at a URI
holding no document still gets one, so the URI is a document of the tree
like any other.
Implementations SHOULD accept the options as a trailing keyword with a default, so callers that pass none keep working unchanged.
@callback queue_detail(pos_integer()) :: {:ok, %{oldest_pending_ms: non_neg_integer() | nil, failed: [map()]}} | {:error, term()}
How much work is outstanding and what went wrong.
Extends queue_stats/0 with the age of the oldest pending job and a bounded
list of failed jobs carrying the reason, so an operator can tell a queue that
is backing up from one whose work is failing.
oldest_pending_ms is the age of the longest-waiting runnable job and is
nil when nothing is pending. failed is newest-first and bounded. Each
entry's last_error is a classified reason -- never content, a prompt, or a
credential.
Job counts by status.
@callback recall_memories( String.t(), String.t() | nil, [:active | :superseded | :candidate], keyword() ) :: {:ok, [memory_row()]} | {:error, term()}
Assertions at prefix or beneath it, most confident first.
opts bounds the read the way search/2 bounds its legs:
:top_k- the bound this read materialises, 1..500 (default 500). Out of range is{:error, {:invalid_limit, value}}rather than a silently substituted bound. Every returned row reports whether the bound was reached astruncated: true | false, so a caller can tell "there is nothing more" from "there is more than I asked for".:source- restricts to assertions recorded from that provenance.
Implementations SHOULD accept opts as a fourth argument with a default,
so existing three-argument calls keep working.
@callback reclaim_vector_index() :: :ok | {:error, term()}
Deletes every index row that names a URI the node store no longer holds.
The reclamation half of the drift contract: removal makes the URI unsearchable immediately, and this reclaims the row afterwards. Rows naming live URIs are left alone, and a store with nothing to reclaim is a no-op.
Removes candidate assertions at uri without a trace. Active and superseded
history is untouched. {:error, :no_candidate} when none is waiting.
@callback remove_subtree(String.t()) :: :ok | {:error, :not_found} | {:error, :is_root}
Removes uri and its whole subtree from every store keyed by URI.
@callback replace_skill(String.t(), [skill_file()]) :: {:ok, %{replaced: boolean(), files: pos_integer()}} | {:error, term()}
Replaces the whole subtree at uri with files, in one atomic step.
Each file is stored at uri followed by its path, with the directories
above it created, and the usual asynchronous work is enqueued for it, so a
file written this way is indistinguishable from one written with
put_document/3.
What was at uri before is removed, from every store keyed by URI, in the same
step: a file the new set does not mention must not survive it, and a failure
part way through must leave the previous contents exactly as they were.
Answers whether anything was there to replace, which is what tells a caller that an import overwrote a skill rather than adding one.
@callback reset_running_jobs() :: :ok | {:error, term()}
Returns jobs left running by a previous run to pending.
@callback restore_session(String.t(), [message()]) :: {:ok, :imported | :skipped} | {:error, term()}
Restores a session with a specific id and its messages in order.
Creates the session when absent and appends the messages. When the session
already holds exactly these messages (compared as {role, content} pairs in
order) it reports {:ok, :skipped} and writes nothing. When it holds
different messages it reports {:error, {:session_conflict, id}} and leaves
the stored session untouched, so an import never silently overwrites a live
conversation.
@callback search_keyword(String.t(), String.t() | nil, pos_integer(), [atom()] | nil) :: {:ok, [tree_node()]} | {:error, term()}
Case-insensitive substring search over stored documents, optionally within
scope_prefix.
At most limit results, ordered deterministically by URI. The bound is
applied by the provider rather than by the caller, so a query matching more
documents than were asked for does not materialize all of them.
fields is the set of per-result fields the caller asked for, or nil for
the default set (URI, content, abstract, overview). A projection is applied
by the statement that reads the rows rather than after they are materialized,
so a caller that did not ask for content does not pay to read it. :uri is
returned regardless of the selection, because a result without its URI
cannot be used to fetch the document it names. The internal columns a
provider still needs (enabled, group_tag) are read as well and reported
alongside, so a filtering caller can drop them itself.
@callback search_vector(binary(), pos_integer(), String.t() | nil, [atom()] | nil) :: {:ok, [hit()]} | {:error, term()}
Nearest neighbours of a query embedding, optionally within scope_prefix.
The index is derived data: a row may survive for URI the node store no longer
holds, and the search is what keeps that inert. A hit is only returned when
the store can still hold the URI it names, so a removed URI is never surfaced
while a row for it survives. Candidates that cannot be hydrated do not count
towards top_k. Out of range top_k values and a query of the wrong
dimensionality are refused rather than answered approximately.
fields is the set of per-result fields the caller asked for, or nil for
the default set (URI, content, abstract, overview, score). A projection is
applied by the statement that reads the rows rather than after they are
materialized, so a caller that did not ask for content does not pay to read
it. :uri is returned regardless of the selection.
Sets the enabled state for uri and its whole subtree.
Disabled is blocked-from-use, not deleted: rows stay readable and editable but are excluded from search and default listings.
Sets the operator group tag for uri and its whole subtree. Empty clears.
Tags are names only, never content.
@callback stats() :: {:ok, %{ documents: non_neg_integer(), directories: non_neg_integer(), by_top_subtree: %{optional(String.t()) => non_neg_integer()}, db_bytes: non_neg_integer() | nil, wal_bytes: non_neg_integer() | nil }} | {:error, term()}
The store's composition and footprint, read-only.
documents and directories count every node of each kind;
by_top_subtree counts documents filed at or below each top-level subtree
(resources, user, peers, ...). db_bytes and wal_bytes are the sizes
of the database file and its write-ahead log, which is a fact about one
implementation: a provider with no single database file reports nil rather
than inventing one.
This counts; it never walks the tree, so its cost does not grow with depth.
@callback vector_index_reclaimable() :: {:ok, non_neg_integer()} | {:error, term()}
How many index rows name a URI the node store no longer holds.
Drift, reported without scanning content. Reclaimable rows are already invisible to search -- the read path joins them away -- so this counts occupancy rather than correctness, and it is what tells an operator that reclamation is falling behind. Reported without touching anything.
@callback vector_index_stats() :: {:ok, %{ available: boolean(), vectors: non_neg_integer() | nil, documents: non_neg_integer() }} | {:error, term()}
Coverage of the vector index relative to what is stored.
available is whether the index can be queried at all; when it is not,
vectors is nil rather than zero, because "not available" and "holds
nothing" are different facts an operator needs to tell apart. vectors
counts indexed rows and documents counts the documents they describe, so a
store can report how much of its content is vector-searchable. reclaimable
counts rows whose URI the store no longer holds -- drift that is harmless to
search results but still occupies the index, and the number that says
reclamation is falling behind rather than merely that it has not run yet.