AgentDb.Core.Storage behaviour (agent_db v0.1.0)

Copy Markdown View Source

Durable state the context store keeps, expressed without reference to any particular database.

A storage adapter owns the authoritative records: documents, memory assertions, sessions, session commit bookkeeping, and the persisted job queue. Those records have to agree with each other, so the contract exposes whole operations rather than connections or statements: a workflow asks for a subtree to be removed, not for a transaction to be opened.

A vector index is different in kind. It is derived: every row in it can be reproduced by work the store already persisted. That makes it safe to let the index lag — it is regenerated, not recovered — and it means the guarantee belongs in the read rather than in the write. search_vector/4 therefore never returns a URI the node store cannot hold, so a row naming a removed URI is inert until reclamation deletes it, and a re-write retires the URI's previous vector so a live document is never ranked on content it no longer has. Drift remains a cost, so it is bounded by cancel_jobs/1 and reclaimed rather than silently accumulated.

Invariants an implementation must uphold

  • A write is durable before it is acknowledged, and a cache in front of the adapter may never serve anything the adapter has not already committed.
  • The authoritative stores change together. remove_subtree/1 removes a URI and everything beneath it from the node store, the memory store, the commit bookkeeping and the job queue in one step, and replace_skill/2 has the same two properties for a whole subtree. A rejected removal leaves all of that untouched.
  • A vector hit is only ever returned for a URI the node store still holds, so a removed URI is never surfaced even while a row naming it survives in the index. This is the guarantee that lets removal skip the index and reclamation pick up afterwards.
  • A write retires the previous vector for the URI it writes, so a document is never ranked against content it no longer has. Retiring derived data only ever makes vector search return less, never something the store cannot hold.
  • A result produced by work that started before a removal never brings the removed URI back. put_embedding_result/3 and put_layer_result/4 check that the node is still there before writing the result and report :discarded when it is not, which avoids most orphans outright; the read-path join is what holds for the rest.

Callbacks exchange core values and {:ok, _} / {:error, _} results, never adapter-specific structs or exceptions.

Summary

Types

A vector search hit, scored by the adapter.

A unit of durable background work.

The layered summary a background worker produced.

A content line match: document URI, one-based line number, and bounded excerpt.

Two active memories at distinct URIs that may conflict.

One assertion ever made at a memory's URI.

One message of a session, in order.

A path-discovery match: URI, name, and node kind, without document content.

One file of a skill, at path below the skill's root.

A stored document or directory.

Callbacks

Appends a message to a session, preserving order.

Drops queued work for uri or its descendants, in any state.

Supervised children the adapter needs, given the resolved path to the database file.

The content hash recorded for a (session, destination) commit, if any.

Marks a job done.

How many jobs target uri or its descendants and are in one of statuses.

How many jobs of kind exist, in any status. A standing job has no URI to count by.

Creates a session and returns its id.

Puts a job back without spending one of its attempts, for work that is merely waiting.

Claims the next runnable job of one of kinds, advancing its attempt count.

How many documents are stored at prefix or beneath it.

Records durable background work.

Records a job failure, rescheduling it with backoff while attempts remain and marking it failed once they are exhausted.

Path-discovery matches for query within scope_uri or beneath it.

The node at uri, or {:ok, nil} when nothing is stored there.

The nodes at these URIs in one read, in the same order as the input (missing ones as nil).

Every message of a session, in order.

Line matches for query in full document content (L2) within scope_uri or beneath it.

Whether the durable state can be reached at all.

Recursive document URIs under scope in deterministic order, paged.

Names of uri's direct children, sorted. {:error, :not_found} when the URI holds no directory.

Ids of every session, ordered, so an export can enumerate them.

Installed skill roots in URI order, paged. Each entry carries name, owner, uri, files, enabled, and group_tag. filter supports :substring, :owner, :include_disabled, and :group.

The stored embedding vectors of these URIs, keyed by URI as float32 blobs.

Records that the assertions with these ids were surfaced by a recall.

Pairs of active memories beneath prefix whose values are similar enough to be possible conflicts. Reuses stored vectors and runs no inference. {:error, :embeddings_unavailable} when conflicts cannot be evaluated.

Whether any assertion is recorded at uri, superseded and candidate ones included.

Counts of memories per type and per status, the number waiting for review, and the number never surfaced, computed by the store without materialising every memory.

Promotes the latest candidate at uri to the active assertion, superseding any prior active one, and carries the promoted value through the read-through document at uri. {:error, :no_candidate} when none is waiting.

Writes the committed document and its commit bookkeeping together.

Writes a document at uri, creating missing parent directories.

Stores a generated embedding for a job, unless its node has since been removed.

Stores a generated summary for a job, unless its node has since been removed.

Writes a memory's document, its assertion, and its required background work together, so a URI never holds a document no assertion backs, and a memory reported as recorded never lacks the job that makes it searchable.

How much work is outstanding and what went wrong.

Job counts by status.

Assertions at prefix or beneath it, most confident first.

Deletes every index row that names a URI the node store no longer holds.

Removes candidate assertions at uri without a trace. Active and superseded history is untouched. {:error, :no_candidate} when none is waiting.

Removes uri and its whole subtree from every store keyed by URI.

Replaces the whole subtree at uri with files, in one atomic step.

Returns jobs left running by a previous run to pending.

Restores a session with a specific id and its messages in order.

Case-insensitive substring search over stored documents, optionally within scope_prefix.

Nearest neighbours of a query embedding, optionally within scope_prefix.

Sets the enabled state for uri and its whole subtree.

Sets the operator group tag for uri and its whole subtree. Empty clears. Tags are names only, never content.

The store's composition and footprint, read-only.

How many index rows name a URI the node store no longer holds.

Coverage of the vector index relative to what is stored.

Types

hit()

@type hit() :: %{
  uri: String.t(),
  content: String.t() | nil,
  abstract: String.t() | nil,
  overview: String.t() | nil,
  score: float()
}

A vector search hit, scored by the adapter.

job()

@type job() :: %{
  id: integer(),
  kind: atom(),
  payload: map(),
  attempts: integer(),
  max_attempts: integer()
}

A unit of durable background work.

layer()

@type layer() :: :abstract | :overview

The layered summary a background worker produced.

line_match()

@type line_match() :: %{
  uri: String.t(),
  line_number: pos_integer(),
  excerpt: String.t()
}

A content line match: document URI, one-based line number, and bounded excerpt.

memory_conflict()

@type memory_conflict() :: %{
  uri_a: String.t(),
  uri_b: String.t(),
  similarity: float()
}

Two active memories at distinct URIs that may conflict.

memory_row()

@type memory_row() :: %{
  id: integer(),
  uri: String.t(),
  value: String.t(),
  confidence: float(),
  importance: float() | nil,
  source: String.t() | nil,
  status: :active | :superseded | :candidate,
  supersedes: integer() | nil,
  last_surfaced_at: integer() | nil,
  updated_at: integer(),
  truncated: boolean() | nil
}

One assertion ever made at a memory's URI.

message()

@type message() :: %{seq: integer(), role: atom(), content: String.t()}

One message of a session, in order.

path_match()

@type path_match() :: %{uri: String.t(), name: String.t(), kind: :doc | :dir}

A path-discovery match: URI, name, and node kind, without document content.

skill_file()

@type skill_file() :: %{path: [String.t()], content: String.t()}

One file of a skill, at path below the skill's root.

tree_node()

@type tree_node() :: %{
  uri: String.t(),
  parent_uri: String.t() | nil,
  name: String.t(),
  kind: :doc | :dir,
  content: String.t() | nil,
  abstract: String.t() | nil,
  overview: String.t() | nil
}

A stored document or directory.

Callbacks

append_message(t, atom, t)

@callback append_message(String.t(), atom(), String.t()) :: :ok | {:error, term()}

Appends a message to a session, preserving order.

cancel_jobs(t)

@callback cancel_jobs(String.t()) :: :ok | {:error, term()}

Drops queued work for uri or its descendants, in any state.

child_specs(keyword)

@callback child_specs(keyword()) :: [
  Supervisor.child_spec() | module() | {module(), term()}
]

Supervised children the adapter needs, given the resolved path to the database file.

commit_hash(t, t)

@callback commit_hash(String.t(), String.t()) ::
  {:ok, String.t() | nil} | {:error, term()}

The content hash recorded for a (session, destination) commit, if any.

complete_job(integer)

@callback complete_job(integer()) :: :ok | {:error, term()}

Marks a job done.

count_jobs(t, list)

@callback count_jobs(String.t(), [String.t()]) :: non_neg_integer()

How many jobs target uri or its descendants and are in one of statuses.

count_jobs_of_kind(atom)

@callback count_jobs_of_kind(atom()) :: non_neg_integer()

How many jobs of kind exist, in any status. A standing job has no URI to count by.

create_session()

@callback create_session() :: {:ok, String.t()} | {:error, term()}

Creates a session and returns its id.

defer_job(integer, non_neg_integer)

@callback defer_job(integer(), non_neg_integer()) :: :ok | {:error, term()}

Puts a job back without spending one of its attempts, for work that is merely waiting.

dequeue_job(list)

@callback dequeue_job([atom()]) :: {:ok, job()} | {:error, :empty} | {:error, term()}

Claims the next runnable job of one of kinds, advancing its attempt count.

Filtered by kind so a worker never claims work it cannot run: a claimed job of another kind would be a job destroyed.

document_count(t)

@callback document_count(String.t()) :: non_neg_integer()

How many documents are stored at prefix or beneath it.

Membership is exact-URI-or-descendant, the same rule find/4 and grep_content/3 use, so a count of a subtree cannot include a sibling whose name merely starts the same way. Directories are not counted.

enqueue_job(atom, map)

@callback enqueue_job(atom(), map()) :: {:ok, integer()} | {:error, term()}

Records durable background work.

fail_job(integer, arg2)

@callback fail_job(integer(), String.t() | nil) :: :ok | {:error, term()}

Records a job failure, rescheduling it with backoff while attempts remain and marking it failed once they are exhausted.

reason is the worker's classified failure, persisted so a failed job can be explained later without a log line. It is a classification -- never content, a prompt, or a credential.

find_paths(t, arg2, pos_integer)

@callback find_paths(String.t(), String.t() | nil, pos_integer()) ::
  {:ok, [path_match()]} | {:error, term()}

Path-discovery matches for query within scope_uri or beneath it.

query is a validated literal substring (1 through 256 characters). scope_uri is a validated exact URI or nil for the whole tree; scope membership is exact-URI-or-descendant, so scope never matches a sibling such as scope-old. Matching is case-insensitive over the URI path (excluding the viking:// scheme), %, _, and \ match literally, results are ordered by URI, and at most limit entries are returned without document content.

get_node(t)

@callback get_node(String.t()) :: {:ok, tree_node() | nil} | {:error, term()}

The node at uri, or {:ok, nil} when nothing is stored there.

get_nodes(list)

@callback get_nodes([String.t()]) :: {:ok, [tree_node() | nil]} | {:error, term()}

The nodes at these URIs in one read, in the same order as the input (missing ones as nil).

get_session(t)

@callback get_session(String.t()) :: {:ok, [message()]} | {:error, term()}

Every message of a session, in order.

grep_content(t, arg2, pos_integer)

@callback grep_content(String.t(), String.t() | nil, pos_integer()) ::
  {:ok, [line_match()]} | {:error, term()}

Line matches for query in full document content (L2) within scope_uri or beneath it.

query is a validated literal substring (1 through 256 characters). scope_uri is a validated exact URI or nil for the whole tree; scope membership is exact-URI-or-descendant. Matching is case-insensitive over L2 content only (abstracts and overviews never match), %, _, \, and regular-expression metacharacters match literally, results are ordered by URI then one-based line number, each excerpt contains the match and is at most 280 characters, and at most limit entries are returned.

healthy?()

@callback healthy?() :: boolean()

Whether the durable state can be reached at all.

list_all_documents(t, pos_integer, non_neg_integer, map)

@callback list_all_documents(String.t(), pos_integer(), non_neg_integer(), map()) ::
  {:ok, {[map()], non_neg_integer()}} | {:error, term()}

Recursive document URIs under scope in deterministic order, paged.

Returns {rows, total} where rows carry uri, enabled, and group_tag without blobs. filter supports :substring, :include_disabled, and :group. limit/offset bound the page; total is the filtered count.

list_children(t)

@callback list_children(String.t()) :: {:ok, [String.t()]} | {:error, term()}

Names of uri's direct children, sorted. {:error, :not_found} when the URI holds no directory.

list_session_ids()

@callback list_session_ids() :: {:ok, [String.t()]} | {:error, term()}

Ids of every session, ordered, so an export can enumerate them.

list_skill_roots(pos_integer, non_neg_integer, map)

@callback list_skill_roots(pos_integer(), non_neg_integer(), map()) ::
  {:ok, {[map()], non_neg_integer()}} | {:error, term()}

Installed skill roots in URI order, paged. Each entry carries name, owner, uri, files, enabled, and group_tag. filter supports :substring, :owner, :include_disabled, and :group.

load_memory_vectors(list)

@callback load_memory_vectors([String.t()]) ::
  {:ok, %{required(String.t()) => binary()}} | {:error, term()}

The stored embedding vectors of these URIs, keyed by URI as float32 blobs.

Vectors come back from the active vector dimension only: a URI not yet embedded, or embedded under another dimensionality, is simply absent from the answer rather than an error. The read is storage-only -- it never waits on inference and never triggers model work -- because it exists so the term recall can rank from what is already on disk instead of re-embedding the candidate values it was handed.

mark_memories_surfaced(list)

@callback mark_memories_surfaced([integer()]) :: :ok | {:error, term()}

Records that the assertions with these ids were surfaced by a recall.

memory_conflict_pairs(t)

@callback memory_conflict_pairs(String.t()) ::
  {:ok, [memory_conflict()]} | {:error, term()}

Pairs of active memories beneath prefix whose values are similar enough to be possible conflicts. Reuses stored vectors and runs no inference. {:error, :embeddings_unavailable} when conflicts cannot be evaluated.

memory_recorded?(t)

@callback memory_recorded?(String.t()) :: {:ok, boolean()} | {:error, term()}

Whether any assertion is recorded at uri, superseded and candidate ones included.

memory_stats()

@callback memory_stats() :: {:ok, map()} | {:error, term()}

Counts of memories per type and per status, the number waiting for review, and the number never surfaced, computed by the store without materialising every memory.

promote_memory(t)

@callback promote_memory(String.t()) :: {:ok, :promoted} | {:error, term()}

Promotes the latest candidate at uri to the active assertion, superseding any prior active one, and carries the promoted value through the read-through document at uri. {:error, :no_candidate} when none is waiting.

put_commit(t, t, t, t)

@callback put_commit(String.t(), String.t(), String.t(), String.t()) ::
  {:ok, :ok} | {:error, term()}

Writes the committed document and its commit bookkeeping together.

put_document(t, t, keyword)

@callback put_document(String.t(), String.t(), keyword()) :: {:ok, :ok} | {:error, term()}

Writes a document at uri, creating missing parent directories.

:abstract and :overview in opts are the caller's own layers; nil for either keeps whatever is already stored, so a re-write does not discard a layer it did not supply.

:jobs in opts is the required background work for the write, as a list of {kind, payload} pairs. A provider persists the document and all jobs as one storage outcome: success means both are durable, and a failure to persist or enqueue any job leaves prior document, cache, and queue state unchanged.

put_embedding_result(integer, t, binary)

@callback put_embedding_result(integer(), String.t(), binary()) ::
  {:ok, :stored | :discarded} | {:error, term()}

Stores a generated embedding for a job, unless its node has since been removed.

Because the index is derived, an embedding for a URI the store no longer holds is simply not wanted: the job is completed either way, and the result is reported :discarded. A write at the URI meanwhile retires the previous vector, so a re-write is never ranked on content it no longer has.

put_layer_result(integer, t, layer, t)

@callback put_layer_result(integer(), String.t(), layer(), String.t()) ::
  {:ok, :stored | :discarded} | {:error, term()}

Stores a generated summary for a job, unless its node has since been removed.

put_memory(t, t, float, arg4, keyword)

@callback put_memory(String.t(), String.t(), float(), String.t() | nil, keyword()) ::
  {:ok, :ok} | {:error, term()}

Writes a memory's document, its assertion, and its required background work together, so a URI never holds a document no assertion backs, and a memory reported as recorded never lacks the job that makes it searchable.

opts carries :importance (0.0..1.0, defaults when absent), :status (:active by default, or :candidate for a record awaiting promotion), and :jobs -- the required background work for the write, as a list of {kind, payload} pairs, as put_document/3 takes. An active re-record of the value the URI already asserts is a no-op: the URI keeps its one assertion. A candidate recorded at a URI that already holds a document does not overwrite that document: the active belief is what the document is read through, and promote_memory/1 is what repairs it on promotion. A candidate at a URI holding no document still gets one, so the URI is a document of the tree like any other.

Implementations SHOULD accept the options as a trailing keyword with a default, so callers that pass none keep working unchanged.

queue_detail(pos_integer)

@callback queue_detail(pos_integer()) ::
  {:ok, %{oldest_pending_ms: non_neg_integer() | nil, failed: [map()]}}
  | {:error, term()}

How much work is outstanding and what went wrong.

Extends queue_stats/0 with the age of the oldest pending job and a bounded list of failed jobs carrying the reason, so an operator can tell a queue that is backing up from one whose work is failing.

oldest_pending_ms is the age of the longest-waiting runnable job and is nil when nothing is pending. failed is newest-first and bounded. Each entry's last_error is a classified reason -- never content, a prompt, or a credential.

queue_stats()

@callback queue_stats() :: {:ok, map()} | {:error, term()}

Job counts by status.

recall_memories(t, arg2, list, keyword)

@callback recall_memories(
  String.t(),
  String.t() | nil,
  [:active | :superseded | :candidate],
  keyword()
) ::
  {:ok, [memory_row()]} | {:error, term()}

Assertions at prefix or beneath it, most confident first.

opts bounds the read the way search/2 bounds its legs:

  • :top_k - the bound this read materialises, 1..500 (default 500). Out of range is {:error, {:invalid_limit, value}} rather than a silently substituted bound. Every returned row reports whether the bound was reached as truncated: true | false, so a caller can tell "there is nothing more" from "there is more than I asked for".
  • :source - restricts to assertions recorded from that provenance.

Implementations SHOULD accept opts as a fourth argument with a default, so existing three-argument calls keep working.

reclaim_vector_index()

@callback reclaim_vector_index() :: :ok | {:error, term()}

Deletes every index row that names a URI the node store no longer holds.

The reclamation half of the drift contract: removal makes the URI unsearchable immediately, and this reclaims the row afterwards. Rows naming live URIs are left alone, and a store with nothing to reclaim is a no-op.

reject_memory_candidate(t)

@callback reject_memory_candidate(String.t()) :: :ok | {:error, term()}

Removes candidate assertions at uri without a trace. Active and superseded history is untouched. {:error, :no_candidate} when none is waiting.

remove_subtree(t)

@callback remove_subtree(String.t()) :: :ok | {:error, :not_found} | {:error, :is_root}

Removes uri and its whole subtree from every store keyed by URI.

replace_skill(t, list)

@callback replace_skill(String.t(), [skill_file()]) ::
  {:ok, %{replaced: boolean(), files: pos_integer()}} | {:error, term()}

Replaces the whole subtree at uri with files, in one atomic step.

Each file is stored at uri followed by its path, with the directories above it created, and the usual asynchronous work is enqueued for it, so a file written this way is indistinguishable from one written with put_document/3.

What was at uri before is removed, from every store keyed by URI, in the same step: a file the new set does not mention must not survive it, and a failure part way through must leave the previous contents exactly as they were.

Answers whether anything was there to replace, which is what tells a caller that an import overwrote a skill rather than adding one.

reset_running_jobs()

@callback reset_running_jobs() :: :ok | {:error, term()}

Returns jobs left running by a previous run to pending.

restore_session(t, list)

@callback restore_session(String.t(), [message()]) ::
  {:ok, :imported | :skipped} | {:error, term()}

Restores a session with a specific id and its messages in order.

Creates the session when absent and appends the messages. When the session already holds exactly these messages (compared as {role, content} pairs in order) it reports {:ok, :skipped} and writes nothing. When it holds different messages it reports {:error, {:session_conflict, id}} and leaves the stored session untouched, so an import never silently overwrites a live conversation.

search_keyword(t, arg2, pos_integer, arg4)

@callback search_keyword(String.t(), String.t() | nil, pos_integer(), [atom()] | nil) ::
  {:ok, [tree_node()]} | {:error, term()}

Case-insensitive substring search over stored documents, optionally within scope_prefix.

At most limit results, ordered deterministically by URI. The bound is applied by the provider rather than by the caller, so a query matching more documents than were asked for does not materialize all of them.

fields is the set of per-result fields the caller asked for, or nil for the default set (URI, content, abstract, overview). A projection is applied by the statement that reads the rows rather than after they are materialized, so a caller that did not ask for content does not pay to read it. :uri is returned regardless of the selection, because a result without its URI cannot be used to fetch the document it names. The internal columns a provider still needs (enabled, group_tag) are read as well and reported alongside, so a filtering caller can drop them itself.

search_vector(binary, pos_integer, arg3, arg4)

@callback search_vector(binary(), pos_integer(), String.t() | nil, [atom()] | nil) ::
  {:ok, [hit()]} | {:error, term()}

Nearest neighbours of a query embedding, optionally within scope_prefix.

The index is derived data: a row may survive for URI the node store no longer holds, and the search is what keeps that inert. A hit is only returned when the store can still hold the URI it names, so a removed URI is never surfaced while a row for it survives. Candidates that cannot be hydrated do not count towards top_k. Out of range top_k values and a query of the wrong dimensionality are refused rather than answered approximately.

fields is the set of per-result fields the caller asked for, or nil for the default set (URI, content, abstract, overview, score). A projection is applied by the statement that reads the rows rather than after they are materialized, so a caller that did not ask for content does not pay to read it. :uri is returned regardless of the selection.

set_node_enabled(t, boolean)

@callback set_node_enabled(String.t(), boolean()) :: :ok | {:error, term()}

Sets the enabled state for uri and its whole subtree.

Disabled is blocked-from-use, not deleted: rows stay readable and editable but are excluded from search and default listings.

set_node_group(t, t)

@callback set_node_group(String.t(), String.t()) :: :ok | {:error, term()}

Sets the operator group tag for uri and its whole subtree. Empty clears. Tags are names only, never content.

stats()

@callback stats() ::
  {:ok,
   %{
     documents: non_neg_integer(),
     directories: non_neg_integer(),
     by_top_subtree: %{optional(String.t()) => non_neg_integer()},
     db_bytes: non_neg_integer() | nil,
     wal_bytes: non_neg_integer() | nil
   }}
  | {:error, term()}

The store's composition and footprint, read-only.

documents and directories count every node of each kind; by_top_subtree counts documents filed at or below each top-level subtree (resources, user, peers, ...). db_bytes and wal_bytes are the sizes of the database file and its write-ahead log, which is a fact about one implementation: a provider with no single database file reports nil rather than inventing one.

This counts; it never walks the tree, so its cost does not grow with depth.

vector_index_reclaimable()

@callback vector_index_reclaimable() :: {:ok, non_neg_integer()} | {:error, term()}

How many index rows name a URI the node store no longer holds.

Drift, reported without scanning content. Reclaimable rows are already invisible to search -- the read path joins them away -- so this counts occupancy rather than correctness, and it is what tells an operator that reclamation is falling behind. Reported without touching anything.

vector_index_stats()

@callback vector_index_stats() ::
  {:ok,
   %{
     available: boolean(),
     vectors: non_neg_integer() | nil,
     documents: non_neg_integer()
   }}
  | {:error, term()}

Coverage of the vector index relative to what is stored.

available is whether the index can be queried at all; when it is not, vectors is nil rather than zero, because "not available" and "holds nothing" are different facts an operator needs to tell apart. vectors counts indexed rows and documents counts the documents they describe, so a store can report how much of its content is vector-searchable. reclaimable counts rows whose URI the store no longer holds -- drift that is harmless to search results but still occupies the index, and the number that says reclamation is falling behind rather than merely that it has not run yet.