# `AgentDb.Core.Storage`
[🔗](https://github.com/manhvu/agent_db/blob/v0.1.0/lib/agent_db/core/storage.ex#L1)

Durable state the context store keeps, expressed without reference to any
particular database.

A storage adapter owns the authoritative records: documents, memory
assertions, sessions, session commit bookkeeping, and the persisted job queue.
Those records have to agree with each other, so the contract exposes whole
operations rather than connections or statements: a workflow asks for a
subtree to be removed, not for a transaction to be opened.

A vector index is different in kind. It is derived: every row in it can be
reproduced by work the store already persisted. That makes it safe to let the
index lag — it is regenerated, not recovered — and it means the guarantee
belongs in the read rather than in the write. `search_vector/4` therefore
never returns a URI the node store cannot hold, so a row naming a removed
URI is inert until reclamation deletes it, and a re-write retires the URI's
previous vector so a live document is never ranked on content it no longer
has. Drift remains a cost, so it is bounded by `cancel_jobs/1` and reclaimed
rather than silently accumulated.

## Invariants an implementation must uphold

  * A write is durable before it is acknowledged, and a cache in front of the
    adapter may never serve anything the adapter has not already committed.
  * The authoritative stores change together. `remove_subtree/1` removes a URI
    and everything beneath it from the node store, the memory store, the commit
    bookkeeping and the job queue in one step, and `replace_skill/2` has the
    same two properties for a whole subtree. A rejected removal leaves all of
    that untouched.
  * A vector hit is only ever returned for a URI the node store still holds, so
    a removed URI is never surfaced even while a row naming it survives in the
    index. This is the guarantee that lets removal skip the index and reclamation
    pick up afterwards.
  * A write retires the previous vector for the URI it writes, so a document is
    never ranked against content it no longer has. Retiring derived data only
    ever makes vector search return less, never something the store cannot hold.
  * A result produced by work that started before a removal never brings the
    removed URI back. `put_embedding_result/3` and `put_layer_result/4` check
    that the node is still there before writing the result and report
    `:discarded` when it is not, which avoids most orphans outright; the
    read-path join is what holds for the rest.

Callbacks exchange core values and `{:ok, _}` / `{:error, _}` results, never
adapter-specific structs or exceptions.

# `hit`

```elixir
@type hit() :: %{
  uri: String.t(),
  content: String.t() | nil,
  abstract: String.t() | nil,
  overview: String.t() | nil,
  score: float()
}
```

A vector search hit, scored by the adapter.

# `job`

```elixir
@type job() :: %{
  id: integer(),
  kind: atom(),
  payload: map(),
  attempts: integer(),
  max_attempts: integer()
}
```

A unit of durable background work.

# `layer`

```elixir
@type layer() :: :abstract | :overview
```

The layered summary a background worker produced.

# `line_match`

```elixir
@type line_match() :: %{
  uri: String.t(),
  line_number: pos_integer(),
  excerpt: String.t()
}
```

A content line match: document URI, one-based line number, and bounded excerpt.

# `memory_conflict`

```elixir
@type memory_conflict() :: %{
  uri_a: String.t(),
  uri_b: String.t(),
  similarity: float()
}
```

Two active memories at distinct URIs that may conflict.

# `memory_row`

```elixir
@type memory_row() :: %{
  id: integer(),
  uri: String.t(),
  value: String.t(),
  confidence: float(),
  importance: float() | nil,
  source: String.t() | nil,
  status: :active | :superseded | :candidate,
  supersedes: integer() | nil,
  last_surfaced_at: integer() | nil,
  updated_at: integer(),
  truncated: boolean() | nil
}
```

One assertion ever made at a memory's URI.

# `message`

```elixir
@type message() :: %{seq: integer(), role: atom(), content: String.t()}
```

One message of a session, in order.

# `path_match`

```elixir
@type path_match() :: %{uri: String.t(), name: String.t(), kind: :doc | :dir}
```

A path-discovery match: URI, name, and node kind, without document content.

# `skill_file`

```elixir
@type skill_file() :: %{path: [String.t()], content: String.t()}
```

One file of a skill, at `path` below the skill's root.

# `tree_node`

```elixir
@type tree_node() :: %{
  uri: String.t(),
  parent_uri: String.t() | nil,
  name: String.t(),
  kind: :doc | :dir,
  content: String.t() | nil,
  abstract: String.t() | nil,
  overview: String.t() | nil
}
```

A stored document or directory.

# `append_message`

```elixir
@callback append_message(String.t(), atom(), String.t()) :: :ok | {:error, term()}
```

Appends a message to a session, preserving order.

# `cancel_jobs`

```elixir
@callback cancel_jobs(String.t()) :: :ok | {:error, term()}
```

Drops queued work for `uri` or its descendants, in any state.

# `child_specs`

```elixir
@callback child_specs(keyword()) :: [
  Supervisor.child_spec() | module() | {module(), term()}
]
```

Supervised children the adapter needs, given the resolved `path` to the database file.

# `commit_hash`

```elixir
@callback commit_hash(String.t(), String.t()) ::
  {:ok, String.t() | nil} | {:error, term()}
```

The content hash recorded for a (session, destination) commit, if any.

# `complete_job`

```elixir
@callback complete_job(integer()) :: :ok | {:error, term()}
```

Marks a job done.

# `count_jobs`

```elixir
@callback count_jobs(String.t(), [String.t()]) :: non_neg_integer()
```

How many jobs target `uri` or its descendants and are in one of `statuses`.

# `count_jobs_of_kind`

```elixir
@callback count_jobs_of_kind(atom()) :: non_neg_integer()
```

How many jobs of `kind` exist, in any status. A standing job has no URI to count by.

# `create_session`

```elixir
@callback create_session() :: {:ok, String.t()} | {:error, term()}
```

Creates a session and returns its id.

# `defer_job`

```elixir
@callback defer_job(integer(), non_neg_integer()) :: :ok | {:error, term()}
```

Puts a job back without spending one of its attempts, for work that is merely waiting.

# `dequeue_job`

```elixir
@callback dequeue_job([atom()]) :: {:ok, job()} | {:error, :empty} | {:error, term()}
```

Claims the next runnable job of one of `kinds`, advancing its attempt count.

Filtered by kind so a worker never claims work it cannot run: a claimed job
of another kind would be a job destroyed.

# `document_count`

```elixir
@callback document_count(String.t()) :: non_neg_integer()
```

How many documents are stored at `prefix` or beneath it.

Membership is exact-URI-or-descendant, the same rule `find/4` and
`grep_content/3` use, so a count of a subtree cannot include a sibling whose
name merely starts the same way. Directories are not counted.

# `enqueue_job`

```elixir
@callback enqueue_job(atom(), map()) :: {:ok, integer()} | {:error, term()}
```

Records durable background work.

# `fail_job`

```elixir
@callback fail_job(integer(), String.t() | nil) :: :ok | {:error, term()}
```

Records a job failure, rescheduling it with backoff while attempts remain and
marking it failed once they are exhausted.

`reason` is the worker's classified failure, persisted so a failed job can be
explained later without a log line. It is a classification -- never content,
a prompt, or a credential.

# `find_paths`

```elixir
@callback find_paths(String.t(), String.t() | nil, pos_integer()) ::
  {:ok, [path_match()]} | {:error, term()}
```

Path-discovery matches for `query` within `scope_uri` or beneath it.

`query` is a validated literal substring (1 through 256 characters).
`scope_uri` is a validated exact URI or `nil` for the whole tree; scope
membership is exact-URI-or-descendant, so `scope` never matches a sibling
such as `scope-old`. Matching is case-insensitive over the URI path
(excluding the `viking://` scheme), `%`, `_`, and `\` match literally,
results are ordered by URI, and at most `limit` entries are returned
without document content.

# `get_node`

```elixir
@callback get_node(String.t()) :: {:ok, tree_node() | nil} | {:error, term()}
```

The node at `uri`, or `{:ok, nil}` when nothing is stored there.

# `get_nodes`

```elixir
@callback get_nodes([String.t()]) :: {:ok, [tree_node() | nil]} | {:error, term()}
```

The nodes at these URIs in one read, in the same order as the input (missing ones as nil).

# `get_session`

```elixir
@callback get_session(String.t()) :: {:ok, [message()]} | {:error, term()}
```

Every message of a session, in order.

# `grep_content`

```elixir
@callback grep_content(String.t(), String.t() | nil, pos_integer()) ::
  {:ok, [line_match()]} | {:error, term()}
```

Line matches for `query` in full document content (L2) within `scope_uri`
or beneath it.

`query` is a validated literal substring (1 through 256 characters).
`scope_uri` is a validated exact URI or `nil` for the whole tree; scope
membership is exact-URI-or-descendant. Matching is case-insensitive over
L2 content only (abstracts and overviews never match), `%`, `_`, `\`,
and regular-expression metacharacters match literally, results are ordered
by URI then one-based line number, each excerpt contains the match and is
at most 280 characters, and at most `limit` entries are returned.

# `healthy?`

```elixir
@callback healthy?() :: boolean()
```

Whether the durable state can be reached at all.

# `list_all_documents`

```elixir
@callback list_all_documents(String.t(), pos_integer(), non_neg_integer(), map()) ::
  {:ok, {[map()], non_neg_integer()}} | {:error, term()}
```

Recursive document URIs under `scope` in deterministic order, paged.

Returns `{rows, total}` where rows carry `uri`, `enabled`, and `group_tag`
without blobs. `filter` supports `:substring`, `:include_disabled`, and
`:group`. `limit`/`offset` bound the page; `total` is the filtered count.

# `list_children`

```elixir
@callback list_children(String.t()) :: {:ok, [String.t()]} | {:error, term()}
```

Names of `uri`'s direct children, sorted. `{:error, :not_found}` when the URI holds no directory.

# `list_session_ids`

```elixir
@callback list_session_ids() :: {:ok, [String.t()]} | {:error, term()}
```

Ids of every session, ordered, so an export can enumerate them.

# `list_skill_roots`

```elixir
@callback list_skill_roots(pos_integer(), non_neg_integer(), map()) ::
  {:ok, {[map()], non_neg_integer()}} | {:error, term()}
```

Installed skill roots in URI order, paged. Each entry carries `name`,
`owner`, `uri`, `files`, `enabled`, and `group_tag`. `filter` supports
`:substring`, `:owner`, `:include_disabled`, and `:group`.

# `load_memory_vectors`

```elixir
@callback load_memory_vectors([String.t()]) ::
  {:ok, %{required(String.t()) => binary()}} | {:error, term()}
```

The stored embedding vectors of these URIs, keyed by URI as float32 blobs.

Vectors come back from the active vector dimension only: a URI not yet
embedded, or embedded under another dimensionality, is simply absent from
the answer rather than an error. The read is storage-only -- it never waits
on inference and never triggers model work -- because it exists so the term
recall can rank from what is already on disk instead of re-embedding the
candidate values it was handed.

# `mark_memories_surfaced`

```elixir
@callback mark_memories_surfaced([integer()]) :: :ok | {:error, term()}
```

Records that the assertions with these ids were surfaced by a recall.

# `memory_conflict_pairs`

```elixir
@callback memory_conflict_pairs(String.t()) ::
  {:ok, [memory_conflict()]} | {:error, term()}
```

Pairs of active memories beneath `prefix` whose values are similar enough to
be possible conflicts. Reuses stored vectors and runs no inference.
`{:error, :embeddings_unavailable}` when conflicts cannot be evaluated.

# `memory_recorded?`

```elixir
@callback memory_recorded?(String.t()) :: {:ok, boolean()} | {:error, term()}
```

Whether any assertion is recorded at `uri`, superseded and candidate ones included.

# `memory_stats`

```elixir
@callback memory_stats() :: {:ok, map()} | {:error, term()}
```

Counts of memories per type and per status, the number waiting for review, and
the number never surfaced, computed by the store without materialising every
memory.

# `promote_memory`

```elixir
@callback promote_memory(String.t()) :: {:ok, :promoted} | {:error, term()}
```

Promotes the latest candidate at `uri` to the active assertion, superseding
any prior active one, and carries the promoted value through the read-through
document at `uri`. `{:error, :no_candidate}` when none is waiting.

# `put_commit`

```elixir
@callback put_commit(String.t(), String.t(), String.t(), String.t()) ::
  {:ok, :ok} | {:error, term()}
```

Writes the committed document and its commit bookkeeping together.

# `put_document`

```elixir
@callback put_document(String.t(), String.t(), keyword()) :: {:ok, :ok} | {:error, term()}
```

Writes a document at `uri`, creating missing parent directories.

`:abstract` and `:overview` in `opts` are the caller's own layers; `nil` for
either keeps whatever is already stored, so a re-write does not discard a
layer it did not supply.

`:jobs` in `opts` is the required background work for the write, as a list
of `{kind, payload}` pairs. A provider persists the document and all jobs
as one storage outcome: success means both are durable, and a failure to
persist or enqueue any job leaves prior document, cache, and queue state
unchanged.

# `put_embedding_result`

```elixir
@callback put_embedding_result(integer(), String.t(), binary()) ::
  {:ok, :stored | :discarded} | {:error, term()}
```

Stores a generated embedding for a job, unless its node has since been removed.

Because the index is derived, an embedding for a URI the store no longer holds
is simply not wanted: the job is completed either way, and the result is
reported `:discarded`. A write at the URI meanwhile retires the previous
vector, so a re-write is never ranked on content it no longer has.

# `put_layer_result`

```elixir
@callback put_layer_result(integer(), String.t(), layer(), String.t()) ::
  {:ok, :stored | :discarded} | {:error, term()}
```

Stores a generated summary for a job, unless its node has since been removed.

# `put_memory`

```elixir
@callback put_memory(String.t(), String.t(), float(), String.t() | nil, keyword()) ::
  {:ok, :ok} | {:error, term()}
```

Writes a memory's document, its assertion, and its required background work
together, so a URI never holds a document no assertion backs, and a memory
reported as recorded never lacks the job that makes it searchable.

`opts` carries `:importance` (0.0..1.0, defaults when absent), `:status`
(`:active` by default, or `:candidate` for a record awaiting promotion), and
`:jobs` -- the required background work for the write, as a list of
`{kind, payload}` pairs, as `put_document/3` takes. An active re-record of the
value the URI already asserts is a no-op: the URI keeps its one assertion. A
candidate recorded at a URI that already holds a document does not overwrite
that document: the active belief is what the document is read through, and
`promote_memory/1` is what repairs it on promotion. A candidate at a URI
holding no document still gets one, so the URI is a document of the tree
like any other.

Implementations SHOULD accept the options as a trailing keyword with a
default, so callers that pass none keep working unchanged.

# `queue_detail`

```elixir
@callback queue_detail(pos_integer()) ::
  {:ok, %{oldest_pending_ms: non_neg_integer() | nil, failed: [map()]}}
  | {:error, term()}
```

How much work is outstanding and what went wrong.

Extends `queue_stats/0` with the age of the oldest pending job and a bounded
list of failed jobs carrying the reason, so an operator can tell a queue that
is backing up from one whose work is failing.

`oldest_pending_ms` is the age of the longest-waiting runnable job and is
`nil` when nothing is pending. `failed` is newest-first and bounded. Each
entry's `last_error` is a classified reason -- never content, a prompt, or a
credential.

# `queue_stats`

```elixir
@callback queue_stats() :: {:ok, map()} | {:error, term()}
```

Job counts by status.

# `recall_memories`

```elixir
@callback recall_memories(
  String.t(),
  String.t() | nil,
  [:active | :superseded | :candidate],
  keyword()
) ::
  {:ok, [memory_row()]} | {:error, term()}
```

Assertions at `prefix` or beneath it, most confident first.

`opts` bounds the read the way `search/2` bounds its legs:

  - `:top_k` - the bound this read materialises, 1..500 (default 500).
    Out of range is `{:error, {:invalid_limit, value}}` rather than a
    silently substituted bound. Every returned row reports whether the
    bound was reached as `truncated: true | false`, so a caller can tell
    "there is nothing more" from "there is more than I asked for".
  - `:source` - restricts to assertions recorded from that provenance.

Implementations SHOULD accept `opts` as a fourth argument with a default,
so existing three-argument calls keep working.

# `reclaim_vector_index`

```elixir
@callback reclaim_vector_index() :: :ok | {:error, term()}
```

Deletes every index row that names a URI the node store no longer holds.

The reclamation half of the drift contract: removal makes the URI
unsearchable immediately, and this reclaims the row afterwards. Rows naming
live URIs are left alone, and a store with nothing to reclaim is a no-op.

# `reject_memory_candidate`

```elixir
@callback reject_memory_candidate(String.t()) :: :ok | {:error, term()}
```

Removes candidate assertions at `uri` without a trace. Active and superseded
history is untouched. `{:error, :no_candidate}` when none is waiting.

# `remove_subtree`

```elixir
@callback remove_subtree(String.t()) :: :ok | {:error, :not_found} | {:error, :is_root}
```

Removes `uri` and its whole subtree from every store keyed by URI.

# `replace_skill`

```elixir
@callback replace_skill(String.t(), [skill_file()]) ::
  {:ok, %{replaced: boolean(), files: pos_integer()}} | {:error, term()}
```

Replaces the whole subtree at `uri` with `files`, in one atomic step.

Each `file` is stored at `uri` followed by its `path`, with the directories
above it created, and the usual asynchronous work is enqueued for it, so a
file written this way is indistinguishable from one written with
`put_document/3`.

What was at `uri` before is removed, from every store keyed by URI, in the same
step: a file the new set does not mention must not survive it, and a failure
part way through must leave the previous contents exactly as they were.

Answers whether anything was there to replace, which is what tells a caller
that an import overwrote a skill rather than adding one.

# `reset_running_jobs`

```elixir
@callback reset_running_jobs() :: :ok | {:error, term()}
```

Returns jobs left running by a previous run to pending.

# `restore_session`

```elixir
@callback restore_session(String.t(), [message()]) ::
  {:ok, :imported | :skipped} | {:error, term()}
```

Restores a session with a specific id and its messages in order.

Creates the session when absent and appends the messages. When the session
already holds exactly these messages (compared as `{role, content}` pairs in
order) it reports `{:ok, :skipped}` and writes nothing. When it holds
different messages it reports `{:error, {:session_conflict, id}}` and leaves
the stored session untouched, so an import never silently overwrites a live
conversation.

# `search_keyword`

```elixir
@callback search_keyword(String.t(), String.t() | nil, pos_integer(), [atom()] | nil) ::
  {:ok, [tree_node()]} | {:error, term()}
```

Case-insensitive substring search over stored documents, optionally within
`scope_prefix`.

At most `limit` results, ordered deterministically by URI. The bound is
applied by the provider rather than by the caller, so a query matching more
documents than were asked for does not materialize all of them.

`fields` is the set of per-result fields the caller asked for, or `nil` for
the default set (URI, content, abstract, overview). A projection is applied
by the statement that reads the rows rather than after they are materialized,
so a caller that did not ask for content does not pay to read it. `:uri` is
returned regardless of the selection, because a result without its URI
cannot be used to fetch the document it names. The internal columns a
provider still needs (`enabled`, `group_tag`) are read as well and reported
alongside, so a filtering caller can drop them itself.

# `search_vector`

```elixir
@callback search_vector(binary(), pos_integer(), String.t() | nil, [atom()] | nil) ::
  {:ok, [hit()]} | {:error, term()}
```

Nearest neighbours of a query embedding, optionally within `scope_prefix`.

The index is derived data: a row may survive for URI the node store no longer
holds, and the search is what keeps that inert. A hit is only returned when
the store can still hold the URI it names, so a removed URI is never surfaced
while a row for it survives. Candidates that cannot be hydrated do not count
towards `top_k`. Out of range `top_k` values and a query of the wrong
dimensionality are refused rather than answered approximately.

`fields` is the set of per-result fields the caller asked for, or `nil` for
the default set (URI, content, abstract, overview, score). A projection is
applied by the statement that reads the rows rather than after they are
materialized, so a caller that did not ask for content does not pay to read
it. `:uri` is returned regardless of the selection.

# `set_node_enabled`

```elixir
@callback set_node_enabled(String.t(), boolean()) :: :ok | {:error, term()}
```

Sets the enabled state for `uri` and its whole subtree.

Disabled is blocked-from-use, not deleted: rows stay readable and editable
but are excluded from search and default listings.

# `set_node_group`

```elixir
@callback set_node_group(String.t(), String.t()) :: :ok | {:error, term()}
```

Sets the operator group tag for `uri` and its whole subtree. Empty clears.
Tags are names only, never content.

# `stats`

```elixir
@callback stats() ::
  {:ok,
   %{
     documents: non_neg_integer(),
     directories: non_neg_integer(),
     by_top_subtree: %{optional(String.t()) => non_neg_integer()},
     db_bytes: non_neg_integer() | nil,
     wal_bytes: non_neg_integer() | nil
   }}
  | {:error, term()}
```

The store's composition and footprint, read-only.

`documents` and `directories` count every node of each kind;
`by_top_subtree` counts documents filed at or below each top-level subtree
(`resources`, `user`, `peers`, ...). `db_bytes` and `wal_bytes` are the sizes
of the database file and its write-ahead log, which is a fact about one
implementation: a provider with no single database file reports `nil` rather
than inventing one.

This counts; it never walks the tree, so its cost does not grow with depth.

# `vector_index_reclaimable`

```elixir
@callback vector_index_reclaimable() :: {:ok, non_neg_integer()} | {:error, term()}
```

How many index rows name a URI the node store no longer holds.

Drift, reported without scanning content. Reclaimable rows are already
invisible to search -- the read path joins them away -- so this counts
occupancy rather than correctness, and it is what tells an operator that
reclamation is falling behind. Reported without touching anything.

# `vector_index_stats`

```elixir
@callback vector_index_stats() ::
  {:ok,
   %{
     available: boolean(),
     vectors: non_neg_integer() | nil,
     documents: non_neg_integer()
   }}
  | {:error, term()}
```

Coverage of the vector index relative to what is stored.

`available` is whether the index can be queried at all; when it is not,
`vectors` is `nil` rather than zero, because "not available" and "holds
nothing" are different facts an operator needs to tell apart. `vectors`
counts indexed rows and `documents` counts the documents they describe, so a
store can report how much of its content is vector-searchable. `reclaimable`
counts rows whose URI the store no longer holds -- drift that is harmless to
search results but still occupies the index, and the number that says
reclamation is falling behind rather than merely that it has not run yet.

---

*Consult [api-reference.md](api-reference.md) for complete listing*