Monitoring

Production monitoring and observability guide for AgentDb. Covers the /admin console, telemetry events, structured logs, health checks, error taxonomy, and alerting recommendations.

/admin Console (Primary Operational Interface)

The /admin console is a real-time LiveView dashboard at http://localhost:6060/admin (or your configured host/port). It subscribes to context changes via PubSub and updates without manual reload.

Console Sections

Using the Console

Quick health check: Open /admin and verify:

• Storage: DB size > 0, document count matches expectation
• Models: All roles show :ready (not :loading or :failed)
• Queue: pending count is low, failed = 0, oldest pending age is recent
• Index: Vector index available with row count ≈ document count
• Recent Errors: Empty or only expected transient errors

Diagnose queue backpressure:

1. Check Queue section: high pending count + growing oldest_pending_ms
2. Click failed jobs to see classified reasons (e.g., :model_load_failed, :inference_timeout)
3. Check Models section: any role stuck in :loading?
4. Check Runtime: BEAM memory/process count normal?

Distinguish cache growth from content growth:

• Cache section shows per-table entry counts and bytes
• Storage section shows document counts per subtree
• If BEAM memory rises but Storage document counts are flat → cache growth
• If both rise proportionally → content growth

Telemetry Events

AgentDb emits :telemetry events with bounded dimensions only — never URIs, content, prompts, users, or credentials as label values.

Event Catalog

Retrieval Pipeline Stages (Child Spans)

When OpenTelemetry is configured, each retrieval stage creates a child span:

Bounded Dimensions Rule: All events/spans carry only stage, mode, outcome, duration. Never URIs, content, prompts, users, credentials.

Consuming Telemetry (Prometheus/Grafana Example)

# In your application startup
:telemetry.attach_many("my-metrics", [
  [:agent_db, :operation, :stop],
  [:agent_db, :job, :stop],
  [:agent_db, :model, :stop]
], &MyMetrics.handle_event/4, %{})

defmodule MyMetrics do
  def handle_event([:agent_db, :operation, :stop], %{duration_ms: duration}, %{operation: op, outcome: outcome}, _config) do
    Prometheus.Histogram.observe(:agent_db_operation_duration, duration, labels: [operation: op, outcome: outcome])
  end

  def handle_event([:agent_db, :job, :stop], %{queue_wait_ms: wait, execution_ms: exec}, %{kind: kind, outcome: outcome}, _config) do
    Prometheus.Histogram.observe(:agent_db_job_queue_wait, wait, labels: [kind: kind, outcome: outcome])
    Prometheus.Histogram.observe(:agent_db_job_execution, exec, labels: [kind: kind, outcome: outcome])
  end

  def handle_event([:agent_db, :model, :stop], %{duration_ms: duration}, %{role: role, outcome: outcome}, _config) do
    Prometheus.Histogram.observe(:agent_db_model_duration, duration, labels: [role: role, outcome: outcome])
  end
end

Dashboard Recommendations:

• Operation latency by operation + outcome (p50, p95, p99)
• Job queue wait vs execution by kind + outcome
• Model inference latency by role + outcome
• Error rate by operation + outcome

Sink Statistics Surface

The store keeps a bounded in-memory summary of everything it has measured, exposed through AgentDb.operation_stats/0 (and AgentDb.Observability.operation_stats/0):

• Counts per {family, dimension} and outcome, so each operation, job kind, and model role reports how many times it succeeded, failed, was deferred, was discarded, or errored.
• Latency percentiles (p50, p95, p99, min, max) over a fixed-length ring of duration samples, retained for successes as well as failures, so a slow-but-successful path is visible.
• Error rate per {family, dimension}: failures divided by total observations.
• Recent failures (AgentDb.recent_errors/1): a bounded newest-first ring of {family, operation, role, reason, duration_ms, at}, where reason is the classified code.

The rings are bounded: at most AgentDb.Observability.Sink.max_latency_samples/0 (100) duration samples per tracked operation, and at most AgentDb.Observability.Sink.max_tracked_operations/0 (64) distinct operations. When the operation bound is exceeded the oldest tracked operation -- counters, samples, and recent failures included -- is dropped, so a store that runs for months keeps a fixed diagnostic footprint. Recent failures are capped at AgentDb.Observability.Sink.max_errors/0 (50).

Structured Logs

All operational failures and background-job state transitions are logged as structured events via Logger with the agent_db metadata key.

Log Fields

Redaction Guarantees: Logs never contain document content, model prompts, authentication tokens, credentials, or unredacted secret-bearing URLs. URLs with credentials are redacted via Observability.redact_url/1.

Log Aggregation Example (JSON)

{
  "timestamp": "2026-01-15T10:30:45.123Z",
  "level": "error",
  "logger": "agent_db",
  "message": "agent_db",
  "metadata": {
    "component": "ModelManager",
    "operation": "embed",
    "kind": "embedding",
    "outcome": "error",
    "reason": "model_load_failed",
    "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
  }
}

Health Checks

Programmatic Health Checks

# Full health check (for orchestration liveness/readiness probes)
AgentDb.health_check()
# => %{status: "ok" | "degraded", checks: %{db: true, models: true}}

# Model status detail (includes loading state)
AgentDb.model_status()
# => %{
#      embedding: %{state: :ready, last_load_ms: 1200, last_latency_ms: 45, ...},
#      llm: %{state: :loading, ...}
#    }

# Queue depth and backlog (counts per status present — a status with no rows is absent)
AgentDb.queue_stats()
# => %{pending: 5, running: 2, done: 150, failed: 0}

# Detailed queue (oldest pending age + failed jobs)
AgentDb.queue_detail(20)
# => %{oldest_pending_ms: 45000, failed: [%{kind: :embed, uri: "...", attempts: 3, last_error: :model_load_failed, failed_at: ...}]}

CLI Health Check

# Human-readable
mix agent_db.doctor

# JSON for automation
mix agent_db.doctor --json
# => {"db":true,"status":"ok","provider":"local","queue":{"done":150,"failed":0},"pubsub":true}

Exit Codes: 0 = healthy/degraded, non-zero = database unreachable only.

Recommended Alert Thresholds

Error Taxonomy (Shared Across All Transports)

Every error response on every transport includes a code from this bounded taxonomy. Operators can alert on codes without parsing messages.

Error Code Categories

Transport Mapping

Alerting on Error Codes

Error codes ride in logs and transport responses. The telemetry metrics carry only the bounded dimensions (operation, kind, role, outcome) — no code label — so alert on outcomes from metrics and on codes from logs.

# Metric side: inference failing at the model layer (role is a real label)
increase(agent_db_model_total{outcome="error",role="embedding"}[5m]) > 0

# Metric side: failed background jobs accumulating
increase(agent_db_job_total{outcome="error"}[5m]) > 10

# Log side: the classified `reason` is the same code every transport returns
{app="agent_db", level="error"} | json | reason="model_load_failed"

Provider availability is a health state, not a metric. Poll AgentDb.model_status/0 — the same state the console's model panel shows — and alert when the provider reports :unreachable or :unauthorized.

Trace Correlation

All operations propagate W3C traceparent headers:

• HTTP: traceparent header
• WebSocket: Top-level traceparent field or inside opts.traceparent
• MCP: Inside opts.trace_context or HTTP header
• Background Jobs: Trace context retained from enqueueing operation

Use trace_id in logs/telemetry to correlate a request across transports and into background job execution.

Verification Commands

# Full health check (JSON)
mix agent_db.doctor --json

# Model status
elixir -e "Application.ensure_all_started(:agent_db); IO.inspect(AgentDb.model_status())"

# Queue stats
elixir -e "Application.ensure_all_started(:agent_db); IO.inspect(AgentDb.queue_stats())"

# Queue detail (failed jobs)
elixir -e "Application.ensure_all_started(:agent_db); IO.inspect(AgentDb.queue_detail(20))"

# Storage stats
elixir -e "Application.ensure_all_started(:agent_db); IO.inspect(AgentDb.storage_stats())"

# Cache stats
elixir -e "Application.ensure_all_started(:agent_db); IO.inspect(AgentDb.cache_stats())"

# Index coverage
elixir -e "Application.ensure_all_started(:agent_db); IO.inspect(AgentDb.index_coverage())"

# Recent errors
elixir -e "Application.ensure_all_started(:agent_db); IO.inspect(AgentDb.recent_errors(20))"