Troubleshooting

Symptom-first diagnostics. Find the symptom, run the diagnostic command, compare against the expected healthy output, and apply the resolution. For alerting and dashboards see MONITORING.md; for the full API surface see API-REFERENCE.md.

Symptom Index

1. Daemon won't start

Diagnose:

mix agent_db.doctor --json

Healthy:

{"db":true,"status":"ok","provider":"local","queue":{"done":150,"failed":0},"pubsub":true}

db false (the task exits non-zero with Database unreachable) — the store directory is missing or not writable. Resolve: point AGENT_DB_DATA_DIR at a writable directory and re-run. status degraded or a models-related warning does not stop the daemon — it degrades to no-model behavior, so treat those separately (symptoms 2 and 3).

2. Model fails to load

Log signature: component: "ModelManager", reason: "download_failed" or reason: "model_load_failed".

Diagnose:

AgentDb.model_status()
# healthy: %{embedding: %{state: :ready, ...}, llm: %{state: :ready, ...}}
# unhealthy: %{embedding: %{state: :loading, ...}} (stuck) or %{state: :failed}

Resolve, in order:

1. Truncated cache — the most common cause: an interrupted download left a partial file. Delete the truncated cache file for the model and let it re-download (SETUP.md documents the model directory layout and pre-placement).
2. Backend mismatch on Apple Silicon — the EXLA backend must match the chip. Re-check the backend selection from SETUP.md.
3. Disk space — a full volume fails mid-download; free space and retry.
4. Remote provider unreachable — model_status/0 reports the provider's health state; :unreachable means the endpoint did not answer, check the base URL and network. :unauthorized means the token was refused.

3. Search returns model_loading

The embedding model is still loading; vector and hybrid search wait for it (:keyword mode works without a model).

Diagnose: AgentDb.model_status() — any role in :loading explains the error. Logs show reason: "model_loading".

Resolve: wait for the load (check last_load_ms after it succeeds). If a role stays :loading for minutes, treat it as symptom 2. Keyword search and find/grep stay available while a model loads — no action needed if the caller can retry.

4. Queue grows without draining

Diagnose:

AgentDb.queue_stats()
# healthy: counts per status present, e.g. %{pending: 0, running: 0, done: 150, failed: 0}
# (a status with no rows is absent from the map; a queue that cannot be read reports %{})

AgentDb.queue_detail(20)
# watch oldest_pending_ms (nil when nothing is pending) and the failed list
# with classified last_error reasons

mix agent_db.doctor   # reports db, status, provider, queue, and pubsub

Resolve, in order:

1. Model loading — embedding/summarization jobs cannot finish without a model; resolve symptom 2 first.
2. Throughput too low — raise the worker count (job_workers in the application config) and restart.
3. Failed jobs — read the classified last_error per failed job in queue_detail/1 (e.g. inference_timeout — check symptom 5) and clear or retry as appropriate.

5. High inference latency

Diagnose: model_status/0 reports last_latency_ms per role; queue_detail/1 reports pending age; telemetry carries [:agent_db, :model, :stop] durations by role.

Resolve, in order:

1. A role in :loading blocks inference — resolve symptom 2.
2. A long queue (symptom 4) delays every job downstream.
3. A remote provider adds network round-trips — model_status/0 shows the provider's health; inference_timeout in logs means the provider exceeded the request deadline. Switch to the local provider or raise the timeout.
4. Sustained load: check queue_stats running depth; if workers are all busy, raise job_workers.

6. Authentication rejected

The daemon returns 401 when auth is on and the token is missing or wrong. Logs carry reason: "unauthorized".

Diagnose: confirm whether auth is enabled and which token you sent:

# from inside the daemon host: the listener and auth settings
mix agent_db.doctor

Resolve: with AGENT_DB_HTTP_AUTH=true, the client must send Authorization: Bearer <token> with a token from AGENT_DB_HTTP_AUTH_TOKENS. The default (auth off) binds loopback only, so a non-loopback client gets refused regardless of token.

7. WSL2 daemon unreachable from Windows

Run the daemon and the installer inside WSL2, and point Windows editors at http://localhost:6060/mcp.

Diagnose:

# from PowerShell
wsl bash tools/install.sh --check
wsl bash -c "curl localhost:6060/api/v1/health"

Resolve, in order:

1. Installer refuses to run natively — tools/install.ps1 refuses native execution on purpose: native Windows has no supported EXLA path, so a native install would look done while inference cannot work. Run it inside WSL.
2. Windows firewall — allow the port (6060 by default) or reach the daemon through WSL's forwarded localhost.
3. Daemon bound inside WSL only — start the daemon inside WSL2 and verify from Windows with the curl check above.

8. Admin console shows errors

The console renders the same data as the health functions.

Diagnose:

AgentDb.recent_errors(20)
# bounded list of recent operational errors with their classified codes

Match the code against the API-REFERENCE error catalog and jump to the matching symptom above. Console panels that stay empty with the daemon up are usually the listener question (symptom 6/7).

9. Export/import fails

Diagnose: the failure reason from AgentDb.export_data/2 / AgentDb.import_data/2, classified with AgentDb.export_data_error_message/1; limits are AgentDb.data_transfer_limits/0.

Resolve, in order:

1. Over a limit (entry count or size) — the bounds are in data_transfer_limits/0; split the transfer or narrow the scope:.
2. Permission or disk — the destination path must be writable with free space.
3. Corrupt archive — a partially written export; re-export.

Reading the Logs

All operational failures log structured fields (never content, prompts, URIs, or credentials):

Pattern: an outcome: :error line's reason names the failure mode and maps 1:1 to the error catalog, so reason: "model_load_failed" is symptom 2 and reason: "inference_timeout" is symptom 5.