AgentDb.ML.ModelManager (agent_db v0.1.0)

Copy Markdown View Source

Manages embedding and LLM models for vector search and summarization.

Handles model download, caching, lazy loading, and inference.

Summary

Types

model_ref()
state()

Functions

backend()

The resolved ML backend (:exla or :emlx).

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

embed(texts)

Generates embeddings for the given texts.

model_status()

Returns the status of both models.

start_link(opts \\ [])
summarize(prompt, opts \\ [])

Generates a summary for the given prompt.

Types

model_ref()

@type model_ref() :: %{
  tokenizer: Bumblebee.Tokenizer.t(),
  model: term(),
  config: map(),
  serving: module(),
  chat_template: String.t()
}

state()

@type state() :: %AgentDb.ML.ModelManager.State{
  config: map(),
  embedding_model: model_ref() | nil,
  in_flight: term(),
  inference_refs: term(),
  last_latency_ms: term(),
  last_load_ms: term(),
  llm_model: model_ref() | nil,
  loading: map(),
  loading_ref: term()
}

Functions

backend()

@spec backend() :: :exla | :emlx

The resolved ML backend (:exla or :emlx).

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

embed(texts)

@spec embed([String.t()]) :: {:ok, [Nx.Tensor.t()]} | {:error, term()}

Generates embeddings for the given texts.

Returns {:error, :model_loading} when the model is not ready and has not become ready within the configured load grace (Config.model_load_grace_ms/0). That is distinct from a load that failed, and the call is safe to repeat.

model_status()

@spec model_status() :: map()

Returns the status of both models.

start_link(opts \\ [])

@spec start_link(keyword()) :: GenServer.on_start()

summarize(prompt, opts \\ [])

@spec summarize(String.t(), keyword()) :: {:ok, String.t()} | {:error, term()}

Generates a summary for the given prompt.

Returns {:error, :model_loading} when the model is not ready and has not become ready within the configured load grace (Config.model_load_grace_ms/0).