# `AgentDb.ML.BumblebeeLoader`
[🔗](https://github.com/manhvu/agent_db/blob/v0.1.0/lib/agent_db/ml/bumblebee_loader.ex#L1)

Loads models through Bumblebee and reports which serving runs them.

Isolates the Bumblebee API behind one module so the load path has a single
seam. `AgentDb.ML.ModelManager` reaches the model-loading library only through
this, which is what lets the load path be exercised without real weights.

`load_model/2` is a single call that returns the already-loaded model as
`{:ok, %{model: model, spec: spec}}`. It does not take the loaded result as
input: Bumblebee's repository normalizer accepts only `{:hf, id}` or
`{:local, dir}` and raises `ArgumentError` on anything else.

# `model_info`

```elixir
@type model_info() :: map()
```

# `repository`

```elixir
@type repository() :: {:hf, String.t()} | {:local, String.t()}
```

# `role`

```elixir
@type role() :: :embedding | :llm
```

# `backend_spec`

```elixir
@spec backend_spec(:cpu | :cuda | :rocm | :emlx | {module(), keyword()}) ::
  :none | {module(), keyword()}
```

Translates an AgentDb backend name into Bumblebee's `:backend` value.

Returns `:none` to mean "do not pass a `:backend` option", which is how
Bumblebee is told to use the default backend.

# `build_serving`

```elixir
@spec build_serving(role(), model_info(), term(), keyword()) ::
  {:ok, Nx.Serving.t()} | {:error, term()}
```

Builds the inference serving for a loaded model.

Bumblebee builds an `Nx.Serving` from the model info and the tokenizer, and
runs inference through that serving rather than through a tokenize/generate
pair on the serving module. Loading therefore produces the serving itself,
once, and `run/2` is the only inference entry point.

`opts` carries the role's compile options (`:batch_size`,
`:sequence_length`) and, for the LLM, the generation settings to configure
(`:max_new_tokens`, `:temperature`, `:top_p`).

# `load_model`

```elixir
@spec load_model(repository(), keyword(), module()) ::
  {:ok, term()} | {:error, term()}
```

Loads the model for a repository onto the backend named by `opts[:backend]`.

`opts[:backend]` is AgentDb's own vocabulary — `:cpu`, `:cuda` or `:rocm`,
the values AgentDb.Config.exla_backend/0 returns. Bumblebee wants a backend
*module* or `{module, opts}`, so it is translated here rather than passed
through: an untranslated `:cpu` is read as a module named `:cpu` and raises.

`:cpu` passes no `:backend` at all, which is Nx's default CPU backend.
`:cuda`/`:rocm` require an EXLA client configured under `config :exla,
clients`; without one, `EXLA.Client.fetch!/1` raises and the caller sees a
load failure rather than a silent fall back to CPU.

# `load_tokenizer`

```elixir
@spec load_tokenizer(repository(), module()) :: {:ok, term()} | {:error, term()}
```

Loads the tokenizer for a repository.

# `run`

```elixir
@spec run(Nx.Serving.t(), term()) :: {:ok, term()} | {:error, term()}
```

Runs a built serving over `input` and returns its result.

`Nx.Serving.run/2` answers with the serving's result directly rather than
wrapping it, so an error is recognised by its own shape and anything else is
a successful result. Inference goes through this one function so the
Bumblebee API is reached from a single place, and a loader without Bumblebee
installed can answer `build_serving/4` and `run/2` instead.

---

*Consult [api-reference.md](api-reference.md) for complete listing*