Providers

Providers

The harness keeps model calls behind a provider-neutral trait (tinyinference::model::ChatModel). A provider adapter translates a ModelRequest into the provider's wire format and normalizes the response, usage, streaming chunks, and errors back into TinyAgents types, so the rest of the harness never sees provider-specific shapes.

Model providers live in tinyinference, a vendored crate that tinyagents-harness depends on (vendor/tinyinference/crates/tinyinference). Import paths below are tinyinference::..., not tinyagents_harness::....

Offline by default

The default build is offline at run time: no provider call happens unless you make one. Two chat models are available out of the box — tinyinference::providers::MockModel (deterministic, network-free, for tests and examples) and tinyinference::providers::openai::OpenAiModel (the hosted OpenAI adapter, always compiled in since tinyagents-harness depends on it unconditionally).

OpenAiModel is also the client used to reach every other provider listed below, because they all speak the OpenAI Chat Completions wire protocol. OpenAiModel is the one concrete HTTP-backed provider implementation; the other providers exist as configuration presets (base URL, default model, API key environment variable) layered on top of it, not as separate native adapters.

Provider kinds and specs

ProviderKind (in tinyinference::providers) enumerates the provider families a factory can select between, and ProviderSpec is the portable configuration bound to a name.

Kind	provider id	Default model	Base URL	API key env	Key required
OpenAi	openai	gpt-4.1-mini	https://api.openai.com/v1	OPENAI_API_KEY	yes
Anthropic	anthropic	claude-3-5-sonnet-latest	https://api.anthropic.com/v1	ANTHROPIC_API_KEY	yes
Ollama	ollama	llama3.2	http://localhost:11434/v1	(none)	no
LmStudio	lmstudio	(unset)	http://localhost:1234/v1	(none)	no
LlamaCpp	llama_cpp	(unset)	http://localhost:8080/v1	(none)	no
Vllm	vllm	(unset)	http://localhost:8000/v1	(none)	no
DeepSeek	deepseek	deepseek-chat	https://api.deepseek.com/v1	DEEPSEEK_API_KEY	yes
Groq	groq	llama-3.3-70b-versatile	https://api.groq.com/openai/v1	GROQ_API_KEY	yes
Xai	xai	grok-2-latest	https://api.x.ai/v1	XAI_API_KEY	yes
OpenRouter	openrouter	openai/gpt-4o-mini	https://openrouter.ai/api/v1	OPENROUTER_API_KEY	yes
Together	together	meta-llama/Llama-3.3-70B-Instruct-Turbo	https://api.together.xyz/v1	TOGETHER_API_KEY	yes
Mistral	mistral	mistral-small-latest	https://api.mistral.ai/v1	MISTRAL_API_KEY	yes
Compatible	compatible	(unset)	(unset)	(none)	yes

ProviderSpec::for_kind(kind) returns the default spec above. Compatible is for an endpoint that implements the OpenAI Chat Completions protocol but does not need a named preset — supply your own base URL, model, and key env.

use tinyinference::providers::{ProviderKind, ProviderSpec};

let spec = ProviderSpec::for_kind(ProviderKind::Groq)
    .with_model("llama-3.3-70b-versatile");

Each spec records the provider kind, the provider id (used in normalized errors), the default model id, the base URL, the API-key environment variable when one is required, and whether a real API key is required. Builder methods override any of these: with_model, with_base_url, with_provider, and with_api_key_env.

Quick start: OpenAI

1. Configure credentials

export OPENAI_API_KEY=sk-...
export OPENAI_MODEL=gpt-4.1-mini        # optional, defaults to gpt-4.1-mini
export OPENAI_BASE_URL=https://api.openai.com/v1   # optional override

Or keep them in a .env file at the repo root — the examples load it via dotenvy. See .env.example at the repo root for the expected keys.

2. Run an example

cargo run -p tinyagents-integration-tests --example openai_chat

OpenAI-backed examples (openai_chat, openai_tools, openai_structured, openai_graph_agent, openai_self_blueprint) require a valid OPENAI_API_KEY at run time.

3. Build a model in code

use tinyagents_harness::Result;
use tinyagents_harness::runtime::AgentHarness;
use tinyinference::message::Message;
use tinyinference::providers::openai::OpenAiModel;

let model = OpenAiModel::from_env()?;

let mut harness: AgentHarness<()> = AgentHarness::new();
harness
    .register_model("openai", std::sync::Arc::new(model))
    .set_default_model("openai");

let run = harness
    .invoke_default(&(), vec![Message::user("hello")])
    .await?;

OpenAiModel::from_env() reads OPENAI_API_KEY (required), OPENAI_MODEL (optional), and OPENAI_BASE_URL (optional). Use OpenAiModel::new(api_key) when you want to pass the key yourself. AgentHarness::register_model and set_default_model bind the model under a runtime name; invoke_default runs the agent loop with the harness's default model. See Harness for the full runtime API.

4. Discover available models

OpenAiModel::list_models() calls the provider's GET {base_url}/models endpoint and returns Vec<ModelListing>. It works for every OpenAI-compatible endpoint, so it doubles as runtime model discovery for local/self-hosted providers (Ollama, LM Studio, llama.cpp, vLLM). Any returned id can be passed to .with_model(..).

let provider = OpenAiModel::from_env()?;
for listing in provider.list_models().await? {
    println!("{}", listing.id);
}

Reaching other providers through the compatible adapter

Because every provider in the table speaks the OpenAI Chat Completions protocol, OpenAiModel ships named constructors that preset the provider id, base URL, and default model. Each hosted-provider constructor takes the API key:

use tinyinference::providers::openai::OpenAiModel;

let anthropic  = OpenAiModel::anthropic(std::env::var("ANTHROPIC_API_KEY")?);
let deepseek   = OpenAiModel::deepseek(std::env::var("DEEPSEEK_API_KEY")?);
let groq       = OpenAiModel::groq(std::env::var("GROQ_API_KEY")?);
let xai        = OpenAiModel::xai(std::env::var("XAI_API_KEY")?);
let openrouter = OpenAiModel::openrouter(std::env::var("OPENROUTER_API_KEY")?);
let together   = OpenAiModel::together(std::env::var("TOGETHER_API_KEY")?);
let mistral    = OpenAiModel::mistral(std::env::var("MISTRAL_API_KEY")?);
let ollama     = OpenAiModel::ollama();   // local, no key

Local runtimes have their own constructors that take a base URL and model rather than an API key: OpenAiModel::ollama_at(base_url, model), OpenAiModel::llama_cpp(base_url, model), OpenAiModel::vllm(..).

Override the default model with .with_model(...):

let groq = OpenAiModel::groq(std::env::var("GROQ_API_KEY")?)
    .with_model("llama-3.1-8b-instant");

From a ProviderSpec

To drive the choice from configuration, build a spec and construct the model from it:

use tinyinference::providers::{ProviderKind, ProviderSpec};
use tinyinference::providers::openai::OpenAiModel;

let spec = ProviderSpec::for_kind(ProviderKind::Mistral)
    .with_model("mistral-small-latest");

let model = OpenAiModel::from_spec_env(spec)?;

from_spec_env reads the spec's configured environment variable. Use from_spec(spec, api_key) when credentials come from another secret source, and compatible(base_url, model) / compatible_provider(..) for a fully custom OpenAI-compatible endpoint.

Ollama (local)

Start Ollama with its OpenAI-compatible endpoint at http://localhost:11434/v1, then:

use tinyinference::providers::openai::OpenAiModel;

let model = OpenAiModel::ollama().with_model("llama3.2");

Ollama ignores the API key, so the spec marks requires_api_key = false.

Error and stream normalization

Providers report failures through the crate's error type, carrying the provider id, the model id when known, the HTTP status when available, a human-readable message, and a retryability hint.

Streaming providers emit normalized stream items: text deltas, usage updates, tool-call chunks, or a final message, with provider-side stream errors surfaced through the same normalized shape as non-streaming failures. See Harness for the streaming and sub-agent APIs built on top of this.

Reasoning/thinking output is normalized separately from visible assistant text. OpenAI-compatible streams that expose delta.reasoning_content or delta.reasoning become a reasoning field on the message delta; delta.content remains the visible text channel.

Provider selection from a model string

ProviderKind::infer(...) supports explicit prefixes such as openai:gpt-4.1-mini, anthropic:claude-..., and ollama:llama3.2, plus conservative bare-model inference for common families (gpt-/o1/o3/o4 → OpenAI, claude → Anthropic, deepseek → DeepSeek, grok → xAI, mistral/mixtral → Mistral).

Prefer explicit ProviderSpec values in application code. Inference is convenient for configuration files, examples, and .rag blueprints where the model string is the only user input.

Capability profiles and the model registry

Provider constructors produce executable models; a separate resolution layer in tinyagents_harness::model_registry decides which registered model a run actually uses. ModelRegistry binds named models for a State, and ResolvedModelBinding is the durable record of which registry name was selected for a call. See Harness and Registry for how the registry and model selection fit together.

See also

• Harness — the provider-neutral model traits these adapters implement.
• Development — how to build and test, including live provider tests.