Sub-100ms Deterministic Decision Routing Engine (System One)

[zsogitbe]

Hi everyone,

I wanted to share a new open-source extension we built to solve the latency and VRAM bottlenecks of multi-agent routing.

We just published TIEG.SystemOne.LLamaSharp (available on [NuGet] and [GitHub]).

Instead of generating text autoregressively to classify inputs or route tasks (which can take several seconds and consume massive VRAM for the KV cache), this engine implements a "System One" fast-path. It executes a single parallel batch pass, extracts the raw logits for the target options, and applies a calibrated Softmax distribution.

The result is deterministic classification in under 100ms.

Key Features

• Threshold-Based Fallbacks: It calculates Margin and DominanceRatio metrics to mathematically evaluate the model's internal hesitation.
• VRAM Handoff: If the fast model hesitates, the engine explicitly unloads it from VRAM so you can load a heavy generative reasoning model (like Qwen 7B or Llama 8B) in its place without hitting OOM exceptions.
• Calibrated for Plumb-4B: While it works with standard models (by setting calibrationTemperature: 1.0f), it is optimized for Plumb-4B, our specialized routing model.

Quick Example

using var engine = new DecisionEngine("path/to/plumb-4b.gguf");

var result = engine.Decide(
    systemContext: "Determine the risk level of the transaction.",
    inputData: "Transaction: $12 Netflix subscription.",
    options: new[] { "High Risk", "Medium Risk", "Low Risk" }
);

if (result.CanAutomate(minMargin: 0.20f))
{
    Console.WriteLine($"Automated: {result.SelectedOption} (Margin: {result.Margin:P1})");
}
else
{
    engine.Unload(); // Instantly frees VRAM
    // ... route to your heavy LLamaSharp fallback model
}

We included a complete, runnable tiered-routing demo in the repository here: [TieredRoutingDemo].

We would love for the community to check it out, test it in your multi-agent setups, and share any feedback!