OpenTest

AI-augmented API testing that complements — not replaces — traditional test automation.

Core principle: LLM translates once, machines execute repeatedly.

Runbook (natural language)
        ↓  LLM reads runbook + API spec + .ftl templates
Flow YAML (reviewable, version-controlled)
        ↓  Orchestrator executes (deterministic, free, repeatable)
Report

Usage

OpenTest has two modes. Use them together or independently.

Mode 1: LLM-Driven (E2E)

Give your LLM a runbook and let it drive the entire process — read the spec, pick templates, call APIs, judge results.

# Point your LLM (Copilot, Claude Code, etc.) at the project with:
#   - A runbook:          runbooks/petstore-purchase.md
#   - The API spec:       specs/petstore.json
#   - Available templates: templates/*.ftl
#   - The test tool:      scripts/opentest.sh

# The LLM reads the runbook, decides which APIs to call,
# renders .ftl templates with the right data, executes via opentest.sh,
# and judges whether the results make sense.

# Debug log captures everything the LLM did:
cat runs/<run-id>/debug.log

When to use: Exploratory testing, first-time flow creation, investigating failures.

Cost: LLM tokens per run.

Mode 2: YAML Execution (Repeatable)

Once a flow YAML exists — whether LLM-generated or hand-written — run it without any LLM involvement.

# Validate
python3 scripts/validate.py flows/petstore-purchase.yaml

# Run
python3 scripts/orchestrator.py flows/petstore-purchase.yaml

# Resume async flows
python3 scripts/orchestrator.py --resume runs/<run-id>/state.json

When to use: Regression, CI/CD, overnight batch verification, re-running a known scenario.

Cost: Zero (no LLM needed).

Typical Workflow

1. Write runbook (once)           → runbooks/my-flow.md
2. LLM generates YAML (once)     → flows/my-flow.yaml        ← costs tokens
3. Review & commit YAML           → version-controlled
4. Run YAML repeatedly (forever)  → python orchestrator.py    ← free
5. On failure → LLM investigates  → adjust YAML or fix API    ← costs tokens

Run Output

Each run creates a self-contained folder:

runs/petstore-ftl-20260320-144357/
  ├── petstore-ftl.yaml     # Flow YAML snapshot (frozen at run time)
  ├── templates/             # .ftl templates used (frozen)
  ├── state.json             # Execution state + variables
  ├── report.md              # Human-readable test report
  └── debug.log              # Full trace: HTTP requests, template renders, assertions

Source files may change later — the run folder preserves exactly what was executed.

Debug Logging

On by default. Set OPENTEST_DEBUG=0 to disable.

Logs cover all three layers:

• orchestrator — step execution, assertions, saved variables
• ftl.py — template path, input vars, rendered output
• opentest.sh — HTTP method/URL, request body, response

Installation

# Clone
git clone https://github.com/apexphere/opentest.git
cd opentest

# Dependencies
pip install pyyaml

# Start test target (Petstore)
docker compose up -d

# Verify
python3 scripts/orchestrator.py flows/petstore-purchase.yaml

Project Structure

opentest/
├── runbooks/           # Human-language test descriptions
├── flows/              # Machine-executable YAML (from runbooks or hand-written)
│   └── schema.md       # YAML schema reference
├── runs/               # Per-run folders: state + report + debug log + snapshots
├── templates/          # FreeMarker (.ftl) request body templates
├── scripts/
│   ├── ftl.py          # Lightweight FTL renderer (Python, no Java needed)
│   ├── validate.py     # Validate flow YAML structure + paths against spec
│   ├── orchestrator.py # YAML → execution (deterministic, no LLM)
│   └── opentest.sh     # Thin curl wrapper (structured JSON output)
├── specs/              # OpenAPI specs
├── config/             # Environment config
└── docker-compose.yml  # Test target (Petstore for PoC)

Async Flow Support

Business flows often span time — a payment created now might not be batched until tonight.

Session 1:  create payment → poll batch → timeout → save state → exit
            (hours pass, batch processes overnight)
Session 2:  resume from state → poll → done → verify → report

State is saved after every step. No long-running sessions. Any session can resume any flow.

FreeMarker Templates

Reuse existing .ftl templates from your Camel/Java projects. The Python FTL renderer supports:

• Variable interpolation (${varName})
• Conditionals (<#if>, <#else>)
• Null checks (??, ?has_content)
• Default values (${var!"default"})
• Iterations (<#list>)

Reference templates in flow YAML:

- name: create-pet
  action: POST /pet
  template: templates/create-pet.ftl
  vars:
    petId: 70001
    petName: "Mochi"
    status: "available"

Docs

• Design — architecture, rationale, component design
• Flow Schema — YAML flow definition reference
• Data Isolation — test environment strategies