Skip to content

README.md

Latest commit

 

History

History
111 lines (73 loc) · 6.07 KB

README.md

File metadata and controls

111 lines (73 loc) · 6.07 KB
Raw

droid-control

Terminal, browser, and computer automation plugin for Droids.

Droids can read and write code. This plugin lets them operate it: launch apps, type commands, click buttons, record what happens, and produce evidence. OS permission grants and consequential actions still require the appropriate user or host authorization.

What you get

Record a demo video from a PR:

/demo pr-1847

Droid reads the PR, scripts the interactions that prove the change works, records both branches (in parallel when each has its own isolated terminal or browser environment), and renders a side-by-side comparison video. Use Factory presets for cinematic warmth or macos/minimal presets for clean utilitarian demos.

Verify a behavior claim:

/verify "ESC cancels streaming in bash mode"

Droid launches the app, tests the claim, and reports what actually happened with screenshots, snapshots, or byte captures. If the claim is false, that is a valid finding, not a failed run.

Run a QA flow against a web app:

/qa-test https://app.example.com -- login, create a project, invite a member

Droid drives the browser through the flow, captures each step, and reports pass/fail with annotated screenshots.

Quick start

# Register the Factory plugins marketplace (if not already added)
droid plugin marketplace add https://github.com/Factory-AI/factory-plugins

# Install the plugin
droid plugin install droid-control@factory-plugins --scope user

# Build the video renderer (one-time, only needed for video rendering; needs Rust, clang, libx264)
# Find the plugin install path with: droid plugin list --scope user
cargo build --release --manifest-path <plugin-path>/fframes/Cargo.toml

Or use the /plugins UI: Browse tab, select droid-control, install.

Then open a Droid session and run /demo, /verify, or /qa-test.

For ordinary desktop work, ask directly: “Using only cua, open Calculator and compute 17 × 23.” Desktop-use runs the observe/act/verify loop without loading video-production stages.

The desktop-use atom includes setup and operating guidance. Install the cua-driver executable if missing; no separate Cua skill installation is needed. Installed driver versions and Wayland compositors may support different capabilities.

Commands

/demo

Plans and records a demo video. Accepts a PR number, GitHub URL, or free-text description. Comparison PRs get side-by-side layout by default; new features get single-branch. Add showcase for cinematic polish or keys for keystroke overlay.

/verify

Tests a specific behavior claim and reports findings with evidence. The droid is an investigator, not an advocate: contradictory evidence is surfaced instead of hidden.

/qa-test

Runs automated QA against terminal CLIs, web apps, or Electron apps. Accepts a URL, CLI command, app name, PR reference, or free-text flow with optional test steps after --.

How it works

droid-control is a composition system for agent attention:

  1. Commands parse user intent into commitments.
  2. The orchestrator routes by target, stage, and artifact needs.
  3. Atoms provide only the mechanics needed right now: drivers, target patterns, capture, compose, verify, and showcase polish.
  4. Workers handle independent capture/render jobs. The parent keeps short interactive desktop tasks, including observations, input, permission waits, and cleanup.
  5. Verify checks the final evidence against the original commitments.

For the full rationale and runtime pipeline, see ARCHITECTURE.md.

Video rendering

The compose stage uses fframes, a Rust SVG video framework, for video compositing. Presets provide window chrome, spacing, palettes, backgrounds, particles, noise, color grading, configurable transitions (motion-blur, flash, whip-pan, light-leak, glitch-lite), zooms, spotlights, callout annotations, keystroke overlays, section headers, and syntax-highlighted code annotations.

The render-showcase.sh helper owns the full pipeline: .cast conversion via agg, per-render clip staging, longest-clip duration, rendering on every core (or a --still preview), and cleanup. It builds the droid-showcase binary from fframes/ on first use. Playback speed is applied once by the composition to every clip.

Prerequisites

Stage Platform Required
terminal-use (tuistory) All tuistory, asciinema, agg
true-input Linux/Wayland cage, wtype, Wayland terminal
true-input Windows (KVM) libvirt, qemu, KVM VM with SSH
true-input macOS (QEMU) qemu, socat, macOS VM with SSH
browser-use All agent-browser
desktop-use All cua-driver
compose All ffmpeg, ffprobe, agg
showcase All Rust (stable, via rustup), clang/libclang, libx264 
npm install -g tuistory                               # virtual PTY driver
pip install asciinema                                 # terminal recording
cargo install --git https://github.com/asciinema/agg  # .cast -> .gif converter
sudo apt-get install -y ffmpeg                        # video processing
agent-browser install                                 # browser automation (downloads Chromium)
curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash  # native desktop GUI automation
sudo apt-get install -y clang libclang-dev libx264-dev  # renderer build deps (macOS: xcode-select --install; brew install x264)
cargo build --release --manifest-path plugins/droid-control/fframes/Cargo.toml  # video renderer

Only install what you need, with approval. Terminal demos need tuistory, asciinema, agg, and ffmpeg. Web/Electron automation defaults to browser-use; an explicit cua-only/native-input request uses desktop-use instead. Native desktop automation needs cua-driver plus the graphical session and OS permissions reported by its preflight. Recording and rendering have additional dependencies; they are not required for ordinary desktop tasks.