Skip to content

pipeline.md

Latest commit

 

History

History
390 lines (318 loc) · 19 KB

pipeline.md

File metadata and controls

390 lines (318 loc) · 19 KB
Raw

The pipeline in git terms

How a work item flows through the git topology, and why each step looks the way it does.

Topology

                 ┌──────────────────────────────┐
                 │  Upstream remote (GitHub)    │  ← orchestrator pushes here
                 │   (optional, host-only)      │     using its own creds
                 └────────────┬─────────────────┘
                              │ phase 3
                ┌─────────────▼──────────────────┐
                │ Host bare repo  (per work item)│  ← source of truth
                │ /var/lib/codeybox/repos/<id> │
                └─┬──────────┬───────────────┬───┘
       phase 1   │  phase 2 │      phase 3   │
       (clone+   ▼  (audit+ ▼     (clone+    ▼
        push)              rework)            agent merge+push)
        ┌─────────────┐  ┌─────────────────┐  ┌─────────────────┐
        │ Work sbx    │  │ Audit sandboxes │  │ Merge sbx       │
        │(agent+creds)│  │ tool: no creds  │  │ (agent+creds,   │
        │             │  │ llm: agent creds│  │  merge profile) │
        └─────────────┘  └─────────────────┘  └─────────────────┘

The host bare repo is the source of truth. Work, audit, rework, and merge sandboxes each clone and push to it. The orchestrator pushes from it to upstream. Sandboxes never see the upstream URL or creds.

On every EnsureRepositoryAsync call with a non-null upstream seed URL, an existing host bare repo refreshes the configured base branch from that upstream before the sandbox clone. If no base branch is configured, the host resolves the upstream-advertised default branch and refreshes that branch instead of assuming main. The fetch updates only the selected base branch ref, so per-work-item refs such as codeybox/<id> are preserved. Before the host fetch, the orchestrator replaces the sandbox-writable bare-repo config with a minimal host-controlled config so repo-local credential helpers, SSH commands, and URL rewrites cannot influence the host command. Host git commands set core.hooksPath to an empty host-controlled directory, so hooks written under the bare repo are not executed during ref updates. If the upstream fetch exits non-zero, the orchestrator logs a redacted warning and continues with the previous local tip instead of deleting the bare repo.

After that refresh, pickup performs a sandboxed work-branch freshness check before any work, audit, or merge agent sees the repository. If the work branch already exists, the sandbox fetches the refreshed base and work refs, checks whether the refreshed base is already an ancestor of the work branch, and when needed runs:

git checkout -B <workBranch> origin/<workBranch>
git rebase --keep-empty origin/<baseBranch>
git push --force-with-lease=refs/heads/<workBranch>:<oldTip> origin HEAD:refs/heads/<workBranch>

First pickup is a no-op because there is no work-branch ref yet. Pickup rebase is only allowed to force-push the server-owned per-item codeybox/<work-item-id-prefix> branch. Explicit work branches created by the API or replay flow, including other codeybox/* names, are not rewritten by pickup rebase. Items resumed from Merged for upstream-push recovery also skip pickup rebase because they will not enter work, audit, or merge. Rebase conflicts are read through bounded, canonical in-worktree file reads, abort the in-sandbox rebase before any push on failure, and move the work item to MergeConflictResolutionFailed, the same operator-facing failure state used by scope-fenced merge conflict resolution.

Phase 1: Work

# Inside the work sandbox:
git clone /repos/<id>.git /work
cd /work
git checkout -B codeybox/<id> origin/<baseBranch>      # first pickup
# or, on retry after the pickup rebase above:
git checkout -B codeybox/<id> origin/codeybox/<id>

# Identity resolution (precedence: project > host global git config > fallback)
git config user.email <resolved-email>
git config user.name  <resolved-name>

# (the agent runs here — claude / copilot / codex / gemini)
# It edits files in /work and may or may not commit.

git add -A
# Strip suggestions file before committing — it is advisory metadata,
# not part of the work branch history. Exit code ignored if not staged.
git rm --cached -- .codeybox/suggestions.json
git diff --cached --quiet && exit 1      # fail if no changes
git commit -m "codeybox: <title>

<trailer block — see Provenance in the README>"
git push origin codeybox/<id>:codeybox/<id>

If the sandbox push of the work branch is rejected as non-fast-forward, the orchestrator performs exactly one in-sandbox reconcile attempt:

git fetch --no-tags origin +refs/heads/<workBranch>:refs/remotes/origin/<workBranch>
git rebase origin/<workBranch>
git push origin <workBranch>:<workBranch>

Rebasing rather than deleting the stale ref keeps the previous attempt's commits reachable, which matches how host-side upstream-push recovery behaves and keeps attempt history in one place. A rebase conflict fails the work item with a manual-resolution error.

Git identity propagation

By default the orchestrator reads the host's global git config at startup (git config --global user.name / user.email) and uses those values for all sandbox commits. This lets git blame / git log --author=alice show the real operator who triggered the work item.

Resolution order (first match wins):

  1. Project.GitAuthorName / Project.GitAuthorEmail — set both fields on a project to override for that project only.
  2. Host global git config — read once at orchestrator startup via git config --global. Requires git on PATH and ~/.gitconfig configured.
  3. Synthetic fallback — CodeyBox <codeybox@local>. Used when no global git config is found (fresh containers, CI runners without a configured identity). A warning is logged at startup.

Commit trailers

Every orchestrator-produced commit ends with a trailer block — CodeyBox-WorkItem, CodeyBox-Agent, CodeyBox-Prompt-Revision when known, CodeyBox-Fallbacks when a fallback happened, and finally Co-Authored-By: CodeyBox <noreply@codeybox.invalid>. Mechanical-fixer commits carry CodeyBox-Mechanical-Fixer in place of CodeyBox-Agent.

The .invalid TLD (RFC 2606) signals "not a real email" while GitHub still renders the trailer in the PR conversation view. The operator's email never appears there — only noreply@codeybox.invalid.

The orchestrator then opens a PR record via IPullRequestService. With the default in-memory impl this is just metadata; with a Gitea/Forgejo backend it would be a real PR on a self-hosted forge.

Phase 2: Audit + rework loop

Skipped if no auditors are registered. See audit.md for the full breakdown. Tool auditors run in a credential-free sandbox; LLM auditors run in a sandbox with agent credentials. On failure the agent reworks (pushing further commits onto the same codeybox/<id> branch) and the loop reruns. On MaxIterations without convergence the work item flips to AuditFailed.

Each iteration is a cost-ordered ladder: code-stage auditors first (mechanical gates, then reviewers — a blocking fail short-circuits the rest with zero deployment cost), then, only on a clean code stage, the deployment stage. When Audit.DeploymentAuditEnabled is on, the project has a deployment recipe, and deployment-targeted auditors are enabled, the runner lazily provisions exactly one verification deployment, hands its endpoint to those auditors, and tears it down on every exit path (pass, fail, abort, timeout — the leak reaper covers orchestrator restarts). The next iteration provisions a fresh deployment. Deployment findings re-enter the normal rework loop with full blocking authority.

Phase 1.D: Delegation (operator-triggered escape hatch)

An item that cannot converge through the normal work/audit/rework cycle has no route back except an operator repairing it by hand. Delegation performs that repair inside the pipeline, with the same sandboxing as every other phase: a single delegate turn runs in a sandbox with the repository (work-profile sandbox target) on the existing work branch.

The operator requests it with POST /workitems/{id}/retry and from='delegation', which moves the item to the Delegating state and arms a one-shot DelegationRequested flag. The delegate prompt combines the composed convergence brief (prior history, failures, audit findings) with an instruction to get the item working — and grants latitude the work phase does not: the delegate may change approach, restructure the change, or alter tests and configuration where those are the actual obstacle.

Exactly one attempt per trigger. On completion the item advances to WorkComplete and re-enters the normal flow at the audit phase, so the result is verified by the same gates as any other change and is never merged on the delegate's assurance. A delegate that fails or produces no change parks the item at NeedsOperatorInput with the reason; it never returns to the cycle that already failed, and the phase can never re-enter itself. Quota/transient parks mid-turn resume the same attempt through the scheduler; they do not consume the attempt.

Every turn is recorded as a first-class delegation event — the brief it was given, the agent and model that ran it, and the resulting branch diff — retrievable via GET /workitems/{id}/delegations.

Phase 3: Merge

The host computes the merge first:

git merge-tree --write-tree --no-messages <preMergeBaseSha> <workTipSha>

No conflicts — no sandbox at all. The host writes the merge commit directly from that tree, then re-verifies before moving the base ref:

git commit-tree <mergedTree> -p <preMergeBaseSha> -p <workTipSha> -m "codeybox: merge <workBranch>…"
# then: both parents reachable, committed tree == merge-tree result
git push origin <baseBranch>:<baseBranch>

Two parents means the work branch stays visible in history, exactly as git merge --no-ff would leave it.

Before the base ref moves, the merge queue builds the exact merge-result tree with the project's required build (the same command as the required-build gate) in a sandbox, serialized per base branch so each verification runs against the base it will land on. A clean textual merge of two branches that each built on their own base can still fail to compile when combined — that tree is not landed. Instead the item takes conflict-rework turns against the refreshed base, briefed with the build errors. If the base moves during verification, the merge is recomposed and re-verified; an unverified tree never lands. Disable with CodeyBox:PipelineTuning:MergeResultBuildVerificationEnabled (default on).

Conflicts — the agent resolves them in its own sandbox. The merge sandbox runs under the project's merge network profile. The agent reads the conflicted files from the working tree, writes resolutions, and stages them through its normal CLI. The orchestrator then checks there are no unmerged paths and no conflict markers left in any originally-conflicted file, commits, and applies the scope fence described in architecture.md: every changed line must fall inside a conflict span plus Audit.MergeScopeBufferLines of context. Anything outside — a new file, a rename, an edit to a file that never conflicted — rejects the resolution and the item lands in MergeConflictResolutionFailed.

Phase 4: Upstream push (host)

The orchestrator calls IUpstreamRemote.CompleteAsync with the work branch name, base branch, and the merge SHA from phase 3. The implementation decides what "complete" means for its upstream type.

git-generic

git -C /var/lib/codeybox/repos/<id>.git push <url> <baseBranch>:<baseBranch>

Pushes the merged base branch. No PR concept.

GitHub

1. git push origin <workBranch>:<workBranch>   # token via GIT_ASKPASS
2. Generate PR description (LLM or static fallback — see below)
3. POST https://api.github.com/repos/{owner}/{repo}/pulls
       { "title": "<pr title>", "head": "<workBranch>", "base": "<baseBranch>",
         "body": "<generated description>" }
4. [if AutoMerge=true]
   PUT  https://api.github.com/repos/{owner}/{repo}/pulls/{n}/merge
       { "merge_method": "<merge|squash|rebase>" }

The PAT is set as a per-request Authorization: token <PAT> header; it never appears on argv, in config files, or in log output (scrubbed from any error message). The named HttpClient "github-upstream" carries the User-Agent: codeybox header required by the GitHub API.

LLM-generated PR descriptions

Step 2 generates a narrative PR body when Upstream.PrDescription.Enabled is true. Two strategies implement IPullRequestDescriptionGenerator, selected per project by Upstream.PrDescription.Strategy (default "Completion" — project configuration reloads without a restart, so switching strategies takes effect on the next work item):

Strategy How it generates Requires
Completion (default) A single tool-less completion call via ICompletionClient against CompletionEndpoint. Creates no sandbox and runs no agent, so generation never contends for sandbox permits. CompletionEndpoint (absolute http(s) URL) plus CompletionModel
Agentic Runs the configured agent (GeneratorAgent, default "claude") inside a minimal sandbox over the diff. Retained as a supported option; places a sandbox acquisition on the merge critical path and executes an agent over attacker-influenceable diff content. SandboxImageReference naming an image with the generator agent's CLI installed

Availability is evaluated against the selected strategy's own requirement: a project selecting Completion is not disabled by a missing SandboxImageReference, and vice versa. When the requirement is missing (or Enabled is false) the static template is used immediately with no LLM call.

Inputs (both strategies):

  • git diff --stat (compact change summary)
  • Full git diff between base and work branches, capped at MaxDiffBytes (default 32 KB) and truncated from the middle — equal portions from the start and end are preserved and a [… N bytes truncated …] marker is inserted — so the LLM sees both the first and last diff hunks of a large changeset.
  • The original work item prompt (truncated to 2 KB).
  • Titles of audit findings addressed during rework iterations.
  • Agent commit messages from the work branch (up to 20, 2 KB each).
  • Last 2 KB of agent stdout (the agent's concluding reasoning).

Both strategies share the same prompt shape, middle-out truncation, and RawOutputRedactor sanitisation of inputs and outputs (accidentally-committed tokens in the diff or echoed back by the model are replaced with ***), via a shared base class — a future third strategy inherits the same safeguards.

Rendered body — the generated prose never stands alone: the deterministic facts (work item id and changed-file list) stay in the body adjacent to the generated text, and the generated section carries a "Machine-generated summary" notice so reviewers and downstream automation do not treat model output over untrusted diff content as authoritative. The same generated body feeds the squash merge commit message (CleanProseForCommitMessage), with the template scaffolding stripped so the commit carries the narrative.

Fallback semantics — generation never fails or delays a pull request:

Condition Behaviour
Enabled = false, or the selected strategy's requirement missing Static template used immediately; no LLM call
Generator succeeds Generated body used as PR description
Generator times out (Timeout, default 30 s, the deadline for the whole round trip) Warning logged; static template used
Generator throws (transport/auth failure, sandbox provisioning failure, empty completion) Warning logged; static template used

The standard footer (Co-Authored-By: CodeyBox <noreply@codeybox.invalid> plus a generated-with link) is appended to the PR body in all cases.

Configuration (appsettings.json, under Upstream for a GitHub project):

"Upstream": {
  "Kind": "github",
  ...
  "PrDescription": {
    "Enabled": true,
    "Strategy": "Completion",
    "CompletionEndpoint": "https://api.openai.com/v1/chat/completions",
    "CompletionModel": "gpt-4o-mini",
    "GeneratorAgent": "claude",
    "GeneratorModelId": null,
    "MaxDiffBytes": 32768,
    "Timeout": "00:00:30",
    "SandboxImageReference": "ghcr.io/myorg/codeybox-agent:latest",
    "AgentAllowedHosts": ["api.anthropic.com"]
  }
}

CompletionApiKey (or the first non-empty CompletionApiKeyEnvVars entry, default CODEYBOX_PR_DESCRIPTION_API_KEY), CompletionWireApi (default OpenAiChatCompletions) and CompletionMaxOutputTokens (default 1024) tune the completion call. SandboxImageReference and AgentAllowedHosts apply only to the Agentic strategy.

Cost — each PR description adds approximately 5 K input tokens and 500 output tokens. This appears in the per-work-item cost report as a separate phase=upstream row. Operators on tight quotas can set Enabled: false.

Soft failures are handled gracefully without retrying:

Status Scenario Behaviour
422 on POST /pulls Branch already has an open PR Log warning, return BranchPushed=true, PR fields null
405 on PUT /pulls/N/merge Branch protection prevents merge Log warning, return PullRequestUrl set, MergedSha null

All other errors throw, triggering the orchestrator retry loop.

If every attempt fails (network outage, unexpected HTTP errors), the item is marked Failed after UpstreamPushMaxAttempts retries with UpstreamPushBackoff between them. The local merge in the host bare repo is unaffected; an operator can re-queue to retry phase 4 independently.

Why bare repos per work item

  • Isolation. Work item A's branch can never end up in work item B's bare repo, so a misbehaving agent on A can't pollute B.
  • Cleanup. Disposing the bare repo is rm -rf — no hunting for orphan branches.
  • Concurrency. Two work items on the same upstream can run in parallel without sharing any git state until they hit upstream.

The trade-off is more disk usage. For deployments with many concurrent items, future work could share a single bare repo across items keyed by upstream URL — IGitHost is the abstraction to evolve.

Conflict handling

Conflicts are resolved by the agent inside the merge sandbox, and accepted only by the deterministic host-side checks above. A resolution that edits outside the conflict spans is rejected rather than pushed — the model cannot use "resolving a conflict" as cover for other edits.