Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ The reusable workflow declares the expected typed review artifacts for Docs Agen

The target repository grants `contents: write`, `pull-requests: write`, and `issues: write`. Docs Agent forwards the caller-scoped `${{ github.token }}` to WP Codebox for same-repository publication, so consumers do not configure `ACCESS_TOKEN`. `OPENAI_API_KEY` is an optional workflow secret and is required only for a live OpenAI run; skipped and dry-run calls do not require it. `EXTERNAL_PACKAGE_SOURCE_POLICY` remains a separate required v1 JSON secret. Migrate its value to the exact one-line JSON in the root README: it authorizes the selected Docs Agent package, the pinned Agents API component, the pinned PHP AI Client overlay, and the checksum-pinned OpenAI provider artifact. Both secrets are forwarded to WP Codebox without serialization into the task descriptor.

Docs Agent consumes the released [WP Codebox v0.12.21](https://github.com/Automattic/wp-codebox/releases/tag/v0.12.21) workflow and passes the matching `wp_codebox_release_ref: v0.12.21` input. WP Codebox validates the paired tags, materializes the declared native runtime closure, executes the selected package, applies bounded workspace changes, runs verification, and publishes approved host changes.
Docs Agent consumes the released [WP Codebox v0.12.23](https://github.com/Automattic/wp-codebox/releases/tag/v0.12.23) workflow and passes the matching `wp_codebox_release_ref: v0.12.23` input. The release tag resolves to immutable revision `a2b02cd99ba645ba2250bf58c943bdd1eb13e690`. WP Codebox validates the paired tags, materializes the declared native runtime closure, executes the selected package, applies bounded workspace changes, runs verification, and publishes approved host changes.

Provider artifacts retain only allowlisted review artifacts, controlled workflow envelopes, and canonical runtime-source and seed provenance. WP Codebox v0.12.21 uploads a reviewer-safe workflow-result projection: public control, verification, publication, output-projection, access, failure, artifact-declaration, and canonical transcript-provenance fields remain, while raw `runtime_result`, `outputs.engine_data`, model/provider/tool payloads, source content, private paths, and secrets are excluded. Before persistence or upload, it sanitizes private runtime paths from nested values, object keys, diagnostics, and command arguments. WP Codebox captures one trusted canonical `codebox-transcript` through a pre-sanitization reviewer-evidence descriptor, validates its artifact-root containment, schema, digest, and size, then emits the compact reviewer transcript as a `wp-codebox/reviewer-agent-transcript/v1` projection. The canonical transcript remains the bounded tool-observability surface. Lifecycle and downstream failures retain runtime evidence in their internal normalized failed result even when artifact preparation cannot complete. Diagnostic messages that name runtime classes remain reviewable, while PHP-shaped runtime source remains blocked. See [WP Codebox #1767](https://github.com/Automattic/wp-codebox/issues/1767), [WP Codebox #1814](https://github.com/Automattic/wp-codebox/issues/1814), and [WP Codebox #1815](https://github.com/Automattic/wp-codebox/pull/1815). Successful canonical reviewer-evidence reference: [run `29350690551`](https://github.com/Automattic/build-with-wordpress/actions/runs/29350690551). The [hosted failure `29468956857`](https://github.com/Automattic/agents-api/actions/runs/29468956857) executed WP Codebox v0.12.20, published [Agents API #430](https://github.com/Automattic/agents-api/pull/430), then failed during upload preparation; it motivated #1814/#1815 and is not a successful v0.12.21 reviewer-safe workflow-result run.
Provider artifacts retain only allowlisted review artifacts, controlled workflow envelopes, and canonical runtime-source and seed provenance. WP Codebox v0.12.23 uploads a reviewer-safe workflow-result projection: public control, verification, publication, output-projection, access, failure, artifact-declaration, and canonical transcript-provenance fields remain, while raw `runtime_result`, `outputs.engine_data`, model/provider/tool payloads, source content, private paths, and secrets are excluded. Before persistence or upload, it sanitizes private runtime paths from nested values, object keys, diagnostics, and command arguments. WP Codebox captures one trusted canonical `codebox-transcript` through a pre-sanitization reviewer-evidence descriptor, validates its artifact-root containment, schema, digest, and size, then emits the compact reviewer transcript as a `wp-codebox/reviewer-agent-transcript/v1` projection. Before apply-back, it compares the seed identity with the host workspace identity; rejected patches preserve the expected and actual identities plus patch and changed-file evidence for upload. The canonical transcript remains the bounded tool-observability surface. Lifecycle and downstream failures retain runtime evidence in their internal normalized failed result even when artifact preparation cannot complete. Diagnostic messages that name runtime classes remain reviewable, while PHP-shaped runtime source remains blocked. See [WP Codebox #1767](https://github.com/Automattic/wp-codebox/issues/1767), [WP Codebox #1814](https://github.com/Automattic/wp-codebox/issues/1814), [WP Codebox #1815](https://github.com/Automattic/wp-codebox/pull/1815), and [WP Codebox #1823](https://github.com/Automattic/wp-codebox/pull/1823). Successful canonical reviewer-evidence reference: [run `29350690551`](https://github.com/Automattic/build-with-wordpress/actions/runs/29350690551). The [hosted failure `29468956857`](https://github.com/Automattic/agents-api/actions/runs/29468956857) executed WP Codebox v0.12.20, published [Agents API #430](https://github.com/Automattic/agents-api/pull/430), then failed during upload preparation; it motivated #1814/#1815 and is not a successful v0.12.23 reviewer-safe workflow-result run.

## Docs Agent Runner Recipe

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/maintain-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -249,9 +249,9 @@ jobs:
contents: write
pull-requests: write
issues: write
uses: Automattic/wp-codebox/.github/workflows/run-agent-task.yml@v0.12.21
uses: Automattic/wp-codebox/.github/workflows/run-agent-task.yml@v0.12.23
with:
wp_codebox_release_ref: v0.12.21
wp_codebox_release_ref: v0.12.23
external_package_source: ${{ needs.prepare.outputs.external_package_source }}
runtime_sources: ${{ needs.prepare.outputs.runtime_sources }}
workload_id: docs-agent-${{ inputs.audience }}-${{ inputs.run_kind }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
uses: actions/checkout@v4
with:
repository: Automattic/wp-codebox
ref: v0.12.21
ref: v0.12.23
path: .wp-codebox
- name: Validate Docs Agent
env:
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Docs Agent declares the review artifacts it expects the runner to materialize as

`maintain-docs.yml` writes `expected_artifacts` and `artifact_declarations` into a portable Docs Agent recipe and exposes the same declaration objects as `declared_artifacts_json`.

WP Codebox v0.12.21 uploads a reviewer-safe workflow-result projection with public control and publication fields plus canonical transcript provenance. It excludes raw `runtime_result`, `outputs.engine_data`, model/provider/tool payloads, source content, private paths, and secrets. The canonical `codebox-transcript` remains the bounded tool-observability surface: its pre-sanitization reviewer-evidence descriptor records the trusted artifact-relative path, schema, verified source digest, and size, which the uploader revalidates before producing the `wp-codebox/reviewer-agent-transcript/v1` projection.
WP Codebox v0.12.23 at `a2b02cd99ba645ba2250bf58c943bdd1eb13e690` uploads a reviewer-safe workflow-result projection with public control and publication fields plus canonical transcript provenance. It excludes raw `runtime_result`, `outputs.engine_data`, model/provider/tool payloads, source content, private paths, and secrets. The canonical `codebox-transcript` remains the bounded tool-observability surface: its pre-sanitization reviewer-evidence descriptor records the trusted artifact-relative path, schema, verified source digest, and size, which the uploader revalidates before producing the `wp-codebox/reviewer-agent-transcript/v1` projection. Before apply-back, it validates that the runner seed and host workspace identities match; rejected patches retain identity, patch, and changed-file evidence for review.

Docs Agent owns native package selection, lane, artifact, prompt, and workspace mapping. Execution, credentials, AI provider selection, sandboxing, and publication are runner-owned concerns outside this repository.

Expand Down
6 changes: 3 additions & 3 deletions tests/validate-docs-agent-packages.php
Original file line number Diff line number Diff line change
Expand Up @@ -70,9 +70,9 @@
$assert( str_contains( $maintain_docs_workflow, 'artifact_declarations<<EOF' ), 'maintain-docs.yml must prepare typed artifact declarations without caller-specific projections.' );
$assert( str_contains( $maintain_docs_workflow, 'artifact_declarations<<EOF' ), 'maintain-docs.yml must expose artifact declarations through workflow outputs.' );

$generic_codebox_agent_task_workflow = 'uses: Automattic/wp-codebox/.github/workflows/run-agent-task.yml@v0.12.21';
$generic_codebox_agent_task_workflow = 'uses: Automattic/wp-codebox/.github/workflows/run-agent-task.yml@v0.12.23';
$assert( str_contains( $maintain_docs_workflow, $generic_codebox_agent_task_workflow ), 'maintain-docs.yml must call the generic Codebox agent-task workflow.' );
$assert( str_contains( $maintain_docs_workflow, 'wp_codebox_release_ref: v0.12.21' ), 'maintain-docs.yml must pass the matching WP Codebox release tag.' );
$assert( str_contains( $maintain_docs_workflow, 'wp_codebox_release_ref: v0.12.23' ), 'maintain-docs.yml must pass the matching WP Codebox release tag.' );

$assert( str_contains( $maintain_docs_workflow, '--arg writablePaths "$INPUT_WRITABLE_PATHS"' ), 'maintain-docs.yml must include writable paths in the portable recipe.' );
$assert( str_contains( $maintain_docs_workflow, 'output_projections:' ), 'maintain-docs.yml must project the bounded runner publication result.' );
Expand All @@ -85,7 +85,7 @@
foreach ( array( 'Docs Agent Runner Recipe', 'portable recipe', 'Docs Agent owns the native package' ) as $migration_note_text ) {
$assert( str_contains( $workflow_readme, $migration_note_text ), "Workflow README missing agent runtime note: {$migration_note_text}" );
}
$assert( str_contains( $workflow_readme, 'v0.12.21' ), 'Workflow README must record the WP Codebox release tag.' );
$assert( str_contains( $workflow_readme, 'v0.12.23' ), 'Workflow README must record the WP Codebox release tag.' );
$assert( str_contains( $workflow_readme, 'Diagnostic messages that name runtime classes remain reviewable' ), 'Workflow README must document diagnostic-versus-source detection.' );
$assert( str_contains( $workflow_readme, 'allowlisted review artifacts' ), 'Workflow README must document the WP Codebox upload allowlist.' );
$assert( str_contains( $workflow_readme, 'normalized failed result' ), 'Workflow README must document normalized WP Codebox failures.' );
Expand Down
21 changes: 18 additions & 3 deletions tests/validate-wp-codebox-run-agent-task-contract.php
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,16 @@
$release = $read_json( $root . '/tests/wp-codebox-release.fixture.json' );
$release_tag = $release['tag'] ?? null;
$assert( is_string( $release_tag ) && preg_match( '/^v\d+\.\d+\.\d+$/', $release_tag ) === 1, 'WP Codebox release fixture must declare an exact release tag.' );
$release_revision = $release['revision'] ?? null;
$assert( is_string( $release_revision ) && preg_match( '/^[0-9a-f]{40}$/', $release_revision ) === 1, 'WP Codebox release fixture must declare the immutable tag commit.' );
$assert( substr( $release_tag, 1 ) === ( $release['package_version'] ?? null ), 'WP Codebox release fixture package version must match its tag.' );
$published_assets = $release['published_assets'] ?? null;
$assert( is_array( $published_assets ), 'WP Codebox release fixture must record published release assets.' );
$assert( array(
'manifest.json' => 'sha256:41784656d82ae07ec84254b004c6e771d52f36ee6e553aae59b93a1be7ced85d',
'wp-codebox-workspace-0.12.23.tgz' => 'sha256:685d3b437d7cc813572aa6a5fa02ebcd285009a19989c99f7de8d8edf8611417',
'wp-codebox.zip' => 'sha256:87320aa1241dd3e45dc7cb1ab09e0f2ddfe37900017f39519a720ccbf7058171',
) === $published_assets, 'WP Codebox release fixture must retain the published v0.12.23 asset digests.' );
$run = $release['run'] ?? null;
$assert( is_string( $run ) && preg_match( '/^\d+$/', $run ) === 1, 'WP Codebox release fixture must retain the regression run reference.' );
$diagnostic_regression_run = $release['diagnostic_regression_run'] ?? null;
Expand Down Expand Up @@ -78,12 +87,16 @@
$assert( true === ( $release['canonical_transcript_privacy_boundary'] ?? null ), 'WP Codebox release fixture must preserve the canonical transcript privacy boundary.' );
$assert( true === ( $release['pre_sanitization_canonical_reviewer_evidence_descriptor'] ?? null ), 'WP Codebox release fixture must preserve the pre-sanitization canonical reviewer-evidence descriptor.' );
$assert( true === ( $release['reviewer_safe_workflow_result_projection'] ?? null ), 'WP Codebox release fixture must require the reviewer-safe workflow-result projection.' );
$assert( true === ( $release['runner_workspace_seed_host_identity_validation'] ?? null ), 'WP Codebox release fixture must validate seed and host identity before apply-back.' );
$assert( true === ( $release['rejected_apply_evidence_retained'] ?? null ), 'WP Codebox release fixture must retain rejected apply evidence.' );
$native_result_path = $release['native_result_path'] ?? null;
$workflow_result_path = $release['workflow_result_path'] ?? null;
$assert( '.codebox/native-agent-task-result.json' === $native_result_path, 'WP Codebox release fixture must declare the controlled native result path.' );
$assert( '.codebox/agent-task-workflow-result.json' === $workflow_result_path, 'WP Codebox release fixture must declare the workflow result path.' );
$producer_package = $read_json( rtrim( $wp_codebox_dir, '/' ) . '/package.json' );
$assert( ( $release['package_version'] ?? null ) === ( $producer_package['version'] ?? null ), 'Checked-out WP Codebox package version must match the release fixture.' );
$producer_revision = trim( (string) shell_exec( 'git -C ' . escapeshellarg( $wp_codebox_dir ) . ' rev-parse HEAD' ) );
$assert( $release_revision === $producer_revision, 'Checked-out WP Codebox producer must match the immutable release tag commit.' );

$workflow = (string) file_get_contents( $root . '/.github/workflows/maintain-docs.yml' );
$assert( preg_match( '/^\s*uses: Automattic\/wp-codebox\/\.github\/workflows\/run-agent-task\.yml@' . preg_quote( $release_tag, '/' ) . '$/m', $workflow ) === 1, 'Docs Agent must call the released WP Codebox workflow tag.' );
Expand Down Expand Up @@ -253,7 +266,9 @@
$assert( str_contains( $producer_execute, 'task_input: {' ) && str_contains( $producer_execute, 'workspaces: runnerWorkspaceSeedSnapshot ?' ), 'WP Codebox must hand runner workspaces through canonical task_input.workspaces.' );
$assert( str_contains( $producer_execute, 'createRunnerWorkspaceSeedSnapshot(workspace)' ) && str_contains( $producer_execute, 'seed: { type: "directory", source: runnerWorkspaceSeedSnapshot.source' ), 'WP Codebox must seed runner workspaces from an external snapshot.' );
$assert( str_contains( $producer_execute, 'runner_workspace_seed: runnerWorkspaceSeedSnapshot.provenance' ) && str_contains( $producer_execute, 'RUNNER_WORKSPACE_SEED_EXCLUDES' ), 'WP Codebox must retain seed provenance while applying the published exclusions.' );
$assert( str_contains( $producer_execute, 'applyRunnerWorkspacePatch({ artifactRoot: artifactsPath, artifactRefs: refs, workspaceRoot: workspace, writablePaths })' ), 'WP Codebox must apply the sandbox patch to the host workspace through its bounded patch contract.' );
$assert( str_contains( $producer_execute, 'applyRunnerWorkspacePatch({ artifactRoot: artifactsPath, artifactRefs: refs, workspaceRoot: workspace, writablePaths, seedIdentity: runnerWorkspaceSeedSnapshot?.provenance.identity })' ), 'WP Codebox must validate the seed identity before applying the sandbox patch to the host workspace.' );
$assert( str_contains( $producer_execute, '...(error?.evidence ? { evidence: error.evidence } : {})' ), 'WP Codebox must retain rejected apply evidence in the normalized failure result.' );
$assert( str_contains( $producer_upload, 'async function stageApplyFailureEvidence(result)' ) && str_contains( $producer_upload, 'apply-failure' ) && str_contains( $producer_upload, 'rejected.patch' ), 'WP Codebox must stage rejected patch and changed-file evidence for upload.' );
$assert( str_contains( $producer_execute, '.filter((ref) => ref.kind === "codebox-patch" || ref.kind === "codebox-changed-files")' ), 'WP Codebox must treat canonical patch and changed-file references as actionable workspace evidence.' );
$assert( str_contains( $producer_execute, 'verifyRunnerWorkspaceIntegrity(workspaceApply.integrity)' ) && str_contains( $producer_execute, 'publishRunnerWorkspace' ), 'WP Codebox must verify host-applied changes before publication.' );
$assert( str_contains( $producer_execute, 'runtime_result: redact(runtimeRecord)' ) && str_contains( $producer_execute, '...(downstreamFailure ? { failure:' ), 'WP Codebox must retain normalized runtime evidence when downstream execution fails.' );
Expand Down Expand Up @@ -285,7 +300,7 @@
$assert( in_array( '.codebox/agent-task-request.json', $producer_upload_regression['observed']['uploaded'] ?? array(), true ), 'WP Codebox upload regression fixture must retain the controlled request upload.' );
$assert( ! array_intersect( array( 'MODEL_PROVIDER_SECRET_1', 'MODEL_PROVIDER_SECRET_2', 'MODEL_PROVIDER_SECRET_3', 'MODEL_PROVIDER_SECRET_4', 'MODEL_PROVIDER_SECRET_5' ), array_keys( $caller_secrets ) ), 'Docs Agent must forward only the OPENAI_API_KEY provider secret name.' );

$assert( str_contains( $workflow, 'output_projections="$(jq -cn --arg path \'metadata.runner_workspace_publication.url\' --argjson required "$success_requires_pr" \'{docs_agent_publication:{path:$path,required:$required}}\')"' ), 'Docs Agent must define the v0.12.21 publication projection descriptor.' );
$assert( str_contains( $workflow, 'output_projections="$(jq -cn --arg path \'metadata.runner_workspace_publication.url\' --argjson required "$success_requires_pr" \'{docs_agent_publication:{path:$path,required:$required}}\')"' ), 'Docs Agent must define the v0.12.23 publication projection descriptor.' );
$docs_projections = array(
'docs_agent_publication' => array(
'path' => 'metadata.runner_workspace_publication.url',
Expand All @@ -295,7 +310,7 @@
$publication_descriptor = $docs_projections['docs_agent_publication'] ?? null;
$assert( is_array( $publication_descriptor ), 'Docs Agent must define the docs_agent_publication projection descriptor.' );
$publication_path = $publication_descriptor['path'] ?? null;
$assert( 'metadata.runner_workspace_publication.url' === $publication_path, 'Docs Agent publication projection must use the v0.12.21 runner workspace publication URL path.' );
$assert( 'metadata.runner_workspace_publication.url' === $publication_path, 'Docs Agent publication projection must use the v0.12.23 runner workspace publication URL path.' );

$producer_request_fixture = $read_json( rtrim( $wp_codebox_dir, '/' ) . '/contracts/agent-task-workflow-request.fixture.json' );
$producer_projection_paths = array_values( $producer_request_fixture['outputs']['projections'] ?? array() );
Expand Down
Loading
Loading