Skip to content

Run the Go test suites verbosely so CI shows what executed - #167

Open
stevebeattie wants to merge 1 commit into
chainguard-dev:mainfrom
stevebeattie:verbose-go-tests
Open

Run the Go test suites verbosely so CI shows what executed#167
stevebeattie wants to merge 1 commit into
chainguard-dev:mainfrom
stevebeattie:verbose-go-tests

Conversation

@stevebeattie

Copy link
Copy Markdown
Member

Why

A bare go test prints only ok <package>. That cannot distinguish a suite
that ran from one that silently covered less than expected — a newly added
table row or fixture that never runs looks exactly like one that passes.

Not hypothetical here: the offline matrix pairs fixtures with package-level
blobs, which Go does not report as unused, so a row dropped in a refactor leaves
a compiling, green, quieter suite. Nothing in the log would say so.

The immediate prompt was #164. It passes the mirror check in CI, but the log
showed only ok — so the thing that PR is about, the compared-file count
going from 7 to 8, could not be confirmed from the run at all.

Change

-v on test-offline and validate_mirrors. One-line each, plus a comment
explaining why.

What the logs now show

    --- PASS: TestOfflineFixtureMatrix/certificate_audit/fail_java_missing_stamp
    ... 47 matrix rows, each named
    mirrors_test.go:390: compared 7 standalone OVAL file(s) against the datastream
    mirrors_test.go:393: present only in the datastream, so not evaluable with
        `oscap oval eval` and with no input for regeneration: oval:org.stub:def:1

The matrix row count and the mirror check's file count are both now readable
from a CI run, so a drop in either is visible rather than inferred.

Verification

make validate_mirrors and make test-offline both pass: 169 named subtests,
47 matrix rows, 0 failures. The full offline log grows to 846 lines, which seems
a fair trade for being able to tell a green run from a quiet one.

Note

Touches the Makefile, as does #166. The edits are in different regions
(lint-go versus these two recipes) so they should merge cleanly in either
order, but worth knowing if both are in flight.

🤖 Generated with Claude Code

@stevebeattie
stevebeattie requested a review from egibs August 26, 2026 05:54
@stevebeattie
stevebeattie enabled auto-merge August 26, 2026 06:17
A bare `go test` prints only `ok <package>`, which cannot distinguish a suite
that ran from one that silently covered less than expected. A newly added
table row or fixture that never runs looks exactly like one that passes, and
that is not a hypothetical failure mode here: the offline matrix pairs
fixtures with package-level blobs that Go does not report as unused, so a row
dropped in a refactor leaves a compiling, green, quieter suite.

Add -v to test-offline and validate_mirrors. The logs now name every test and
subtest, so coverage is checkable after the fact rather than assumed:

  - the offline matrix lists its 47 rows by name
  - the mirror check reports "compared 7 standalone OVAL file(s)" and which
    definitions exist only in the datastream, which were previously only
    visible when running it locally

That last point is what prompted this: chainguard-dev#164 passes the mirror check in CI, but
its log showed only `ok`, so the thing the PR is about — the file count going
from 7 to 8 — could not be confirmed from the run.

Costs 846 log lines for the full offline suite, which is a fair trade for
being able to tell a green run from a quiet one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@stevebeattie
stevebeattie disabled auto-merge August 26, 2026 16:27
@stevebeattie
stevebeattie marked this pull request as draft August 26, 2026 16:27
@stevebeattie
stevebeattie marked this pull request as ready for review August 26, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant