Skip to content

[IMPROVEMENT] Fail the tests when a command line option has no test - #2410

Open
Deepak-negi11 wants to merge 2 commits into
CCExtractor:masterfrom
Deepak-negi11:test/cli-option-coverage
Open

Deepak-negi11 wants to merge 2 commits into
CCExtractor:masterfrom
Deepak-negi11:test/cli-option-coverage

Conversation

@Deepak-negi11

@Deepak-negi11 Deepak-negi11 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Repro instructions:

This PR only adds a check, so there's no bug to reproduce. Here's how to see what it catches.

Run it. It passes:

cd src/rust && cargo test --lib test_every_option_has_a_case

Then try one of these changes and run it again:

Change What you get
Remove the --ucla entry from cases() "These options have no case in cases(), please add one: ["ucla"]"
Add an entry for --quiet (still on the list) "These options have a case now, remove them from UNTESTED_OPTIONS: ["quiet"]"
Add "no-such-option" to the list "These are not options, remove them from UNTESTED_OPTIONS: ["no-such-option"]"
Write "--myth" in a comment or in a parser.rs test still passes, because only entries in cases() count

Part of #2408 (step 2 of 4). This branch includes #2409, so please look at #2409 first.

What's wrong

Nothing stops someone from adding or changing an option without a test that checks what C receives.

What this adds

  • test_every_option_has_a_case: takes every option from clap. For each entry in cases(), it asks clap which options that entry's arguments set, after the old spellings are converted the same way ccxr_parse_parameters does it. An option counts as tested only if an entry sets it, and test_cases_reach_c runs that entry and checks the C side.
  • UNTESTED_OPTIONS: the 158 options that have no entry yet (159 in builds with the with_libcurl feature). The test fails if a new option has no entry and isn't on the list, if an option on the list now has an entry, or if a name on the list isn't an option. So the list can only get shorter.
  • Entries for --stream: --stream 5 and -s 5 give a C timeout of 5 seconds, and --stream and -s with no number give -1 (no timeout). The tests from [FIX] Accept --stream with one input file and without a timeout value #2415 stop at the Rust options, so they don't count.

The list is long because only real checks on the C side count now. Before, any test with "--name" in it counted, including parser.rs tests that never reach C. The next PRs shorten it: each new entry removes one line.

Testing

  • cargo test (440 passed), cargo fmt --check, cargo clippy -- -D warnings.
  • The changes in the table above.

Copilot AI balanced review requested due to automatic review settings October 20, 2026 19:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cfsmp3 cfsmp3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is Carlos' claude doing the first pass. Before Carlos takes a look, please address these issues.

We want the gate, but it has to be harder to satisfy by accident.

  1. It fails on current master. #2415 added --stream tests, so after merging this branch with master, cargo test --lib test_every_option_has_a_test fails:

    These options have a test now, remove them from UNTESTED_OPTIONS: ["stream"]
    

    Rebase and remove stream from the list.

  2. "Has a test" is a text search. is_used_in_tests counts an option as tested if the string "--opt" or "--opt= appears anywhere in the searched sources. That includes a comment, an unrelated assertion, or a parser.rs test that stops at the Rust Options, which is exactly the gap this stack says it closes. Please make the gate check something that runs. For example, use one table with an entry per option (arguments plus a check on the resulting ccx_s_options / ccx_s_teletext_config). One test iterates over the table, running each entry through parse_to_c, and the gate compares the table's options against Args::command(). An option then counts as tested only when a check on the C side actually ran for it.

@ccextractor-bot

Copy link
Copy Markdown
Collaborator
CCExtractor CI platform finished running the test files on linux. 172/238 tests matched the approved output:
Report Name Tests Passed
Broken 9/13
CEA-708 2/14
DVB 0/7
DVD 3/3
DVR-MS 2/2
General 26/28
Hardsubx 1/1
Hauppage 3/3
MP4 3/3
NoCC 10/10
Options 69/86
Teletext 0/21
WTV 12/13
XDS 32/34

66 tests do not match the approved output. That is the pass/fail verdict. Whether this branch caused it is a separate question, answered below.


Compared with the tip of master — test 9984, commit 24d1e0a:

  • 0 pass there and fail here
  • 0 fail there and pass here
  • 0 fail on both, with different output
  • 66 fail on both, byte for byte the same

Compared with the commit this branch was cut from: the same run as the tip of master (test 9984), so the comparison above already covers it.


No test changes behaviour relative to the tip of master: every failure above fails there too, byte for byte. The approved output for those tests is out of date, which is a baseline to review rather than a regression in this branch.

@ccextractor-bot

Copy link
Copy Markdown
Collaborator
CCExtractor CI platform finished running the test files on windows. 172/238 tests matched the approved output:
Report Name Tests Passed
Broken 9/13
CEA-708 2/14
DVB 0/7
DVD 3/3
DVR-MS 2/2
General 26/28
Hardsubx 1/1
Hauppage 3/3
MP4 3/3
NoCC 10/10
Options 69/86
Teletext 0/21
WTV 12/13
XDS 32/34

66 tests do not match the approved output. That is the pass/fail verdict. Whether this branch caused it is a separate question, answered below.


Compared with the tip of master — test 9985, commit 24d1e0a:

  • 0 pass there and fail here
  • 0 fail there and pass here
  • 0 fail on both, with different output
  • 66 fail on both, byte for byte the same

Compared with the commit this branch was cut from: the same run as the tip of master (test 9985), so the comparison above already covers it.


No test changes behaviour relative to the tip of master: every failure above fails there too, byte for byte. The approved output for those tests is out of date, which is a baseline to review rather than a regression in this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants