Repository navigation
[IMPROVEMENT] Fail the tests when a command line option has no test - #2410
Deepak-negi11 wants to merge 2 commits into
Conversation
c2111df to
2202b9b
Compare
cfsmp3
left a comment
There was a problem hiding this comment.
This is Carlos' claude doing the first pass. Before Carlos takes a look, please address these issues.
We want the gate, but it has to be harder to satisfy by accident.
-
It fails on current master. #2415 added
--streamtests, so after merging this branch with master,cargo test --lib test_every_option_has_a_testfails:These options have a test now, remove them from UNTESTED_OPTIONS: ["stream"]Rebase and remove
streamfrom the list. -
"Has a test" is a text search.
is_used_in_testscounts an option as tested if the string"--opt"or"--opt=appears anywhere in the searched sources. That includes a comment, an unrelated assertion, or aparser.rstest that stops at the RustOptions, which is exactly the gap this stack says it closes. Please make the gate check something that runs. For example, use one table with an entry per option (arguments plus a check on the resultingccx_s_options/ccx_s_teletext_config). One test iterates over the table, running each entry throughparse_to_c, and the gate compares the table's options againstArgs::command(). An option then counts as tested only when a check on the C side actually ran for it.
2202b9b to
9758c69
Compare
CCExtractor CI platform finished running the test files on linux. 172/238 tests matched the approved output:
66 tests do not match the approved output. That is the pass/fail verdict. Whether this branch caused it is a separate question, answered below.
Compared with the tip of master — test 9984, commit 24d1e0a:
Compared with the commit this branch was cut from: the same run as the tip of master (test 9984), so the comparison above already covers it. No test changes behaviour relative to the tip of master: every failure above fails there too, byte for byte. The approved output for those tests is out of date, which is a baseline to review rather than a regression in this branch. |
CCExtractor CI platform finished running the test files on windows. 172/238 tests matched the approved output:
66 tests do not match the approved output. That is the pass/fail verdict. Whether this branch caused it is a separate question, answered below.
Compared with the tip of master — test 9985, commit 24d1e0a:
Compared with the commit this branch was cut from: the same run as the tip of master (test 9985), so the comparison above already covers it. No test changes behaviour relative to the tip of master: every failure above fails there too, byte for byte. The approved output for those tests is out of date, which is a baseline to review rather than a regression in this branch. |
Repro instructions:
This PR only adds a check, so there's no bug to reproduce. Here's how to see what it catches.
Run it. It passes:
Then try one of these changes and run it again:
--uclaentry fromcases()--quiet(still on the list)"no-such-option"to the list"--myth"in a comment or in a parser.rs testcases()countPart of #2408 (step 2 of 4). This branch includes #2409, so please look at #2409 first.
What's wrong
Nothing stops someone from adding or changing an option without a test that checks what C receives.
What this adds
test_every_option_has_a_case: takes every option from clap. For each entry incases(), it asks clap which options that entry's arguments set, after the old spellings are converted the same wayccxr_parse_parametersdoes it. An option counts as tested only if an entry sets it, andtest_cases_reach_cruns that entry and checks the C side.UNTESTED_OPTIONS: the 158 options that have no entry yet (159 in builds with thewith_libcurlfeature). The test fails if a new option has no entry and isn't on the list, if an option on the list now has an entry, or if a name on the list isn't an option. So the list can only get shorter.--stream:--stream 5and-s 5give a C timeout of 5 seconds, and--streamand-swith no number give -1 (no timeout). The tests from [FIX] Accept --stream with one input file and without a timeout value #2415 stop at the Rust options, so they don't count.The list is long because only real checks on the C side count now. Before, any test with
"--name"in it counted, including parser.rs tests that never reach C. The next PRs shorten it: each new entry removes one line.Testing
cargo test(440 passed),cargo fmt --check,cargo clippy -- -D warnings.