Add a new option --fallback-sampling-target so users can specify different ratio with --target#1352
Merged
Merged
Conversation
…erent ratio with --target
Without this, mypy cannot prove that execution stops after the call, causing a false positive on the unconditional float(self.fallback_sampling_target) in _fallback_result.
Mypy cannot narrow instance attributes across method calls, so the assert gives _fallback_result the local type narrowing it needs.
Konboi
reviewed
Jul 7, 2026
Konboi
left a comment
Konboi
left a comment
Contributor
There was a problem hiding this comment.
I left comments however not to block your task, I'll approve this PR
| @@ -1,5 +1,6 @@ | |||
| # TODO: add cli-specific custom exceptions | |||
| import sys | |||
| from typing import NoReturn | |||
| "(no duration estimates are available in this path).", | ||
| )] = FallbackMode.RUN_ALL, | ||
| fallback_sampling_target: Annotated[Percentage | None, typer.Option( | ||
| "--fallback-sampling-target", |
Contributor
There was a problem hiding this comment.
The --random-sample option is a hidden option however, this --fallback-sampling-target option isn't a hidden option.
Is it intentionally?
Konboi
approved these changes
Jul 7, 2026
Konboi
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I noticed that --fallback-mode=random-sample can be used without --target option technically. In that case, we don’t know the how many (ratio) we choose tests from all tests. Relevant code is subset.py
To overcome the situation, I considered several options below:
Option1: Add a new option --fallback-sampling-target so users can specify different ratio with --target
Option2: Raise error when --target is not specified with random-sample -> --fallback-mode=random-sample must use with --target
Option3: Return 50% of tests as subset when --target is not specified
We chose the option 1 to specify the target of random sampling percentage when using fallback mode = random-sample