[py] run the BiDi tests against the Grid server - #17980
Open
AutomatedTester wants to merge 6 commits into
Open
Conversation
Add test-<browser>-remote-bidi targets for chrome and firefox, combining the BiDi test sources with the grid data/env wiring the other remote suites use. BIDI_TESTS was excluded from every remote suite glob, so the Python BiDi tests had never run through Grid's websocket proxy. No client change was needed: --remote and --bidi are independent code paths, LocalNode already rewrites webSocketUrl to /session/<id>/se/bidi, and ProxyNodeWebsockets tunnels it to the node. Kept out of the test-<browser>-remote aggregate for now, since this is new coverage. No xfail_remote markers were added: the suite passes except for two tests that fail identically without --remote (a headless-Chrome document.hasFocus() artifact, and a pre-existing firefox wheel-scroll timing flake that fails 3/3 runs locally).
Add a test-remote-bidi aggregate test_suite alongside test-remote, and a separate remote-bidi-tests job in ci-python.yml that queries it. Keeping it as its own job means a red BiDi run cannot redden the established remote job while this coverage is new. Document the remote targets and the --remote / --bidi --remote pytest flags in py/TESTING.md, including the Grid jar prerequisite.
The bidi and per-browser suites pass BROWSERS[browser]["target_compatible_with"] through; the remote ones did not. It is a no-op today because the value is empty for chrome and firefox, but the divergence would silently drop the platform constraint the first time a remote suite is added for a browser that has one (ie, safari, webkitgtk).
The "skip unless --remote" guard was written out three times (driver, firefox_options, chromium_options) and the xfail_<driver>/xfail_remote marker dance twice (driver, clean_driver). Extract _skip_unless_remote and _apply_xfail_markers. The two marker blocks had drifted apart, so unifying them fixes two latent bugs in the clean_driver path: - it did not support the condition kwarg, and passed it straight to pytest.xfail(), which does not accept it (TypeError) - neither block popped run before calling pytest.xfail(), so an explicit run=True raised TypeError as well The `yield`/`return` after pytest.skip() in both blocks was dead code; pytest.skip() raises. Verified against alerts_tests, which exercises all three marker paths (condition, run=False, plain xfail): 20 passed, 1 skipped, 2 xfailed both before and after.
The remote suites set tags = ["no-sandbox", "remote", "<browser>-remote"],
omitting BROWSERS[browser]["tags"] (COMMON_TAGS + the bare browser name) that
every non-remote suite carries. Compose from the sibling expression instead and
keep the two remote markers on top.
browser-test and requires-network are inert here - nothing in .bazelrc*,
.github/, scripts/ or the Rakefile filters on either, and requires-network is
moot anyway because no-sandbox already disables the sandbox. The tag that
matters is the bare browser name: py/TESTING.md documents
bazel test //py/... --test_tag_filters=chrome
and that silently matched no remote target. It now matches all 26 per-browser
remote-bidi targets. Ruby already does this and says why in a comment
(rb/spec/tests.bzl), so this aligns Python with the reference implementation.
The omission was incidental rather than deliberate: the per-browser remote
targets in b1904f8 (#16851) inherited the minimal tag set of the single
test-remote target they replaced, and no rationale for the tag choice is
recorded.
Both tests read an asynchronous result exactly once, so they raced it. test_activate_browsing_context asserted that creating a second window takes focus off the first. Focus is handed over asynchronously, so poll for it. That alone is not enough: headless Chromium has no window manager and never hands focus over at all, confirmed by a 5s wait timing out 5/5 runs. So skip in that case only - gated on the new `headless` fixture, and only after the wait has actually expired. Headless Firefox does hand focus over and keeps the full assertions; headed runs still fail loudly if the handover regresses, because the skip cannot trigger outside headless. test_wheel_scroll_with_duration spreads a 100px scroll over 500ms and then asserted pageYOffset == 100 immediately; Firefox was reliably one pixel short. Wait for the scroll to settle before asserting, keeping the exact assertion rather than widening it to a tolerance. Verified per browser and per mode: - test_activate: chrome headless skips (was failing), chrome headed passes with full assertions, firefox headless passes with full assertions and 0 skips - test_wheel_scroll_with_duration: failed 3/3 runs at HEAD, 0/3 after Both tests exist twice (bidi/ and _bidi/); both copies are updated. Note: py/test/selenium/webdriver/common/_bidi/input_tests.py has further pre-existing firefox flakiness not addressed here - test_key_input_with_value_attribute, test_key_input_special_characters and test_combined_input_actions each failed in some runs, a different one each time, both before and after this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔗 Related Issues
Part of #17978 (the cross-binding half of that gap remains open).
Uncovered #17979 along the way.
💥 What does this PR do?
Runs the Python BiDi tests against Grid, which no binding did before.
BIDI_TESTSwas excluded from every remote suite glob, so there was notest-<browser>-remote-biditarget and the path where the client speaks BiDi over Grid's websocket proxy had never been exercised from Python. No client orconftest.pychange was needed —--remoteand--bidiare independent code paths that already compose,LocalNodealready rewriteswebSocketUrlto/session/<id>/se/bidi, andProxyNodeWebsocketsalready tunnels it. The missing piece was purely the Bazel target.Result across all 26 BiDi targets, chrome and firefox: zero remote-specific failures. Every failure observed reproduced identically without
--remote, so noxfail_remotemarkers were added — marking them would have mislabelled a headless/timing problem as a Grid problem. The two genuine ones are fixed instead:test_activate_browsing_contextasserted a focus handover that happens asynchronously, read exactly once. Now polled. A plain wait is not sufficient on its own: headless Chromium has no window manager and never hands focus over at all (a 5s wait timed out 5/5), while headless Firefox does. So it skips only when the wait expires and the run is headless — headless Firefox keeps the full assertions, and headed runs still fail loudly on a real regression because the skip cannot fire outside headless.test_wheel_scroll_with_durationspread a 100px scroll over 500ms then assertedpageYOffset == 100immediately; Firefox was reliably one pixel short. Now waits for the scroll to settle, keeping the exact assertion rather than widening it to a tolerance. Failed 3/3 runs before, 0/3 after.Also included, all pre-existing inconsistencies found while building the above:
tags = ["no-sandbox", "remote", "<browser>-remote"], omitting theCOMMON_TAGS+ bare browser name every non-remote suite has. The tag that mattered is the browser name:py/TESTING.mddocuments--test_tag_filters=chrome, and that matched no remote target. Ruby already does this and says why in a comment.target_compatible_with, which the bidi and per-browser suites pass through.conftest.pyrepeated the "skip unless--remote" guard three times and thexfail_<driver>/xfail_remotemarker dance twice.🔧 Implementation Notes
The new suite is deliberately not folded into the
test-<browser>-remoteaggregate, and gets its ownremote-bidi-testsCI job, so a red run in new coverage cannot redden the established remote job. Easy to merge in later once it has proven itself.The two duplicated marker blocks in
conftest.pyhad drifted apart, so unifying them fixes two latent bugs on theclean_driverpath: it did not support theconditionkwarg and passed it straight intopytest.xfail(), which rejects it (TypeError); and neither block poppedrunbefore callingpytest.xfail(), so a documentedrun=TrueraisedTypeErrortoo. Theyield/returnafterpytest.skip()in both was dead code, sincepytest.skip()raises. Verified behaviour-identical againstalerts_tests, which exercises all three marker paths (it carriescondition=sys.platform == "darwin", run=Falseplus plain xfails): 20 passed, 1 skipped, 2 xfailed, before and after.The
COMMON_TAGSomission was incidental rather than deliberate — the per-browser remote targets in #16851 inherited the minimal tag set of the singletest-remotetarget they replaced, and no rationale for the choice is recorded. One behaviour change to be aware of:--test_tag_filters=chromenow also matches remote targets, which is slower but is what the docs imply should happen.skip-rbewas considered and deliberately not added. These targets are selected by the RBE run (.skipped-testsis empty and-skip-rbeis the only filter), butCI - RBEon trunk is green and Ruby has run remote tests on RBE since #13906, so the Grid-on-RBE path works.Each concern is a separate commit if that is easier to review.
🤖 AI assistance
conftest.pyhelpers, the two test fixes, and the doc updates, plus the verification runs behind the claims above💡 Additional Considerations
Verified on macOS arm64 with pinned browsers, headed and headless, chrome and firefox. The new
remote-bidi-testsjob has not yet run on ubuntu/Xvfb, so its first CI run is the real test of the headless reasoning above — CI is headed under fluxbox, where the chrome skip should not fire and the full assertions should run.Follow-ups, not in this PR:
_start_bidipasses only(url, timeout, interval)toWebSocketConnection, droppingClientConfig'susername/password/auth_type,extra_headersandca_certs(wss://hardcodescert_reqs: CERT_NONE) pluswebsocket_max_message_size. Not reachable through the local standalone used here (plain http, no auth), so it is out of scope, but it would break an authenticated or TLS Grid. It wants to stack on [py]: Apply ClientConfiguser_agentandextra_headersper connection #17944, which reworks how per-connection headers are resolved.🔄 Types of changes
🤖 Generated with Claude Code
Co-Authored-By: Copse noreply@copse.dev
Copse-Models: acp:claude-agent-acp#opus[1m]