Skip to content

[py] run the BiDi tests against the Grid server - #17980

Open
AutomatedTester wants to merge 6 commits into
trunkfrom
copse/i-want-to-run-some-of-the-python-bidi-test-e24877
Open

[py] run the BiDi tests against the Grid server#17980
AutomatedTester wants to merge 6 commits into
trunkfrom
copse/i-want-to-run-some-of-the-python-bidi-test-e24877

Conversation

@AutomatedTester

Copy link
Copy Markdown
Member

🔗 Related Issues

Part of #17978 (the cross-binding half of that gap remains open).
Uncovered #17979 along the way.

💥 What does this PR do?

Runs the Python BiDi tests against Grid, which no binding did before.

BIDI_TESTS was excluded from every remote suite glob, so there was no test-<browser>-remote-bidi target and the path where the client speaks BiDi over Grid's websocket proxy had never been exercised from Python. No client or conftest.py change was needed--remote and --bidi are independent code paths that already compose, LocalNode already rewrites webSocketUrl to /session/<id>/se/bidi, and ProxyNodeWebsockets already tunnels it. The missing piece was purely the Bazel target.

Result across all 26 BiDi targets, chrome and firefox: zero remote-specific failures. Every failure observed reproduced identically without --remote, so no xfail_remote markers were added — marking them would have mislabelled a headless/timing problem as a Grid problem. The two genuine ones are fixed instead:

  • test_activate_browsing_context asserted a focus handover that happens asynchronously, read exactly once. Now polled. A plain wait is not sufficient on its own: headless Chromium has no window manager and never hands focus over at all (a 5s wait timed out 5/5), while headless Firefox does. So it skips only when the wait expires and the run is headless — headless Firefox keeps the full assertions, and headed runs still fail loudly on a real regression because the skip cannot fire outside headless.
  • test_wheel_scroll_with_duration spread a 100px scroll over 500ms then asserted pageYOffset == 100 immediately; Firefox was reliably one pixel short. Now waits for the scroll to settle, keeping the exact assertion rather than widening it to a tolerance. Failed 3/3 runs before, 0/3 after.

Also included, all pre-existing inconsistencies found while building the above:

  • Remote suites carried tags = ["no-sandbox", "remote", "<browser>-remote"], omitting the COMMON_TAGS + bare browser name every non-remote suite has. The tag that mattered is the browser name: py/TESTING.md documents --test_tag_filters=chrome, and that matched no remote target. Ruby already does this and says why in a comment.
  • Remote suites omitted target_compatible_with, which the bidi and per-browser suites pass through.
  • conftest.py repeated the "skip unless --remote" guard three times and the xfail_<driver>/xfail_remote marker dance twice.

🔧 Implementation Notes

The new suite is deliberately not folded into the test-<browser>-remote aggregate, and gets its own remote-bidi-tests CI job, so a red run in new coverage cannot redden the established remote job. Easy to merge in later once it has proven itself.

The two duplicated marker blocks in conftest.py had drifted apart, so unifying them fixes two latent bugs on the clean_driver path: it did not support the condition kwarg and passed it straight into pytest.xfail(), which rejects it (TypeError); and neither block popped run before calling pytest.xfail(), so a documented run=True raised TypeError too. The yield/return after pytest.skip() in both was dead code, since pytest.skip() raises. Verified behaviour-identical against alerts_tests, which exercises all three marker paths (it carries condition=sys.platform == "darwin", run=False plus plain xfails): 20 passed, 1 skipped, 2 xfailed, before and after.

The COMMON_TAGS omission was incidental rather than deliberate — the per-browser remote targets in #16851 inherited the minimal tag set of the single test-remote target they replaced, and no rationale for the choice is recorded. One behaviour change to be aware of: --test_tag_filters=chrome now also matches remote targets, which is slower but is what the docs imply should happen.

skip-rbe was considered and deliberately not added. These targets are selected by the RBE run (.skipped-tests is empty and -skip-rbe is the only filter), but CI - RBE on trunk is green and Ruby has run remote tests on RBE since #13906, so the Grid-on-RBE path works.

Each concern is a separate commit if that is easier to review.

🤖 AI assistance

  • No substantial AI assistance used
  • AI assisted (complete below)
    • Tool(s): Claude Code
    • What was generated: the Bazel targets, the CI job, the conftest.py helpers, the two test fixes, and the doc updates, plus the verification runs behind the claims above
    • I reviewed all AI output and can explain the change

💡 Additional Considerations

Verified on macOS arm64 with pinned browsers, headed and headless, chrome and firefox. The new remote-bidi-tests job has not yet run on ubuntu/Xvfb, so its first CI run is the real test of the headless reasoning above — CI is headed under fluxbox, where the chrome skip should not fire and the full assertions should run.

Follow-ups, not in this PR:

🔄 Types of changes

  • Cleanup (formatting, renaming)
  • Bug fix (backwards compatible)
  • New feature (non-breaking change which adds functionality and tests!)

🤖 Generated with Claude Code

Co-Authored-By: Copse noreply@copse.dev
Copse-Models: acp:claude-agent-acp#opus[1m]

Add test-<browser>-remote-bidi targets for chrome and firefox, combining the
BiDi test sources with the grid data/env wiring the other remote suites use.
BIDI_TESTS was excluded from every remote suite glob, so the Python BiDi tests
had never run through Grid's websocket proxy.

No client change was needed: --remote and --bidi are independent code paths,
LocalNode already rewrites webSocketUrl to /session/<id>/se/bidi, and
ProxyNodeWebsockets tunnels it to the node.

Kept out of the test-<browser>-remote aggregate for now, since this is new
coverage. No xfail_remote markers were added: the suite passes except for two
tests that fail identically without --remote (a headless-Chrome
document.hasFocus() artifact, and a pre-existing firefox wheel-scroll timing
flake that fails 3/3 runs locally).
Add a test-remote-bidi aggregate test_suite alongside test-remote, and a
separate remote-bidi-tests job in ci-python.yml that queries it. Keeping it as
its own job means a red BiDi run cannot redden the established remote job while
this coverage is new.

Document the remote targets and the --remote / --bidi --remote pytest flags in
py/TESTING.md, including the Grid jar prerequisite.
The bidi and per-browser suites pass BROWSERS[browser]["target_compatible_with"]
through; the remote ones did not. It is a no-op today because the value is empty
for chrome and firefox, but the divergence would silently drop the platform
constraint the first time a remote suite is added for a browser that has one
(ie, safari, webkitgtk).
The "skip unless --remote" guard was written out three times (driver,
firefox_options, chromium_options) and the xfail_<driver>/xfail_remote marker
dance twice (driver, clean_driver). Extract _skip_unless_remote and
_apply_xfail_markers.

The two marker blocks had drifted apart, so unifying them fixes two latent bugs
in the clean_driver path:

- it did not support the condition kwarg, and passed it straight to
  pytest.xfail(), which does not accept it (TypeError)
- neither block popped run before calling pytest.xfail(), so an explicit
  run=True raised TypeError as well

The `yield`/`return` after pytest.skip() in both blocks was dead code;
pytest.skip() raises.

Verified against alerts_tests, which exercises all three marker paths
(condition, run=False, plain xfail): 20 passed, 1 skipped, 2 xfailed both
before and after.
The remote suites set tags = ["no-sandbox", "remote", "<browser>-remote"],
omitting BROWSERS[browser]["tags"] (COMMON_TAGS + the bare browser name) that
every non-remote suite carries. Compose from the sibling expression instead and
keep the two remote markers on top.

browser-test and requires-network are inert here - nothing in .bazelrc*,
.github/, scripts/ or the Rakefile filters on either, and requires-network is
moot anyway because no-sandbox already disables the sandbox. The tag that
matters is the bare browser name: py/TESTING.md documents

    bazel test //py/... --test_tag_filters=chrome

and that silently matched no remote target. It now matches all 26 per-browser
remote-bidi targets. Ruby already does this and says why in a comment
(rb/spec/tests.bzl), so this aligns Python with the reference implementation.

The omission was incidental rather than deliberate: the per-browser remote
targets in b1904f8 (#16851) inherited the minimal tag set of the single
test-remote target they replaced, and no rationale for the tag choice is
recorded.
Both tests read an asynchronous result exactly once, so they raced it.

test_activate_browsing_context asserted that creating a second window takes
focus off the first. Focus is handed over asynchronously, so poll for it. That
alone is not enough: headless Chromium has no window manager and never hands
focus over at all, confirmed by a 5s wait timing out 5/5 runs. So skip in that
case only - gated on the new `headless` fixture, and only after the wait has
actually expired. Headless Firefox does hand focus over and keeps the full
assertions; headed runs still fail loudly if the handover regresses, because the
skip cannot trigger outside headless.

test_wheel_scroll_with_duration spreads a 100px scroll over 500ms and then
asserted pageYOffset == 100 immediately; Firefox was reliably one pixel short.
Wait for the scroll to settle before asserting, keeping the exact assertion
rather than widening it to a tolerance.

Verified per browser and per mode:
- test_activate: chrome headless skips (was failing), chrome headed passes with
  full assertions, firefox headless passes with full assertions and 0 skips
- test_wheel_scroll_with_duration: failed 3/3 runs at HEAD, 0/3 after

Both tests exist twice (bidi/ and _bidi/); both copies are updated.

Note: py/test/selenium/webdriver/common/_bidi/input_tests.py has further
pre-existing firefox flakiness not addressed here - test_key_input_with_value_attribute,
test_key_input_special_characters and test_combined_input_actions each failed in
some runs, a different one each time, both before and after this change.
@selenium-ci selenium-ci added C-py Python Bindings B-build Includes scripting, bazel and CI integrations B-devtools Includes everything BiDi or Chrome DevTools related labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

B-build Includes scripting, bazel and CI integrations B-devtools Includes everything BiDi or Chrome DevTools related C-py Python Bindings

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants