input-sync.md

Native synchronization v1

The synchronization implementation is maintained in patches 0149–0162 listed in patches/series. It is an experimental Windows implementation; source checks and SDK tests do not prove that a Chromium build or real input replay succeeds. Use a matching rebuilt binary for the manual checks below. Previously published releases lack this module.

Windows CI checks that its thread is attached to the active input desktop before compilation (INPUT_SYNC_CHECK_DESKTOP=1). Final-bundle acceptance repeats this check and requires actual capture/replay to succeed; a locked or noninteractive desktop fails explicitly instead of silently skipping native tests.

The first version supports one headed browser window per process: mouse movement, clicks, wheel, drag, keyboard events and committed text. It reproduces window input; it does not align page contents or create corresponding tabs. Keep window geometry, tab order and initial page state aligned. Separate popup/menu windows, system dialogs, multi-window mapping, semantic tab lifecycle, touch and cross-platform replay are not implemented. IME composition UI and clipboard contents are not transferred; committed text has a dedicated path. These inputs require manual verification on the rebuilt browser.

Use functional, brand-neutral names for native directories, namespaces, protocol identifiers and runtime messages. The browser adapter namespace is input_sync; launch switches and the pipe prefix use uxr-.

Ownership and layout

The paths below refer to the Chromium source tree after applying the patches.

• chrome/browser/ui/input_sync/sync_service.cc: session validation, bounded capture queue, sequence checks, role, lease and window lifecycle.
• input_adapter.*: every Aura / Views input conversion and replay dependency.
• pipe_server.*: Windows transport, frame limits and current-user access control.
• ui/base/ime/text_commit_observer.*: a UI-sequence callback registry without browser, protocol, role or transport dependencies.

BrowserView registers its Widget once; the adapter observes destruction itself. The process starts the pipe before window creation and stops it before browser shutdown. Each registered Widget has an adapter entry in the service. Closing the last window clears the session but preserves the connection while the process is alive; it does not keep an otherwise exiting browser alive. There are no replaceable global function pointers, eight-bit window indexes or fallback to whichever browser happens to be active. Adding or losing a window invalidates the current session. The SDK must explicitly start a new one.

Chromium integration boundary

Existing file	Contract
chrome/browser/chrome_browser_main.cc	Start on the UI sequence before window creation; stop before browser shutdown.
chrome/browser/ui/views/frame/browser_view.cc	Register after ClientView::AddedToWidget, only on Windows. Registration is idempotent.
ui/views/controls/textfield/textfield.cc	Notify once after accepting filtered committed text into the model.
content/browser/renderer_host/render_widget_host_view_aura.cc	Notify once after forwarding a committed string to the active renderer widget.
chrome/browser/ui/BUILD.gn	Compile browser adapters and transport only for Windows.
ui/base/ime/BUILD.gn	Compile the dependency-neutral text notification registry.

Do not put protocol handling or synchronization decisions in these hooks. Their placement, accepted-text semantics, observer lifetime and event coverage must be checked on every Chromium upgrade, even when a patch applies without conflicts.

The patch series is the sole maintained source of the native implementation. Patches 0149–0156 add the module files; 0157–0162 contain the six integration changes above. Follow the existing one-patch-per-file convention: apply the series to the pinned Chromium tree, edit the affected files there, and update their corresponding patches. Keep native behavior in chrome/browser/ui/input_sync/ and leave only narrow hooks in the existing Chromium files.

Run python tools/check_patches.py and verify that the updated patches apply to the pinned source. The normal patch pipeline and existing patch hash include these files; no separate source copy, mapping manifest or patch generator is needed.

Protocol

Explicit launch switch: --uxr-sync-pipe=<random endpoint id>. With no switch, window registration is a no-op. The pipe is \\.\pipe\uxr-sync-<id>, accepts one local client and grants access only to the current Windows user. Endpoint IDs are opaque identifiers, not remotely usable credentials. Browser lifetime remains independent of SDK lifetime.

Frames are a four-byte little-endian length followed by UTF-8 JSON, at most 1 MiB. Requests and responses are serialized per connection; unsolicited frames are not allowed. The commands are:

• hello: protocol version: 1, capabilities, and current windows count. Available even when there is no replay target; begin still requires one window.
• begin: session, fresh epoch, role (capture or replay), initial seq.
• poll: up to eight operations; only valid in capture mode.
• apply: packet: {version, session, epoch, seq, operation}; replies with seq.
• ping: keeps an active session alive without changing its sequence.
• stop: releases held replay input and clears the session; idempotent.

Errors are { "error": "message" }. Sequence numbers are positive signed 32-bit integers. Replay accepts the next sequence, acknowledges old sequences without executing again, and rejects gaps and mismatched sessions/epochs. An acknowledgement means the adapter dispatched the operation, not that a website completed a task.

Operation examples live in each language's existing test file: sdk/node/test/sync.test.mjs and sdk/python/tests/test_sync.py. Coordinates cover the registered Aura window in normalized device-independent coordinates, including browser UI.

The following values are defined by protocol v1, not Chromium enum numbers:

• modifiers: a unique list from shift, control, alt, meta, altGraph, capsLock, numLock, scrollLock. Unknown or duplicate names are rejected.
• Mouse buttons: a unique list of held left, right, middle, back, forward buttons. button is the changed button, or none for movement/wheel. Click operations also carry clicks (1–3); wheel dx/dy are integer deltas in Chromium wheel units (120 per Windows wheel detent), not CSS pixels.
• Keyboard key is the UI Events logical key string, and code is the UI Events physical code string. shortcut uses the same code names to represent the equivalent US-layout shortcut key, preserving shortcut meaning on non-US layouts. An empty code/shortcut means no mapping. repeat and char are booleans; character is a UTF-16 code unit for character events.
• text carries UTF-8 committed text and cursorBefore determines cursor placement.

Only input_adapter.cc translates these fields to/from native flags and key values. The SDK forwards the operation object unchanged. New native-only flags must not silently become wire fields. Changes to these meanings require a protocol version change; v1 is still unpublished.

The native queue is limited to 512 operations, with 64 KiB of encoded JSON per operation. Overflow stops capture. SDK follower queues are independent, limited to 128 pending operations and reject operations queued for more than two seconds. A five-second lease stops a silent connection; the SDK keeps idle sessions alive. No historical operations are replayed automatically after a disconnect.

Verification on a matching Windows build

The existing Windows final-bundle check runs the native case in sdk/python/tests/test_sync.py against the freshly extracted chrome.exe, after installing the existing fingerprint acceptance dependencies. Failure blocks the final bundle. The normal SDK tests skip that case when no binary is specified. To run it locally on an unlocked Windows desktop with Python Playwright installed:

$env:INPUT_SYNC_BROWSER = 'C:\path\to\rebuilt\chrome.exe'
python -X utf8 -m unittest discover -s sdk/python/tests -p test_sync.py -v

The native case checks connection before the first window, Windows mouse/key capture, replay into a real page, duplicate sequence acknowledgement, renderer text commits, held-key release, invalid modifiers, multiple-window rejection, last-window closure, reopening through the same connection, and process shutdown. It uses Win32 SendInput for native capture and CDP only for setup/observation and triggering a renderer text commit. It requires an interactive desktop with focus available to the test browser; failure to obtain focus fails the test.

This does not automate a real OS IME composition session, address-bar text commits, non-US keyboard layouts, drag/wheel semantics or system dialogs. Those still need the following manual checks; a passing automated case is not evidence for them.

1. Run the Node or Python example with the rebuilt chrome.exe; open one aligned page in each process and start the group.
2. Verify click, double click, wheel, drag, modifiers and normal typing both in page inputs and the address bar. Verify repeated and non-US characters.
3. Verify IME text is inserted once in each location. Check Enter, Backspace and shortcuts after composition. Menus and system dialogs are outside v1 scope.
4. Pause while dragging or holding a modifier: followers must release input.
5. Stop the SDK during input: followers release within the lease, browsers stay open.
6. Rapidly pause/resume and switch leader: old events must not leak into the new epoch.
7. Create a second window or destroy the target: synchronization must stop with an error rather than send input to a different window.

Compile and real-browser verification are required before declaring this feature ready or publishing a binary that advertises synchronization support.