Capture video and camera components, add accessibility mode, modernize for RN 0.76+ - #4
Merged
Merged
Conversation
…dernize for RN 0.76+ Addresses #2 without MediaProjection. Capture rework - Android: copy each SurfaceView, install the result as a ViewOverlay on that same SurfaceView, then do a single window PixelCopy. The overlay is painted by the view system into the window surface, so z-order, clipping and transforms come out right with no manual occlusion maths. Replaces the old paste-back approach, which ignored z-order and ran the copies serially with a 5s latch each. - iOS: discover AVFoundation objects by runtime introspection through public properties (AVCaptureVideoPreviewLayer.session, AVPlayerLayer.player, AVPlayerViewController.player), pull the current frame, inject it into the media component's own layer, and render the hierarchy once. Covers VisionCamera, react-native-video, expo-camera and expo-video with no cooperation from those packages and no app setup. Providers attach lazily and detach after ~3 idle seconds, so an app that never captures pays nothing. - Android: fix the SDK_INT >= 24 guard; PixelCopy's Window overload is API 26. Accessibility mode (Android 11+, opt-in) - Whole-display capture via AccessibilityService#takeScreenshot. - The service is deliberately NOT declared in the library manifest: merging would push an accessibility service onto every consumer app and drag them all into Play's Accessibility API policy review. App authors declare it themselves. React Native modernization - TurboModule with old-architecture fallback; TypeScript source built with builder-bob. - AGP 8, namespace, compileSdk 36, minSdk 24, react-android; drop unused zxing. - Podspec: iOS/tvOS 15.1, install_modules_dependencies, Obj-C++ module. - Screenshot detection uses Activity#registerScreenCaptureCallback on API 34+. - Remove the iOS status bar hack: it swizzled the private UIStatusBar class, which stopped existing in iOS 13, and was an App Store review risk. MediaProjection was evaluated and rejected: since Android 14 the consent Intent cannot be cached or reused, so consent cannot survive a process restart. See docs/IMPLEMENTATION_PLAN.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
The old `Example/` was a leftover `ReactNavigationTVDemo` scaffold: named after an unrelated
demo down to the Android applicationId and the iOS target folders, pinned to
react-native-tvos@0.64.2-0, and carrying react-navigation 5 / reanimated 2.2 / screens 3.4
plus two patches that this library never used. Its App.js also called the 1.x API. Upgrading
that in place would have been migration work for code with no bearing on this package.
`example/` is a fresh RN 0.81 app wired to the library through `link:..` plus a
react-native.config.js override, so autolinking picks the library up from the repo root and
Metro resolves it from TypeScript source.
It demonstrates the case that motivated the rework: a react-native-video player with
`useTextureView={false}`, forcing a SurfaceView on Android -- exactly what a plain window
readback cannot see. Plus the mode selector, the accessibility permission flow, screenshot
detection, warmUp/coolDown and dumpHierarchy.
Building it is also the strongest verification available without a device. Both pass:
- new architecture (the 0.81 default): codegen emits NativeScreenCaptureSpec into
com.lewin.capture plus the JNI artifacts, and the newarch source set compiles against it
- old architecture (-PnewArchEnabled=false): javap confirms ScreenCaptureSpec then extends
ReactContextBaseJavaModule with no codegen spec present, so the source-set switch really
works rather than reusing the other AAR
Doing this surfaced a real bug: the library pinned com.android.tools.build:gradle:8.7.3 in
its own buildscript while the app resolves AGP from a version catalog. Removed the pin so the
host app's classpath wins -- only a real Gradle build could have caught that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
…orms Everything that was checked by hand while building this branch now runs on every push, so it stays checked. | job | runner | guards | | --- | --- | --- | | library | ubuntu | tsc, bob build, codegen | | package | ubuntu | npm pack, assert tarball contents | | android (new/old arch) | ubuntu | build the example APK, assert the right spec compiled | | ios (new/old arch) | macos-15 | pod install + xcodebuild for the simulator | The iOS job matters most: the Objective-C on this branch has never been compiled anywhere, and that job is the first thing that will tell us whether it holds together. macOS minutes cost a multiple of Linux, but the two architectures take different paths through ScreenCapture.mm, so the matrix is worth it. The Android job does not stop at "Gradle succeeded". The newarch/oldarch source-set swap fails silently -- Gradle will happily build the wrong one -- so it asserts with javap that ScreenCaptureSpec extends the codegen spec on the new architecture and ReactContextBaseJavaModule on the old one, and that codegen actually emitted its Java spec. fix: stop publishing local codegen output to npm Writing the packaging job caught it immediately: `files` listed "ios", and the `!ios/build` negation did not cover `ios/generated`, so locally generated codegen artifacts were going into the tarball. Narrowed to `ios/ScreenCapture`, and the job now asserts on the contents both ways. Also restores the example's Gemfile, deleted earlier as if it were scaffolding noise. It is not -- it pins CocoaPods and xcodeproj away from versions known to break React Native builds, and CI goes through `bundle exec pod install` because of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
ruby/setup-ruby errors out when neither ruby-version nor a .ruby-version file is present, so both iOS jobs failed before xcodebuild ever ran. Pins 3.3 in the workflow and adds a .ruby-version so local `bundle install` matches CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
The repo had no LICENSE file at all, and its two declarations disagreed: package.json said ISC while the podspec said MIT. Worse, the 2.0.0 rework silently changed package.json from ISC to MIT and dropped LewinJun's copyright headers when the iOS files were rewritten -- relicensing forked code is not a change to make quietly. Settles it as MIT with both copyright holders, since one file is still upstream code verbatim: android/src/main/java/com/fugood/screencapture/ScreenCapturetListenManager.java, the pre-Android-14 MediaStore screenshot fallback. It now carries a header naming its origin. Renames com.lewin.capture -> com.fugood.screencapture. This reverses an earlier "cosmetic, defer it" call. The package name is a namespace, not attribution -- credit lives in LICENSE -- but it leaks into user code in exactly one place: app authors paste com.fugood.screencapture.ScreenCaptureAccessibilityService into their own AndroidManifest. That feature is new in 2.0.0, so nobody has it baked in yet. Renaming now costs nothing; renaming after release would be a real breaking change for every app using accessibility mode. The npm and pod names are unaffected. Also clears the remaining personal-name leakage: the "lewinScreen" log tag and the "lewin-screen-capture" iOS cache folder (old cached files simply orphan, they are cache). Verified: both example/android architectures build clean after the rename -- autolinking now imports com.fugood.screencapture.ScreenCapturePackage, and javap confirms the right spec superclass. The packaging job asserts LICENSE ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
The podspec claims tvOS support -- it always has, and the example this branch replaced was a react-native-tvos app -- but AVCaptureSession and AVCaptureVideoPreviewLayer are API_UNAVAILABLE(tvos). RNSCCameraFrameProvider would not have compiled there at all. CI would never have told us: the example has no tvOS target, so the iOS job only ever builds the iOS simulator. This would have shipped broken and invisible. Puts the camera provider behind #if !TARGET_OS_TV. Video capture still works on tvOS through AVPlayerLayer, which is available there; tvOS simply has no camera to capture. Adds a tvOS syntax check to the iOS job to stop this recurring. The five RNSC*.m sources depend only on UIKit/AVFoundation, so clang can parse them straight against the tvOS SDK without pods or a target. A real tvOS build would need a tvOS target in the example; noted as follow-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
…r both in CI Found by running the example on a real device, which is also where the library's core claim finally got tested: on a Pixel 7 Pro (Android 17 / API 37), new architecture, release build, the captured 1080x2340 image contains the actual video frame. The video region measures mean RGB (81, 96, 57) with 80.5% non-black pixels, against (28, 32, 41) and 0% for a control patch of app background. The SurfaceView content is composited in, not left as a black hole. Three things were broken on the way there: - metro.config.js did not work at all. builder-bob's monorepo helper requires yarn workspaces, but yarn 1 only permits workspaces in private projects and this repo root is the published package. Replaced with an explicit config: watch the root, map the package name onto it, block the root's own node_modules (otherwise Metro resolves a second copy of react/react-native through the symlink), and fall back to the example's node_modules for everything else -- including the @babel/runtime helpers Babel injects into the library's source. - The sample video URL was dead. Google's commondatastorage sample bucket now answers 403, which is why the player rendered black. Switched to test-videos.co.uk, verified reachable. - CI could not have caught either one. The Android job builds a *debug* APK, which loads JS from Metro and never bundles, so a broken Metro config sails straight through. Adds a `bundle` job that runs `react-native bundle` for both platforms, which is what actually exercises the resolution path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
…stricted-settings trap The example advertised an `accessibility` capture mode but never declared the service, so that button could only ever fail. The example is an app author opting in, which is exactly the setup the README describes, so it belongs here rather than in the library manifest. Verified end to end on a Pixel 7 Pro (Android 17 / API 37), with the device restored afterwards: - the platform grants the service capabilities=128 (CAPABILITY_CAN_TAKE_SCREENSHOT), confirming android:canTakeScreenshot="true" in the XML is doing its job - getPermissionStatus reports denied while off, granted once bound - capturing while off rejects cleanly instead of crashing - capturing while on returns a full 1080x2340 image that includes the real system status bar and navigation bar: 6.0% bright pixels in the top strip against 0.3% for the same strip of a view-mode capture. Whole display, not just the app window. Two platform behaviours cost real time to diagnose and are now in the README, because every developer using this mode will hit them: - Android blocks accessibility services for apps installed outside the Play Store, which covers anything adb-installed. The setting silently reverts. This looked precisely like a bug in getPermissionStatus reporting denied; it was not -- the service genuinely had been switched back off. Cleared with `appops set <pkg> ACCESS_RESTRICTED_SETTINGS allow`. - `am force-stop` on the host app makes the system disable the service, since a dying host process reads as the service failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
`takeScreenshot` was handed `service.getMainExecutor()`, so the full-screen hardware-to-software bitmap copy -- 1080x2340 on the device this was measured on -- ran on the UI thread. Moves it to a dedicated background executor. Also fixes a double-settle hazard in the same callback: `callback.onResult(...)` sat inside the try block that guards the buffer work, so anything throwing downstream would settle the same Promise a second time from the catch. The callback is now invoked exactly once, after the buffer is closed. Measured on a Pixel 7 Pro, end to end from the JS call to the resolved promise: PNG full resolution, view mode 513 / 451 / 471 ms PNG full resolution, accessibility mode 479 / 478 / 497 ms JPEG q80 scale 0.5, view mode 63 / 50 / 69 ms The two modes now land within noise of each other, and the real cost is the encoder: a full-resolution PNG is roughly 8x everything else combined. Documented in the README, since "which mode is faster" is the wrong question to be asking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
…ro warning Verified on the Mac (Xcode 26.6, iPhone 17 Pro simulator) that the iOS pipeline works as far as a simulator can show: m=1/86 | capture OK 1206x2622 in 272ms react_native_video.RCTVideo layer=CALayer <- AVPlayerLayer player:0x10423ec50 hasFrame=YES gravity=resizeAspect Discovery matches the component, the AVPlayerItemVideoOutput really does hand over a decoded frame, gravity is carried across, and capture completes without crashing. Worth noting react-native-video 6.19.2 renders through AVPlayerLayer, not the AVPlayerViewController its master branch uses -- both rules exist and the AVPlayerLayer one fired. `hasFrame` was previously unobservable, which made "found the component" indistinguishable from "the pipeline works". dumpHierarchy now attaches the providers and reports it, which is how the above was confirmed and is exactly what a user needs when their video is not being captured. Also fixes a warning Metro printed on every start: package.json `exports` points at `lib`, which metro.config blocks on purpose, so resolution failed and fell back noisily. Preferring the `source` condition sends Metro to src/index.ts directly. Verified gone, and bundling still works on both platforms. Documents the Xcode 26 wall in the README: RN 0.81 pins fmt 11.0.2, which does not compile under that toolchain. Upstream, not ours -- it hits any RN 0.81 project -- but anyone opening this example on a current Mac will hit it immediately. Still open: whether drawViewHierarchy picks up the placeholder on real hardware. The simulator captures AVFoundation layers natively, so it cannot answer that. Blocked on the Mac's Apple Development certificate, which expired 2025-01-08. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
…e app at startup 89b1e68 added `unstable_conditionNames: ['source', ...]` to silence a Metro warning. That option is global -- it changes resolution for every package, not just this one -- and it breaks the app on boot with `[runtime not ready] TypeError: ... is not a function`. It shipped because `react-native bundle` was the only check run against it, and bundling successfully is not the same as running. Confirmed on an iPhone: reverting this single line is what made the app boot again, after which the whole capture path ran first try. The warning it was trying to silence is harmless: `exports` points into `lib`, metro.config blocks `lib` on purpose so edits to `src` hot-reload, Metro says so and falls back to file-based resolution, which lands on `src`. Left in place with a comment saying not to "fix" it this way again. Records the device result in the plan, which went against the design: with the placeholder pass skipped, the video is still captured, identically (mean RGB 66.3/81.1/55.5 vs 66.0/81.0/55.3). The provider was live at the time (hasFrame=YES), so this is not a case of the placeholder silently doing nothing -- drawViewHierarchy simply captures AVPlayerLayer natively on iOS 18.7. Whether that also holds for camera preview now decides whether the iOS provider stack is worth keeping at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
…nclusion An earlier round of testing concluded that drawViewHierarchy captures AVPlayerLayer and AVCaptureVideoPreviewLayer natively, and that the iOS provider/placeholder stack was therefore redundant and should be deleted. That was wrong. The A/B harness was broken: `capture()` rebuilds its options object from known keys only, so the debug flag that was supposed to skip the placeholder pass was dropped in JS and never reached native. Both sides of the comparison ran the same code path, so of course they matched. Caught when the same test on Android produced a result that contradicted a known-good earlier run. With the harness calling the native module directly, measured on an iPhone XR / iOS 18.7.9 using @fugood/react-native-video-player 1.0.0-beta.0 and a native AVCaptureVideoPreviewLayer: region with placeholder skipped video (AVPlayerLayer) 94.1% non-black, 102046 col (0,0,0), 1 colour camera (AVCaptureVideoPreviewLayer) 49.6% non-black, 3150 col (0,0,0), 1 colour Both come back as a single uniform black without the placeholder pass. The design premise holds and the iOS provider stack stays. Android, retested with the same player (SurfaceView by default): overlay compositing required, same as before. tvOS: verified through a minimal native harness built against the tvOS SDK and run on an Apple TV simulator -- compiles, links, discovery finds the AVPlayerLayer, and the frame pipeline delivers (hasFrame=YES). But the tvOS simulator captures natively, so the A/B is inert there. Noted as compile- and runtime-verified only. The simulator/device split is the real trap and is now in the README: both simulators composite these layers into the snapshot, devices do not, so verifying this in a simulator gives a pass that does not hold on hardware. Apple's QA1817 enumerates only in-process drawing techniques, which is consistent, but reads like a blanket guarantee. Example now uses @fugood/react-native-video-player 1.0.0-beta.0, which fixes the iOS crash its 0.6.0 had on the new architecture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
The README had accumulated things that came out of building and verifying this, not out of using it: an analysis of Apple's QA1817, measurements from the placeholder A/B, and a workaround for React Native 0.81 not compiling under Xcode 26. None of that changes what a consumer of the package does. What survives is the one consequence a user has to act on -- verify on a real device, because simulators composite these layers themselves and cannot tell you whether capture works. The reasoning behind that sentence now lives in AGENTS.md. AGENTS.md collects what someone changing this package needs: why the compositing works the way it does, and the traps that cost real time here -- - simulators capture these layers natively and devices do not, with the numbers - `capture()` rebuilds its options object from known keys, so a debug flag added to a capture() call never reaches native. That turned an A/B into two identical runs and produced a confidently wrong conclusion. - `react-native bundle` succeeding says nothing about whether the app boots; specifically, do not reintroduce `unstable_conditionNames` in the example's Metro config - Android: force-stop disables accessibility services, sideloaded apps need the restricted- settings appop, autolinking caches the package list - iOS: the fmt/Xcode 26 wall, mixed-architecture pod installs, derived data reuse - codesign cannot work over plain SSH; drive builds through a GUI-started tmux, and wait for the pane to be idle before sending keys CLAUDE.md points at it. Neither ships in the npm tarball. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
`accessibility` mode handed the raw full-display bitmap straight to the encoder, so
`capture({ mode: 'accessibility', excludeStatusBar: true })` returned an image that still had the
status bar in it. The option was only ever applied on the `view` path inside WindowCapture --
and accessibility is the one mode that actually captures the real system bars, so it is the mode
where the option matters most.
Adds a context-based status bar height lookup, since a full-display capture has no decor view
whose insets could be read, and crops before encoding.
Verified on a Pixel 7 Pro: the same capture returns 1080x2340 without the option and 1080x2234
with it, cropping the device's 106px status bar.
Found by a Codex review. The same review also flagged the SurfaceView PixelCopy call as unusable
below API 26; that one is not a defect. `api-versions.xml` in the SDK gives
`PixelCopy` since 24 and the four-argument `request(SurfaceView, Bitmap, listener, Handler)`
overload with no `since` of its own, meaning it inherits 24 -- only the `Rect` and `Window`
overloads are 26, which is exactly why the API 24-25 path falls back to `decor.draw()` for the
window while still copying surfaces normally. Noted in AGENTS.md so the next reader can check it
in ten seconds instead of re-deriving it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
…l status-bar inset Follow-up to the review of faeb3f6. The crop ran inside the callback that ScreenCaptureAccessibilityService deliberately invokes outside its own try/catch, on the premise that the callback only hands the bitmap off. It was allocating a full-display bitmap there — ~10MB at 1080x2340, right after the service had already allocated another one. An OutOfMemoryError would have escaped uncaught and left the JS promise unsettled forever: no resolve, no reject, just a hung await. The crop now happens in encode(), which is already guarded, and the callback is back to handing off. Folded the crop into the same allocation as the rescale. A cropped-and-scaled capture used to build three full-display bitmaps in sequence; now `Bitmap.createBitmap(src, 0, top, w, h, matrix, true)` does both in one. Status-bar height now prefers the inset actually in effect over the platform's nominal `status_bar_height` dimen, so an app running immersive or behind a cutout crops the right amount. Measured on a Pixel 7 Pro: the crop moved from 106px (nominal) to 108px (real inset), 2340 -> 2232. The remaining limit is documented in the code: the inset describes this app's window while the capture is of the whole display, so a different foreground app in a different mode can still disagree. Also from the review: renamed the resource lookup to `nominalStatusBarHeight` so it can no longer be selected by accident in place of the insets-aware version (an Activity is a Context, so the overload resolved silently); deduped the API 24-25 crop into a shared `cropTop`, which also drops an `if (cropped != source)` guard that could never be false; dropped the intermediate local that left a recycled bitmap named in scope; and updated the class doc, which still claimed the class only dealt with the activity window. Regression-checked on device: accessibility + excludeStatusBar gives 1080x2232, JPEG at scale 0.5 gives 540x1170 (exactly half, no stray crop). README now says what excludeStatusBar does per platform and mode, including that the navigation bar is not cropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
…emaining throw paths Second review pass on efd4ac2. Three of its findings were defects that commit introduced, and one showed the change was aimed the wrong way. Wrong measure. efd4ac2 switched the accessibility crop to this app's window inset and called that "the real status-bar inset". For a whole-display capture it is the wrong quantity: the inset describes our window, which is 0 when we are immersive or in the bottom split-screen pane, and accessibility mode usually runs while a *different* app is foreground -- where getCurrentActivity() is null anyway and the code silently fell back to the nominal value the commit claimed to have replaced. Reverted to the platform's nominal status_bar_height for that mode, which is the height the system gives the bar on the display being captured. `view` mode keeps the inset, which is correct there because it really is cropping its own window. Both the code and the README now say what each mode measures and what happens when the foreground app hides the bar. Off the UI thread. The height was being resolved inside the @ReactMethod, which runs on the module/JS thread, by calling getDecorView() and getRootWindowInsets() -- ViewRootImpl state that every other View access in this file reaches through WindowCapture's UI.post(). It could tear or throw before any callback existed. Now resolved on the encoder thread from resources only, which also stops it going stale across a rotation or the accessibility service's 400ms retry. The Promise could still hang. The comment added in efd4ac2 claimed nothing in the accessibility callback could throw, but encode() calls encoder.execute(), and invalidate() shuts that executor down: a reload mid-capture would throw RejectedExecutionException out of the service's deliberately unguarded callback and leave the Promise unsettled -- the exact failure the previous commit set out to fix, just moved. Now caught and rejected. Also: restored the >= 1px clamp lost when crop and scale were fused (a small enough `scale` made createBitmap throw where the old path returned a 1x1 image); fixed the API 24-25 branch, where cropTop() recycled the bitmap the catch block would then recycle again while the cropped one leaked, and where a throwing callback could settle the result twice; restored cropTop()'s identity guard now that it is a shared helper; and replaced the two identical Callback interfaces with one CaptureCallback so the accessibility path no longer needs a pass-through adapter. Verified on a Pixel 7 Pro: accessibility + excludeStatusBar 1080x2234, accessibility JPEG at scale 0.5 540x1170, view mode 1080x2340. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
… paths
Third review pass, this time over the whole branch rather than the last commit. Most of what it
found was mine.
Screenshot detection has been broken on every device below API 34 since the rewrite. The
vendored ScreenCapturetListenManager asserts it is on the main thread in newInstance, startListen
and stopListen; master wrapped those calls in runOnUiThread and the rewrite dropped it, so
detector.start() threw IllegalStateException off the module thread, the Promise rejected, and
addScreenshotListener swallowed it with `.catch(() => {})`. Silent no-op on the majority of
Android devices. start/stop/invalidate now go through UiThreadUtil, and invalidate no longer
lets a throw there skip encoder.shutdown() and super.invalidate().
Two ways a capture could hang forever, both leaving frozen overlays on top of the live video:
registerFrameCommitCallback is not guaranteed to fire (a window that has stopped drawing never
commits) and had no deadline on API 29+, where the older branch at least had one; and
copySurfaceViews only caught IllegalArgumentException, so an OutOfMemoryError from a large
SurfaceView allocation left the pending counter above zero and `done` never ran. Added a
250ms commit deadline and widened the catch.
capture() could also settle twice: captureWindow deliberately calls the callback outside its try
to avoid exactly that, but a throw then unwound into capture()'s own catch(Throwable). All paths
now settle through a once-only wrapper.
`auto` mode could not fall back. getPermissionStatus reported "granted" when the accessibility
service was enabled in Settings but not yet bound, so auto chose accessibility and the capture
hard-rejected instead of using view mode. It now reports granted only when the service is
actually connected.
dumpHierarchy walked the View tree on the module thread, which is the rule AGENTS.md had just
been given. Now on the UI thread, matching what iOS already did.
iOS: a paused AVPlayer never reports a *new* pixel buffer, so pump() bailed out and the video
came back black for the very common "pause, then screenshot" case -- the flag is now an
optimisation for when a frame is already held, not a precondition. warmUp() attached the
providers but never re-armed the idle timer it documents, so a warmUp with no following capture
kept an AVPlayerItemVideoOutput and an AVCaptureVideoDataOutput alive indefinitely. invalidate()
reached into the registry off the main thread, racing discovery and invalidating a main-run-loop
timer from the wrong thread.
Smaller parity fixes: Android now clamps quality to 0..100 like iOS, writes .jpg for both `jpg`
and `jpeg` so the URI suffix agrees across platforms, and encodes once for includeBase64 instead
of compressing the bitmap a second time. Dropped an unused isRunning(), and fixed tsconfig
excludes still naming the pre-rename `Example` directory.
Verified on a Pixel 7 Pro (API 37): capture still works, screenshot detection starts and fires
(detected: 1 after a real screenshot). The pre-API-34 detector path cannot be exercised on this
device -- it takes the registerScreenCaptureCallback route -- so that fix is restoring what
master did, not something measured here.
Not fixed: the camera rotation correction is still iOS 17+ only and its interaction with the
placeholder's masksToBounds is unverified, and the borrowed camera path still retains one buffer
from the host's pool. Both sit in the rotation/borrow paths already flagged as untested.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F
Android
- WindowCapture: everything after copySurfaceViews() runs from Handler
callbacks, outside capture()'s try. A throw there crashed the main thread,
left the overlays frozen on screen and never settled the Promise. Both
continuations now catch, remove the overlays and settle.
- The per-surface PixelCopy stage had no deadline: the framework can accept a
request and drop it (a surface destroyed mid-capture) without ever calling
the listener. Added one, single-shot, cancelled on the normal path so it
does not keep the activity alive for an extra 1.5s.
- Moved the Bitmap allocation inside the try whose catch already claimed to
cover it.
- ScreenCaptureModule: the accessibility branch and dumpHierarchy's UI-thread
runnable were the two entry points that could leave a Promise pending.
- ScreenshotDetector: the API 34 callback was bound to one Activity instance
and held it strongly, so detection died silently on activity recreation and
leaked the destroyed activity. It now rebinds on host resume -- which is
also how a detector that had to start on the legacy path, because no
activity existed yet, upgrades once one does.
- requestPermission no longer sends the user to Settings for an accessibility
service whose toggle is already on but which has not bound yet.
iOS
- RNSCCameraFrameProvider: detach restored a weak _previousDelegate together
with a live queue. AVFoundation requires (delegate != nil) == (queue != nil)
and raises NSInvalidArgumentException otherwise -- a crash whenever the host
released its delegate while we were attached. It also now only restores when
the delegate is still ours, so a delegate the host installed mid-capture
survives.
- RNSCPlayerFrameProvider: observing currentItem forced a strong reference to
the host's AVPlayer, which the registry then kept alive -- audio and all --
for up to 3 idle seconds after the host let go, and the notification arrived
off-main for queue players, racing detach and pump. The output now binds to
the current item from -pump, and the player reference is weak.
- RNSCWindowCapture: drew each window at window.frame, which is in screen
coordinates, into a context anchored at the primary window's origin. Correct
only while that origin is (0,0) -- not under iPad Split View.
JavaScript
- screenCapture() spread options and then wrote a possibly-undefined
excludeStatusBar over it, losing 1.x's Android default of true, and passed
1.x's scale 0 ("native size") straight through as a real scale factor.
- clearCache() takes 1.x's callback again; startListener() accepts and ignores
1.x's keyWords argument.
Verified on a Pixel 7 Pro (API 37) and an iPhone XR (iOS 18.7.9): capture over
live video still composites correctly on both (99.9% / 100% non-black in the
video area), the magenta occluder still lands above the video, and the API 34
screenshot callback still fires.
…nd pass
Four of these were introduced by the previous commit.
Android
- copySurfaceViews posted its deadline *after* the request loop, so a loop that
finished synchronously ran `finish` inline, removed nothing, and then had the
deadline posted behind it -- pinning the capture closure for another 1.5s
after the Promise had settled. It is posted before the loop now.
- afterNextFrame had a deadline only on API 29+. View.post() on a detached
decor is held by ViewRootImpl until re-attach, so the pre-Q path could hang
the same way the Q+ comment describes. Both paths are guarded now.
- The frame-commit callback was never taken back off the ViewTreeObserver when
the deadline won. Those callbacks are only dropped when they fire, so every
timed-out capture left one holding the activity, decor, overlays and Promise.
- encode() resolved the Promise inside the try whose catch rejects it, so a
throw from resolve() came back round as a reject on the same Promise. The
accessibility branch had the same shape; both settle once now, the latter
through WindowCapture.once(), which is what it was written for.
- The screenshot listener emitted into the React instance with no active check
and no catch, on the main thread. invalidate() only posts detector.stop(), so
a screenshot taken in that window turned into an app crash.
- bindModern assigned `callback` before registerScreenCaptureCallback, which
throws when the app has not declared DETECT_SCREEN_CAPTURE. That left the
detector matching neither branch of onHostResume -- permanently dead, with
the legacy fallback never tried. It now reports failure and falls back.
- isModeAvailable('accessibility') answered from Build.VERSION alone, so it
said true for apps that never declared the service, and requestPermission
sent them to a Settings page listing nothing to enable.
iOS
- Re-binding the video output to a new AVPlayerItem left the previous item's
frame in _latest. A freshly added output reports no new pixel buffer for the
first few frames, so pump took its early-out and hasFrame said YES: the
capture composited the last frame of the *previous* video over the one
playing. The held frame is released on every rebind now.
- The registry overwrote a dead-but-cached provider without detaching it,
leaving cleanup to dealloc -- where the camera provider's "is the output
still pointing at me" guard can never match, because AVFoundation holds the
delegate weakly. The host's own camera pipeline stayed disconnected.
- width/height were size * scale in points; the renderer rounds, so a scaled
capture reported a fractional width that disagreed with the file it had just
written, and with Android. They come from the CGImage now.
JavaScript
- 'auto' mode did a native round-trip per capture() for a value that only
changes while the app is backgrounded in Settings. Cached, dropped on
foreground and on requestPermission().
- clearCache's 1.x callback got a number; 1.x passed { code }. It gets an
object again.
- Documented that the screenshot event payload is *not* 1.x-compatible --
base64 is gone and iOS has no uri. The old note implied only keyWords had
changed.
Re-verified on the Pixel 7 Pro and the iPhone XR: video area 100% non-black on
both (257340 / 104227 colours), occluder still above the video, API 34
detection still firing.
Modes and permissions disagreed across three call sites
- iOS reported every mode but the literal "view" as unavailable, so
isModeAvailable('auto') was false there and true on Android -- for the
library's own default mode, which always resolves to `view` on iOS. An app
gating its capture button on isModeAvailable(getMode()) hid it on iOS.
- Android's getPermissionStatus('accessibility') answered "denied" for an app
that never declared the service, while requestPermission and isModeAvailable
had just learned to answer "unavailable". "denied" means "the user has to
act", and no user action can end that loop.
Costs that were invisible
- waitForFrames spent its whole 8 x 16ms budget on every capture, forever,
once any provider could not produce frames -- DRM content, or a camera
session that refused an output. Providers that exhaust the budget are
remembered and no longer blocked on; one that recovers is reinstated.
- The camera provider re-ran the output scan and canAddOutput probe on every
capture after a refusal that will not change within the session.
- clearCache listed the cache directory and unlinked file by file on the
calling thread, which under TurboModules is the JS thread.
- 'auto' mode cached the resolved status but not the in-flight promise, so a
burst starting in one tick still paid a round-trip per call.
Lifetime and correctness
- The pre-Q afterNextFrame path queues into ViewRootImpl when the decor is
detached, and there is no API to take that back. It now reaches the guarded
runnable through a slot the deadline empties, so the activity, decor,
overlays and Promise are released either way.
- The hardware-wrapped screenshot bitmap was only recycled on the success
path; copying a full display costs ~18MB and can throw.
- capture() rebuilt its options object key by key, silently dropping anything
the TS union does not name. That is the trap recorded in AGENTS.md, which
once turned an A/B test into two identical runs. It spreads now.
- The screenshot refcount came from emitter.listenerCount() on a bare global
device event, so a 1.x-era DeviceEventEmitter listener or a second copy of
the package kept native detection running after our last subscriber left.
- onHostResume treated a null current activity as "the activity changed" and
tore down a working API 34 registration in favour of a MediaStore watcher
whose permission the app need not even hold.
- The camera orientation correction was dead below iOS 17 although the podspec
targets 15.1; videoOrientation covers 15 and 16.
- iOS reported width/height that could disagree with the file on disk -- fixed
in the previous commit, and CI now takes the Ruby version from
example/.ruby-version instead of a pin whose comment claimed that file did
not exist.
Duplication
- Two settle-once wrappers per capture. WindowCapture owns the guarantee for
the view path; the accessibility path arranges its own, and each is now
load-bearing.
- Discovery and dumpHierarchy each had their own media-layer walk with the
same class checks and the same skip rule. One walk, two consumers -- so the
next layer class is one edit, not two that must stay in step.
Re-verified on the Pixel 7 Pro and the iPhone XR: video area 100% non-black on
both (252594 / 104227 colours), occluder above the video, detection firing.
Detection could stop for good - onHostDestroy() unbinds both paths, so "nothing is bound" is a state a live listener can be left in -- and neither resume branch matched it. A destroyed and recreated activity ended detection for the rest of the React instance's life, silently: the promise from startScreenshotDetection() resolved long ago. - stopListener() zeroed the subscription count without telling the outstanding subscriptions, so a later remove() on a stale one drove the count to zero again and stopped detection under a live listener -- or, with no intervening add, to -1, after which it could never be stopped at all. Subscriptions are tracked by identity now. Hangs and double settles - The whole-window PixelCopy was the last async hop with no deadline. Its listener is the only thing that takes the overlays back off the SurfaceViews, so a dropped callback left frozen stills pinned over live video and never settled the Promise. - iOS encodeImage resolved and rejected inside the @Try whose @catch rejects, the hazard the Android side hoisted its settles out of the try to avoid. Frames that never arrived - waitForFrames broke out of the readiness loop at the first not-ready provider. -hasFrame is what pumps a player provider, so everything behind it went unpumped for the whole budget and was then declared hopeless off a single pump -- and the video came back black. - That hopeless set was keyed on `%p` identifiers and never pruned, so a new AVPlayer on a recycled address inherited a dead one's verdict and was never waited for again. It is keyed on the provider object with weak membership now. - contentsTransform rotated the placeholder about its centre without swapping its bounds, so a quarter turn -- a front camera in landscape -- rendered overhanging the preview rect with the aspect transposed. Threading - The camera provider's borrowed-delegate state was written by -detach on the main thread and read by the sample-buffer callback on the capture queue. For a __weak ivar that is an unsafe access to the weak table, not just a stale read. Both sides take the existing lock. - The vendored MediaStore watcher ran its content observers on the main looper, so every image written by any app on the device did a cross-process query on the UI thread for as long as detection was on. They get their own thread; only the listener callback is posted back. This is the one change from the original, and its header now says so. - iOS clearCache listed the folder and unlinked file by file on the calling thread, which under TurboModules is the JS thread -- the same work Android had already moved off it. Also: `auto` no longer rejects a capture when the permission probe fails, or caches the transient "enabled but not bound yet" denial for the whole foreground session; AGENTS.md no longer documents the option-key trap as live, since capture() spreads now; and the example drops a safe-area dependency nothing imported. Re-verified on the Pixel 7 Pro and the iPhone XR: video area 100% non-black on both (255659 / 104227 colours), occluder above the video, detection firing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #2, without MediaProjection.
The problem
Video players and camera previews are composited by the GPU, not by the drawing APIs a
screenshot goes through. Capture them the usual way and you get a black rectangle. On iOS the
current
drawViewHierarchyInRectapproach (81f5581) does not fix this — it renders fine in theSimulator and comes out black on device, which is what made it look like it worked.
Why not MediaProjection
Since Android 14 the consent
Intentcannot be cached or reused —createVirtualDisplaythrowsSecurityException. So consent cannot survive a process restart and every cold start wouldshow a system dialog. It also requires a foreground service with a persistent notification,
shows a status-bar chip from Android 15 QPR1, and stops itself when the device locks. Wrong shape
for a screenshot utility.
screenshottywas considered and rejected too: last commit 2021, noAndroid 14 handling at all.
The approach
Both platforms use the same idea, which removes all manual z-order and occlusion maths:
Because the placeholder lives inside the media view, anything drawn on top of that view is still
drawn on top of the placeholder, and clipping / corner radius / transforms are applied by the
view system for free. Cost is O(1) full-screen renders no matter how many media components are
on screen.
PixelCopyeachSurfaceView, install the result as aViewOverlayon thatsame
SurfaceView, then one windowPixelCopy. Replaces the old paste-back, which ignoredz-order and ran serially with a 5s latch each (N SurfaceViews = worst case N×5s).
(
AVCaptureVideoPreviewLayer.session,AVPlayerLayer.player,AVPlayerViewController.player),never private layer classes. One rule covers a whole class of packages, so VisionCamera,
react-native-video, expo-camera and expo-video all work with no cooperation from those
packages and no app setup.
Providers attach lazily and detach after ~3 idle seconds — an app that never captures pays
nothing. Camera capture borrows the existing
AVCaptureVideoDataOutputdelegate and forwardsevery callback rather than reconfiguring somebody else's live session.
Also in here
deliberately not declared in the library manifest — merging would push an accessibility
service onto every consumer app and drag them all into Play's Accessibility API policy review,
including apps that only use
viewmode.compileSdk 36, minSdk 24, podspec on iOS/tvOS 15.1. Dropped an unused zxing dependency.
UIStatusBarclass, whichstopped existing in iOS 13. Dead code and an App Store review risk.
Example/replaced, not upgraded. It was a leftoverReactNavigationTVDemoscaffold onRN 0.64 with react-navigation/reanimated/screens deps and patches this library never used.
The new
example/demos a video player forced onto aSurfaceView, which is exactly the casethat motivated the rework.
Verification
tsc,bob build, codegen — cleanexample/androidbuilds a real debug APK on both architectures. Verified withjavapthatthe
newarch/oldarchsource-set swap genuinely takes effect rather than reusing the other AAR.d.ts, which also proves theexportsmap resolvesCI (
.github/workflows/ci.yml) now runs all of the above on every push, plus a packaging job andan iOS job.
What reviewers should know
it was written in. The
iosCI job on this PR is the first time it gets built — if it is red,that is expected to be the reason.
docs/IMPLEMENTATION_PLAN.md§12 has thechecklist: does the placeholder pass actually capture VisionCamera and react-native-video, is
afterScreenUpdates: falseviable, and camera mirroring/rotation (the transform collapses tothe identity in the common case, so the interesting path is untested).
FLAG_SECUREsurfaces still come out black. Those frames never leave the hardware secure path.
viewmode captures the activity window only — Dialogs and<Modal>are separatewindows and reaching them needs non-SDK reflection, which this does not do.
🤖 Generated with Claude Code
https://claude.ai/code/session_01VJkjpaPEdjLJTfQUr55n2F