Skip to content

Add face detection #787

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
scarlac merged 6 commits into from
May 12, 2026
Merged

Add face detection #787

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
cb03eb4
Add face detection
mike-rbrts Apr 24, 2026
debc37a
Add onFaceDetectionInstallStatus event for MLKit module download
mike-rbrts Apr 29, 2026
752472e
Skip MLKit face detection install on non-Google Play devices
mike-rbrts Apr 29, 2026
0eda355
Allow simultaneous barcode scanning and face detection on Android
mike-rbrts May 7, 2026
6fdaf55
Address Copilot comments, face detection threading and prop reactivity
mike-rbrts May 8, 2026
f85984d
Map face detection bounds to preview-space coordinates
mike-rbrts May 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 46 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
<li><h3>Cross Platform (iOS and Android)</h3></li>
<li><h3>Optimized for performance and high photo capture rate</h3></li>
<li><h3>QR / Barcode scanning support</h3></li>
<li><h3>Face detection support</h3></li>
<li><h3>Camera preview support in iOS simulator</h3></li>
</ul>
</td>
Expand Down Expand Up @@ -170,6 +171,45 @@ Additionally, the Camera can be used for barcode scanning
/>
```

#### Face Detection

Detect faces in real time. iOS uses Apple Vision; Android uses Google ML Kit.

> **Android requires a Google Play Store device**

```tsx
<Camera
...
faceDetectionEnabled={true}
faceDetectionThrottleMs={100} // optional, default 100ms
onFaceDetected={(event) => {
// event.nativeEvent.faces: FaceData[]
// each face: { id, yaw, pitch, roll, boundsX, boundsY, boundsWidth, boundsHeight }
}}
// Android only — track MLKit face module download progress
onFaceDetectionInstallStatus={(event) => {
// event.nativeEvent.state: FaceDetectionInstallState
// 'pending' | 'downloading' | 'installing' | 'ready' | 'failed' | 'unavailable'
}}
/>
```

##### Android: pre-download the ML Kit model

This library depends on the unbundled `play-services-mlkit-face-detection` variant — the model is downloaded by Google Play Services rather than bundled in the APK. By default the download happens on first use, and `onFaceDetectionInstallStatus` reports the progress.

To have Play Services [pre-download the model in the background after the app is installed](https://developers.google.com/ml-kit/tips/installation-paths#how_to_download_models), add this to your app's `AndroidManifest.xml`:

```xml
<application ...>
<meta-data
android:name="com.google.mlkit.vision.DEPENDENCIES"
android:value="face" />
</application>
```

> **Note:** Requires Google Play Services on the device. On devices without Google Play Services, `onFaceDetectionInstallStatus` fires once with `'unavailable'`.

### Camera Props (Optional)

| Props | Type | Description |
Expand Down Expand Up @@ -197,7 +237,7 @@ Additionally, the Camera can be used for barcode scanning
| `resizeMode` | `'cover' / 'contain'` | Determines the scaling and cropping behavior of content within the view. `cover` (resizeAspectFill on iOS) scales the content to fill the view completely, potentially cropping content if its aspect ratio differs from the view. `contain` (resizeAspect on iOS) scales the content to fit within the view's bounds without cropping, ensuring all content is visible but may introduce letterboxing. Default behavior depends on the specific use case. |
| `scanThrottleDelay` | `number` | Duration between scan detection in milliseconds. Default 2000 (2s) |
| `maxPhotoQualityPrioritization` | `'balanced'` / `'quality'` / `'speed'` | [iOS 13 and newer](https://developer.apple.com/documentation/avfoundation/avcapturephotooutput/3182995-maxphotoqualityprioritization). `'speed'` provides a 60-80% median capture time reduction vs 'quality' setting. Tested on iPhone 6S Max (66% faster) and iPhone 15 Pro Max (76% faster!). Default `balanced` |
| `iOsDeferredStart` | `boolean` | iOS 26+ only. Enables `AVCaptureOutput.deferredStartEnabled` when supported to get the preview visible faster. Default `true`. When enabled, the first capture can be delayed by a few hundred milliseconds. Ignored on Android and on older iOS versions. |
| `iOsDeferredStart` | `boolean` | iOS 26+ only. Enables `AVCaptureOutput.deferredStartEnabled` when supported to get the preview visible faster. Default `true`. When enabled, the first capture can be delayed by a few hundred milliseconds. Ignored on Android and on older iOS versions. |
| `onCaptureButtonPressIn` | Function | Callback when iPhone capture button is pressed in or Android volume or camera button is pressed in. Ex: `onCaptureButtonPressIn={() => console.log("volume button pressed in")}` |
| `onCaptureButtonPressOut` | Function | Callback when iPhone capture button is released or Android volume or camera button is released. Ex: `onCaptureButtonPressOut={() => console.log("volume button released")}` |
| **Barcode only** |
Expand All @@ -207,6 +247,11 @@ Additionally, the Camera can be used for barcode scanning
| `laserColor` | Color | Color of barcode scanner laser visualization. Default: `red` |
| `frameColor` | Color | Color of barcode scanner frame visualization. Default: `yellow` |
| `onReadCode` | Function | Callback when scanner successfully reads barcode. Returned event contains `codeStringValue`. Default: `null`. Ex: `onReadCode={(event) => console.log(event.nativeEvent.codeStringValue)}` |
| **Face detection** |
| `faceDetectionEnabled` | `boolean` | Enable real-time face detection. Default: `false` |
| `faceDetectionThrottleMs` | `number` | Minimum milliseconds between `onFaceDetected` emits. Default: `100` |
| `onFaceDetected` | Function | Callback while face detection is active, with one entry per detected face (empty array if none). |
| `onFaceDetectionInstallStatus` | Function | **Android only.** Callback while the MLKit face detection module is being downloaded by Google Play Services. See [Android: pre-download the ML Kit model](#android-pre-download-the-ml-kit-model) above to skip the first-run download. |

### Imperative API

Expand Down
1 change: 1 addition & 0 deletions android/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ dependencies {
// implementation "androidx.camera:camera-extensions:${camerax_version}"

implementation 'com.google.mlkit:barcode-scanning:17.3.0'
implementation 'com.google.android.gms:play-services-mlkit-face-detection:17.1.0'
}
repositories {
mavenCentral()
Expand Down
80 changes: 77 additions & 3 deletions android/src/main/java/com/rncamerakit/CKCamera.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ import android.graphics.Rect
import android.graphics.RectF
import android.util.Size
import com.facebook.react.uimanager.UIManagerHelper
import com.google.android.gms.tasks.Task
import com.google.android.gms.tasks.Tasks
import com.google.mlkit.vision.barcode.common.Barcode
import com.rncamerakit.events.*

Expand Down Expand Up @@ -83,6 +85,7 @@ class CKCamera(context: ThemedReactContext) : FrameLayout(context), LifecycleObs
private var preview: Preview? = null
private var imageCapture: ImageCapture? = null
private var imageAnalyzer: ImageAnalysis? = null
private var faceAnalyzer: FaceAnalyzer? = null
private var orientationListener: OrientationEventListener? = null
private var viewFinder: PreviewView = PreviewView(context)
private var rectOverlay: RectOverlay = RectOverlay(context)
Expand Down Expand Up @@ -112,6 +115,10 @@ class CKCamera(context: ThemedReactContext) : FrameLayout(context), LifecycleObs
private var barcodeFrameSize: Size? = null
private var allowedBarcodeTypes: Array<CodeFormat>? = null

// Face detection props
private var faceDetectionEnabled: Boolean = false
private var faceDetectionThrottleMs: Long = FaceAnalyzer.DEFAULT_THROTTLE_MS

private fun getActivity() : Activity {
return currentContext.currentActivity!!
}
Expand Down Expand Up @@ -142,11 +149,21 @@ class CKCamera(context: ThemedReactContext) : FrameLayout(context), LifecycleObs
override fun onDetachedFromWindow() {
super.onDetachedFromWindow()

faceAnalyzer?.close()
faceAnalyzer = null
cameraExecutor.shutdown()
orientationListener?.disable()
cameraProvider?.unbindAll()
}

override fun onWindowFocusChanged(hasWindowFocus: Boolean) {
super.onWindowFocusChanged(hasWindowFocus)
if (hasWindowFocus && cameraProvider == null &&
ContextCompat.checkSelfPermission(context, Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED) {
viewFinder.post { setupCamera() }
}
}

override fun dispatchKeyEvent(event: KeyEvent?): Boolean {
val keyCode = event?.getKeyCode()
val action = event?.getAction()
Expand Down Expand Up @@ -341,8 +358,11 @@ class CKCamera(context: ThemedReactContext) : FrameLayout(context), LifecycleObs

val useCases = mutableListOf(preview, imageCapture)

if (scanBarcode) {
val analyzer = QRCodeAnalyzer({ barcodes, imageSize ->
faceAnalyzer?.close()
faceAnalyzer = null

val barcodeAnalyzer: QRCodeAnalyzer? = if (scanBarcode) {
QRCodeAnalyzer({ barcodes, imageSize ->
if (barcodes.isEmpty()) return@QRCodeAnalyzer

// 1. Filter by allowed barcode formats
Expand Down Expand Up @@ -390,7 +410,27 @@ class CKCamera(context: ThemedReactContext) : FrameLayout(context), LifecycleObs
onBarcodeRead(filteredBarcodes)
}
}, scanThrottleDelay)
imageAnalyzer!!.setAnalyzer(cameraExecutor, analyzer)
} else null

faceAnalyzer = if (faceDetectionEnabled) {
FaceAnalyzer(
faceDetectionThrottleMs,
context,
viewFinder,
{ state -> onFaceDetectionInstallStatus(state) },
{ payloads -> onFaceDetected(payloads) }
)
} else null

val activeFaceAnalyzer = faceAnalyzer
if (barcodeAnalyzer != null || activeFaceAnalyzer != null) {
imageAnalyzer!!.setAnalyzer(cameraExecutor) { image ->
val tasks = mutableListOf<Task<*>>()
barcodeAnalyzer?.analyzeWithoutClosing(image)?.let { tasks.add(it) }
activeFaceAnalyzer?.analyzeWithoutClosing(image)?.let { tasks.add(it) }
if (tasks.isEmpty()) image.close()
else Tasks.whenAllComplete(tasks).addOnCompleteListener { image.close() }
}
useCases.add(imageAnalyzer)
}

Expand Down Expand Up @@ -547,6 +587,27 @@ class CKCamera(context: ThemedReactContext) : FrameLayout(context), LifecycleObs
?.dispatchEvent(ReadCodeEvent(surfaceId, id, barcodes.first().rawValue, codeFormat.code))
}

private fun onFaceDetected(payloads: List<FacePayload>) {
// CameraX auto-mirrors the front-camera Preview but not ImageAnalysis,
// so flip X here to keep bounds in preview-space for JS consumers.
val mirrored = if (lensType == CameraSelector.LENS_FACING_FRONT) {
payloads.map { it.copy(boundsX = 1.0 - it.boundsX - it.boundsWidth) }
} else {
payloads
}
val surfaceId = UIManagerHelper.getSurfaceId(currentContext)
UIManagerHelper
.getEventDispatcherForReactTag(currentContext, id)
?.dispatchEvent(FaceDetectedEvent(surfaceId, id, mirrored))
}

private fun onFaceDetectionInstallStatus(state: String) {
val surfaceId = UIManagerHelper.getSurfaceId(currentContext)
UIManagerHelper
.getEventDispatcherForReactTag(currentContext, id)
?.dispatchEvent(FaceDetectionInstallStatusEvent(surfaceId, id, state))
}

private fun onOrientationChange(orientation: Int) {
val remappedOrientation = when (orientation) {
Surface.ROTATION_0 -> RNCameraKitModule.PORTRAIT
Expand Down Expand Up @@ -674,6 +735,19 @@ class CKCamera(context: ThemedReactContext) : FrameLayout(context), LifecycleObs
if (restartCamera) bindCameraUseCases()
}

fun setFaceDetectionEnabled(enabled: Boolean) {
val restartCamera = enabled != faceDetectionEnabled
faceDetectionEnabled = enabled
if (restartCamera) bindCameraUseCases()
}

fun setFaceDetectionThrottleMs(throttleMs: Int) {
val newThrottle = if (throttleMs < 0) FaceAnalyzer.DEFAULT_THROTTLE_MS else throttleMs.toLong()
if (faceDetectionThrottleMs == newThrottle) return
faceDetectionThrottleMs = newThrottle
faceAnalyzer?.throttleMs = newThrottle
}

fun setScanThrottleDelay(delayMs: Int) {
val newDelay = if (delayMs < 0) 2000L else delayMs.toLong()
val restartCamera = scanThrottleDelay != newDelay && scanBarcode
Expand Down
Loading