Asymmetric -2*f_x/screenWidth in cameraProjectionMatrix — origin of X-mirror unclear

[kalwalt]

Code reference: improve-viewMatrix_GL / dev.

Summary

WebARKitGL.cpp::cameraProjectionMatrix writes
projectionMatrix[0] = -2.0f * f_x / screenWidth (with a leading minus), while the symmetric Y formula
projectionMatrix[5] = 2.0f * f_y / screenHeight has no negation. The asymmetry doesn't match the standard
pinhole-to-OpenGL derivation (projectionMatrix[0] = +2 * fx / width), but in practice removing the negation
breaks AR overlay placement for a static-image example — overlays land on the wrong screen half. So the
negation is doing real work; it just isn't obvious what X-mirror it's compensating for.

Empirical evidence

Tested in webarkit/webarkit-testing#31's static-image Teblid example (no webcam, no display mirror):

proj[0] sign	Rendered position	Axis rotation (THREE.AxesHelper)
-2*f_x/w (current)	✅ overlay sits on the actual marker (left panel of the printout)	red X points "wrong" (toward screwdriver, off the marker)
+2*f_x/w (textbook)	❌ overlay placed on the opposite screen half (right panel, no marker)	red X rotated correctly to align with the marker's horizontal

So the negation is consistent for position but creates the impression of rotation errors — and the textbook
sign does the opposite. Looks like an X-mirror is being applied somewhere and the projection's negation
compensates for it.

What I've ruled out

• Camera intrinsics (WebARKitCamera::setupCamera)
  build a standard pinhole [fx 0 cx; 0 fy cy; 0 0 1] with the principal point at the image center. No X flip there.
• Display side — in the static-image example, the <img> is shown with object-fit: cover and no CSS
  transform: scaleX(-1). The canvas overlay has the same layout. Both consume the image's native orientation.
• Frame processing — context_process.drawImage(image, 0, 0, vw, vh) does no mirror. getImageData(...) reads
  the raw canvas pixels and they go into the WASM frame buffer via HEAPU8.set at the buffer offset.
• convert2Grayscale (WebARKitUtils.h)
  wraps the buffer in a cv::Mat of declared (rows, cols) and runs cv::cvtColor(..., COLOR_RGBA2GRAY) — no flip.
• arglCameraViewRHf (JS path in WebARKitController.js) negates Y and Z rows only — no X negation.
• cameraPoseFromPoints / solvePnPRansac / cv::hconcat(rMat, tvec, pose) are standard OpenCV; the
  resulting [R|t] is laid out the conventional way.

So the X mirror has to be hiding somewhere between solvePnP's output and the GL projection, and I can't pinpoint
where.

Working hypotheses (not verified)

• Inherited artoolkit convention that paired with a horizontally mirrored webcam display, where the negation
  pre-compensates for the display flip (would explain why webcam examples look OK).
• A subtle sign issue in how transMat/trans is read into the JS-side pose via transMatToGLMat and
  then composed with the projection.
• A solvePnP rvec-to-rotation handedness that's offset by a 180° rotation around an unexpected axis.

What I'd like

A targeted look from someone who knows the artoolkit-derived pipeline well: where is the X-mirror that the
-2*f_x/w negation is silently compensating for? Until that's identified, the negation should stay (removing
it visibly breaks placement), but the asymmetry vs the symmetric Y formula is a real source of confusion and
blocks cleaning up the projection.

Refs

• Tracked pose renders behind the camera (OpenCV vs OpenGL convention); matrixGL_RH throws #34 (related pose CV→GL conversion fix that's proceeding)
• 3D pose renders behind the camera (CV vs GL convention); matrixGL_RH throws webarkit-testing#31 (downstream)
• Add static image example with Teblid tracker (#30) webarkit-testing#32 / static-image example surfaced this

Test that locks in the current behavior

tests/webarkit_test.cc::TestCameraProjectionMatrix
already asserts projectionMatrix[0] == -1.7851850084276433. If the asymmetry is intentional, that assertion is
fine — but a comment in the source explaining why would save future readers from the same hunt.

[kalwalt]

Investigation findings (parking with results)

Deep investigation into the tilted/mis-projected AR overlay. Core feature is unaffected — the marker tracks and AR content anchors on the marker (shipped in webarkit/webarkit-testing#33). This is the residual axis/position-alignment polish. Documenting everything so it can be picked up by someone with rendering-pipeline knowledge.

Decisive finding

Using a debug overlay that draws the tracker's own processed frame (canvas_process) plus the detected corners on top (in the same frame coords): the processed frame is correctly oriented (pinball on the left, matching the display), but the detected corners land on the opposite side (right). So the tracker localizes the marker X-mirrored within its own correctly-oriented frame. The raw pose has the same mirror — rendering puts the overlay on the mirror-image side.

Empirically characterized behavior

With modelview D·R (D = diag(1,−1,−1)) and the GL projection:

projection proj[0]	Position	Orientation (axes)
−2·f_x/W (current)	✅ on marker	❌ red/green mirrored
+2·f_x/W	❌ mirror-image side	✅ red/green aligned

Position and orientation are mirror-coupled — flipping the projection X sign flips both. No single projection sign yields both correct, because the object origin is off the projection centre.

Hypotheses eliminated (with evidence)

1. Projection sign = ArtoolkitX X+/Y− — refuted. ArtoolkitX's X+/Y− pairs with its Y-up marker object-point convention (the Y,Z column negation in ARTrackable2d::updateWithTwoDResults). WebARKit uses Y-down marker points (pt3d = (px,py,0) image pixels), so the consistent pair is D·R modelview + X+/Y+. But X+ alone mirrors position (see table).
2. Pose convention (revert Fix CV->GL pose conversion in updateTrackable() #36 / D·R vs D·R·D) — D·R (i.e. Fix CV->GL pose conversion in updateTrackable() #36, no column negation) is correct for WebARKit's Y-down markers; reverting Fix CV->GL pose conversion in updateTrackable() #36 to D·R·D did not fix the axes and is the wrong convention here. Fix CV->GL pose conversion in updateTrackable() #36 should stay.
3. Reference image mirrored — checked data/pinball.jpg vs the demo photo: same orientation (POLICE BOX, MIL targets, face all match), not mirrored.
4. Feed orientation (GrayScale vs canvas-2D) — refuted by reading GrayScale's shader: it only touches Y (tex_coords.y = 1−y, gl_Position.y *= flipY); X is never flipped. So the working teblid_example (GrayScale/GRAY path) and the static example (canvas-2D/RGBA path) feed the same X orientation. The X-mirror is not a feed-orientation difference.
5. Degenerate bbox — when the static example processed a single frame then stopped, corners were degenerate (extending far outside the frame). Processing continuously (like teblid_example, every found/not-found re-process()s) stabilizes them — but they then stabilize on the mirrored side.

Where it points

The anomaly appears tracker-internal (the homography/_bBoxTransformed and the solvePnP pose both localize the marker mirrored), and is specific to this data path — teblid_example (GRAY via GrayScale, continuous webcam) shows corners correctly on the marker, while the static RGBA path does not, despite identical X feed orientation. Candidate areas for someone with pipeline knowledge:

• the RGBA vs GRAY convert2Grayscale / initFrameBuffer path,
• the homography handedness from getHomographyInliers on this input,
• any interaction with the marker being loaded as decoded RGBA pixels (initTrackerGray(..., RGBA)) at 1637×2048.

Repro

webarkit/webarkit-testing static Teblid example (threejs_teblid_static_image_ES6_example.html). A debug overlay drawing canvas_process + corners makes the mirror visible immediately.

Related: #34 (resolved by #36), webarkit/webarkit-testing#31, #33.