rh264: decoder performance series (~1.2x, -18.7% instructions) - #19307
Merged
Conversation
added 9 commits
July 30, 2026 06:24
The P/B deblock walked chroma edges one sample at a time through the scalar rh264_filter_chroma_edge - the vectorised edge filter existed (the all-intra path uses it) but takes one strength and tc for the whole run, and the P/B derivation gives every 4-sample luma segment its own, so that path was left scalar. On a CABAC+B stream the per-sample walk was 4% of decoded instructions. New per-segment kernels take the strength class and tc0 as per-lane vectors: both filter branches are computed vector-wide and blended per lane, inactive lanes keeping their samples, so one pass covers the whole chroma edge of a macroblock regardless of how the four segments differ. Lane k of n belongs to segment k*4/n, which also covers 4:2:2's doubled rows on vertical edges. The two P/B call sites hoist the per-plane alpha/beta/tc derivation out of the segment loop and issue one call per edge per plane; SSE2 and NEON kernels with the same scalar fallback shape as the existing edge filters (verified bit-exact in a forced-scalar build as well). Instruction count (callgrind, 60-frame CABAC+B stream, -O2): -2.7%. Decode fps, 300 frames, single thread, x86_64 -O2 -march=native, best of four, same-session A/B: CAVLC I/P 480p: 345 -> 362 (1.05x) CABAC I/P 480p: 225 -> 238 (1.06x) CABAC+B 720p: 132 -> 134 Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile and 720p B-frame streams, and against openh264 on the left/top/bottom-cropped vectors, in both SIMD and forced-scalar builds. Sanitizer sweep: ASan+UBSan clean over the seven streams and 500 mutated/truncated inputs with plane emission active; LeakSanitizer strict clean including decode-error paths; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
The LPS/MPS decision in rh264_cabac_decode is the decoded bin itself, so as a conditional branch it mispredicts at exactly the bin entropy; bypass bins (signs, exp-golomb suffixes) are near-uniform and pay the full misprediction rate. The engine also kept pStateIdx and valMPS in separate arrays with a separate valMPS-flip branch at state 0, and refilled its bit buffer one byte at a time. The decision is now taken as an all-ones mask - both outcomes run the same straight-line code, with offset, range, bin and the state update selected by the mask. pStateIdx and valMPS fuse into one byte (pStateIdx * 2 | valMPS) and both transitions come from a 2x128 table built per instance at context init from rh264_transIdxMPS/LPS, the valMPS flip at state 0 folded into the LPS row (9.3.3.2.1.1), so the spec tables stay the single source of truth. The renormalisation refill tops the buffer up past 16 bits so consecutive renormalisations share one refill; bytes past the end still read as zero. Decode fps, 300 frames, single thread, x86_64 -O2 -march=native, best of four, same-session A/B: CABAC I/P 480p: 230 -> 236 (1.03x) CABAC+B 480p: 233 -> 239 (1.03x) CABAC+B 720p: 131 -> 135 (1.03x) CAVLC I/P: unchanged Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile and 720p B-frame streams, and against openh264 on the left/top-cropped vectors. Sanitizer sweep: ASan+UBSan clean over all five streams and 400 mutated/truncated inputs; LeakSanitizer strict clean; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
Every luma MC call gathered a (bw+5)x(bh+5) patch and every chroma call a (bw+1)x(bh+1) one - a memcpy per call whose only purpose is baking the picture-edge clamp into a dense buffer, paid even though the overwhelming majority of interpolation windows lie entirely inside the picture. Chroma even routed full-pel copies through it. The filters index the source through a pointer and a stride, so the in-bounds case now aliases them straight onto the reference plane and the patch is gathered only at the borders. The direct path requires the window to sit strictly above the last row: the vector tails read a few bytes past the 6-tap/bilinear margin, and the spare row keeps those reads inside the plane allocation (bottom-edge blocks take the patch, as edge blocks always did). Decode fps, 300 frames, single thread, x86_64 -O2 -march=native, best of four, same-session A/B: CABAC+B 480p: 229 -> 255 (1.11x) CABAC+B 720p: 132 -> 143 (1.08x) CAVLC I/P 480p: 370 -> 385 (1.04x) CABAC I/P 480p: 235 -> 242 (1.03x) Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile and 720p B-frame streams, and against openh264 on the left/top/bottom-cropped vectors. Sanitizer sweep: ASan+UBSan clean over the streams and 400 mutated inputs (corrupt vectors exercise the border fallback); LeakSanitizer strict clean; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
Every displayed picture was plane-copied from the working frame into its output slot - a full frame of memcpy per picture whose content already exists, since the reference copy (when the picture is one) sits in the DPB by the time the output queue takes it. Frame pictures now swap their plane buffers with the slot instead: the slot was allocated from the same SPS so geometry and strides are identical, every macroblock of the next picture rewrites the working planes, and the slot being displayed is already excluded from selection, so its buffer is never handed back while borrowed. The derived plane views are re-pointed on both sides. Field pairs keep the copy - their two pictures fill the working planes incrementally, which a swapped-in stale buffer would break. Instruction count (callgrind, 60-frame CABAC+B stream, -O2): -1.5%, the output-queue share of memcpy traffic eliminated (3.9% -> 2.4% of decoded instructions). Wall-clock is within noise on a desktop whose memory system hides a frame of copy per picture; the traffic reduction is aimed at the bandwidth-constrained targets this frontend actually ships to. Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile and 720p B-frame streams (reorder-heavy paths exercise the swap), and against openh264 on the left/top/bottom-cropped vectors. Sanitizer sweep: ASan+UBSan clean over the streams and 350 mutated inputs with plane emission active; LeakSanitizer strict clean; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
The per-segment chroma filter transposed all four loaded columns back to rows to store them, but a chroma edge filter only ever writes p0 and q0 - p1 and q1 are read-only taps, so half the stored columns were unchanged and the second 8x8 transpose existed to put them back where they already were. The two filtered columns are now interleaved into per-row byte pairs (one punpcklbw, or vst2 on NEON) and written two bytes a row, which drops the store-side transpose entirely. The load side still transposes: p1 and q1 are needed as columns to filter. Instruction count (callgrind, 60-frame CABAC+B stream, -O2): -0.3%. Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile and 720p B-frame streams, and against openh264 on the left/top-cropped vectors. Sanitizer sweep: ASan+UBSan clean over all five streams and 350 mutated/truncated inputs including a crop corpus with plane emission active; LeakSanitizer strict clean; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
rh264_dequant4x4 rebuilt the per-coefficient scale from two table lookups and a multiply on every one of sixteen coefficients, and re-tested both the has_dc_sep and the per>=4 conditions per coefficient, though all three are invariant across a block. rh264_itransform4x4 was a scalar butterfly. Together they were 8.8% of decoded instructions on a CABAC+B stream. The scale vector is now built once per block, the shift decision taken once, and the body is a 4-lane multiply-and-shift; has_dc_sep restores blk[0] after the pass rather than branching per lane. The transform transposes into lanes and runs the same butterfly twice - rows, then columns - leaving the vectors holding rows to store, so two transposes serve both passes. SSE2 and NEON, with the original scalar code as the fallback (verified bit-exact in a forced-scalar build). SSE2 has no 32-bit low multiply before SSE4.1, so the even and odd 32x32->64 products are combined instead. Instruction count (callgrind, 60-frame CABAC+B stream, -O2): -4.3%; dequantisation 6.3% -> 3.1% and the transform 2.5% -> 1.6% of decoded instructions. Wall clock gains about 2% on a coefficient-dense (CRF 8) stream and is inside run-to-run variance elsewhere on a wide out-of-order desktop core, which absorbed the scalar dependency chains; the instruction reduction is aimed at the narrow in-order cores this frontend also ships to. Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile and 720p B-frame streams, and against openh264 on the left/top/bottom-cropped vectors, in both SIMD and forced-scalar builds. The two new intrinsic helpers were differentially tested against scalar references in isolation (200k random multiplies, 500k random transform blocks) after each showed a lane-ordering fault in first-pass integration. Sanitizer sweep: ASan+UBSan clean over six streams and 450 mutated inputs including a coefficient-dense corpus; LeakSanitizer strict clean; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
Twelve places - intra and inter, luma and chroma, CAVLC and CABAC, 4x4 and 8x8 - each ended with their own nested per-sample loop adding the rounded residual to the prediction and clipping, a byte load and store per sample. They were 6% of decoded instructions between them and identical apart from the plane stride and the block size. They now share rh264_add_residual, which works a row at a time: unpack the predicted bytes into lanes, add, and narrow with saturation, which performs the clip for free. The sum is formed in 32-bit lanes and saturated 32->16 then 16->8 rather than added in 16-bit lanes; the shorter form wraps on the out-of-range residuals a corrupt stream can produce, which would make the vector and scalar paths disagree. SSE2 and NEON, with the original loop as the scalar fallback. Also drops a dead clear of the per-block coefficient array in the CABAC P residual path: it is only read under nz, and the scan scatter writes all sixteen entries from an array rh264_cabac_residual has already zero-filled. Instruction count (callgrind, 60-frame CABAC+B stream, -O2): -6.0%. Decode fps, 300 frames, single thread, x86_64 -O2 -march=native, best of four, same-session A/B: CAVLC I/P 480p: 440 -> 474 (1.08x) CABAC+B 480p: 286 -> 315 (1.10x) CABAC I/P 480p: 279 -> 294 (1.05x) CRF 8 (dense): 260 -> 279 (1.07x) CABAC+B 720p: 179 -> 183 (1.02x) Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile, 720p B-frame and CRF 8 streams, and against openh264 on the left/top/bottom-cropped vectors, in both SIMD and forced-scalar builds. The kernel was differentially tested against the scalar reference in isolation over 400k random blocks spanning both block sizes, unaligned strides and residuals well past the clipping range, which is what caught the 16-bit wrap. Sanitizer sweep: ASan+UBSan clean over eight streams and 550 mutated inputs; LeakSanitizer strict clean; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
The significance and last-flag context derivation ran a four-deep chain of tests per coefficient - category, field flag, 4:2:2 flag - though all of those are fixed for the whole block and only the coefficient index varies. It is now settled once before the scan: a ctxIdxInc map pointer and a context base, with identity and chroma-DC maps added alongside the existing 8x8 ones so every category reduces to a table load. This is a small win, not a large one: -0.6% of decoded instructions (callgrind, 60-frame CABAC+B stream), with wall clock inside run-to-run variance on a desktop core - the compiler was already hoisting most of the invariant tests, and what remains of rh264_cabac_decode's cost is the arithmetic engine's serial dependency rather than context selection. It is committed for the instruction reduction and because the scan loop is now three lines instead of twenty. Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile, 720p B-frame and CRF 8 streams, and against openh264 on the left/top-cropped vectors, in both SIMD and forced-scalar builds. Sanitizer sweep: ASan+UBSan clean over seven streams and 550 mutated inputs weighted to CABAC and 4:2:2/high-profile paths; LeakSanitizer strict clean; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
Bi-predicted blocks were averaged a row at a time by a helper that built its weight, bias, shift and offset vectors on entry, so a 16-row block paid sixteen identical setups and the four call sites each drove it from their own row loop. The weights are constant across a block. The helper becomes a block-level one that sets up once and walks the rows itself, taking source and destination strides; the four call sites lose their loops. The per-row helper is folded in rather than kept as a fallback, so the SSE2, NEON and plain-C paths all hoist alike and no dead per-row version is left behind. Instruction count (callgrind, 60-frame CABAC+B stream, -O2): -2.5%. Decode fps, 300 frames, single thread, x86_64 -O2 -march=native, best of five, same-session A/B: 720p CABAC+B 172 -> 176 (1.02x), 480p CABAC+B unchanged. Output verified bit-exact against ffmpeg on CAVLC I/P, CABAC I/P, CABAC+B main, high-profile, 720p B-frame and CRF 8 streams, and against openh264 on the left/top/bottom-cropped vectors, in both SIMD and forced-scalar builds. Sanitizer sweep: ASan+UBSan clean over seven streams and 580 mutated inputs weighted to the B-frame paths this touches; LeakSanitizer strict clean; TSan clean with four concurrent decoder instances on CAVLC and CABAC+B streams.
LibretroAdmin
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Nine commits continuing the rh264 audit, all performance work — no behaviour change. Every commit is bit-exact against reference decoders and passed a full sanitizer sweep before landing; the per-commit messages carry the detailed rationale and measurements.
Measured effect of the whole series
End-to-end A/B of the branch base against its tip, same binary flags, same machine, same session, best of five runs. 300-frame x264-encoded streams, single thread, x86_64,
-O2 -march=native:Instruction count (callgrind, deterministic, 60-frame CABAC+B stream,
-O2): −18.7%.Two commits are wall-clock-neutral on this desktop and were landed for their instruction and memory-traffic reductions, which should matter more on the narrow in-order cores and bandwidth-limited handhelds RetroArch also ships to. Those commits say so explicitly rather than claiming a speedup.
What the commits do
Entropy decoding
38fdb3f— branchless CABAC engine. The LPS/MPS decision is the decoded bin, so as a branch it mispredicts at the bin entropy; it becomes an all-ones mask selecting range, offset, bin and state.pStateIdx/valMPSfuse into one byte with a per-instance 2×128 transition table built from the spec tables.af96580— significance/last context derivation settled once per block instead of a four-deep test chain per coefficient. Honest result: −0.6%, wall clock in the noise; the compiler was already hoisting most of it.Prediction
111af54— motion compensation runs directly off the reference plane when the interpolation window is in-bounds, instead of gathering a(bw+5)×(bh+5)patch per call whose only purpose is baking in the edge clamp. Border blocks still take the patch.d8d66e3— weighted bi-prediction sets up its weight/bias/shift vectors once per block rather than once per row.Reconstruction
22d5887— 4×4 dequantisation and the inverse transform vectorised (SSE2/NEON). SSE2 lacks a 32-bit low multiply before SSE4.1, so the even/odd 32×32→64 products are combined.53afaff— twelve separate per-sample add-and-clip loops (intra/inter, luma/chroma, CAVLC/CABAC, 4×4/8×8) collapse into one row-at-a-time kernel where saturating narrowing performs the clip for free. Largest single win in the series at −6.0% instructions.Deblocking and output
88607af— P/B chroma deblocking vectorised across the whole edge, with per-lane strength andtc0vectors so the four differing 4-sample segments are handled in one pass.f4bb40f— chroma vertical edges skip the store-side transpose, since onlyp0/q0are written.961ecb7— frame pictures swap plane buffers into the output slot instead of copying a full frame per displayed picture. Field pairs keep the copy.Validation
Every commit was verified bit-exact against libavcodec on six streams (CAVLC I/P, CABAC I/P, CABAC+B main, high profile, 720p B-frame, CRF 8) and against openh264 on left/top/bottom-cropped vectors, in both the SIMD build and a forced-scalar build so the fallback paths are covered too.
Per commit: ASan + UBSan across all streams plus 350–700 mutated/truncated/spliced inputs, LeakSanitizer in strict mode including the decode-error paths, and TSan with four concurrent decoder instances on both entropy modes. C89 verified with
C89_BUILD=1.New SIMD kernels were additionally differential-tested against their scalar references in isolation before integration — 200k random multiplies, 500k random transform blocks, 400k random residual blocks. That caught three lane-ordering and saturation faults that every conformance stream would have passed, including one where adding in 16-bit lanes wrapped on out-of-range residuals from corrupt input and would have made the SIMD and scalar builds disagree.
Notes for review
The riskiest commit is
111af54: aliasing onto the reference plane means the SIMD tails read a few bytes past the filter margin, so the direct path requires the window to sit strictly above the last row and bottom-edge blocks fall back to the gathered patch. Worth a close look, and it is the commit the corrupt-vector fuzz corpus was aimed at.961ecb7changes buffer ownership rather than arithmetic — the working planes and the output slot exchange buffers. The argument for safety is that the DPB already holds the reference copy, every macroblock of the next picture rewrites the working planes, and the displayed slot is excluded from selection while borrowed.Related Issues
None filed; these are follow-ups from the decoder audit rather than reported bugs.
Related Pull Requests
Continues #19304 (correctness and first-round performance work) on the same branch. Self-contained in
libretro-common/formats/h264/rh264.c.