Optimize text shaping pipeline and introduce new shaping API - #550
Merged
Conversation
Introduces a new shaping API (`TextShaper.Shape`) and `ShapedGlyph` output model so callers can get logical-order shaped glyph streams directly from the layout pipeline. Adds `TextOptions.Culture` and rewires GSUB/GPOS language-system selection to use resolved OpenType language tags (including `dflt` handling), with updated tag generation/mappings aligned to HarfBuzz behavior. Internally, replaces per-glyph feature collections with shared bitmask-based feature tracking (`ShapingFeatureMap`) and adds `GlyphSetDigest` prefilters to skip lookups that cannot match current glyph sets. This reduces shaping overhead while preserving correctness. The change also updates glyph metric cloning/caching behavior for lower allocation cost, adds shaping/language regression tests (including HarfBuzz fixture coverage), and adds new shaping benchmarks.
Refactors shaping collections by replacing `IGlyphShapingCollection` with a shared `GlyphShapingCollection` base class, centralizing digest tracking, feature mask operations, and language/tag state. Improves shaping performance by caching GSUB/GPOS feature lookup resolution (with variable-font bypass), adding digest-based lookup/glyph gates, and using range feature assignment to reduce per-glyph overhead. Also switches `GlyphShapingBounds` to an in-place mutable struct, reuses substitution data during positioning, and streamlines glyph-skip logic via packed shaping-class bitmasks.
Adds a new PositionedGlyphMetrics struct that pairs FontGlyphMetrics with its post-shaping positioned state (advance width, advance height, offset). GlyphLayoutData and TextLine now store IReadOnlyList<PositionedGlyphMetrics> instead of IReadOnlyList<FontGlyphMetrics>, allowing metrics instances to remain immutable and shared while layout positions are captured separately.
Refactors text shaping/layout/rendering to stop mutating or cloning `FontGlyphMetrics` per glyph. Positioned advances and offsets now live in shaping/positioned data (`GlyphPositioningCollection` and `PositionedGlyphMetrics`), while rendering and measurement paths explicitly pass `TextRun` and position offset through `Glyph` and `GetBoundingBox`/`RenderOutlineTo` APIs. This also updates synthetic bold/oblique logic to take an explicit nullable `TextRun` context, preserves shared metric immutability, and aligns related layout, GPOS, and tests with the new data flow.
…tching, iterator fast path Inherited from the interrupted session, gated as one checkpoint on the full test suite (4683 passed, 0 failed). Contains: - Per-subtable coverage digests in GSUB/GPOS so application skips virtual probes whose gating coverage cannot contain the current glyph. - Static-lambda Match overloads threading state, removing closure/delegate allocations per ligature and contextual rule attempt. - Chained sequence rules match input/lookahead before backtrack (HarfBuzz order). - SkippingGlyphIterator skipsNothing fast path and inline shaping-class cache. - ShapingFeatureMap single-entry memo for GetMask. - Bidi left-to-right fast path skipping the UAX#9 pass for uniform LTR text. - Extended Enabled-gated ShapingProbe counters (diagnostic only). Benchmarks follow in the next report entry; the complex-script SyllableType work and local dev-env sln/csproj edits from the same session were reverted (no covering benchmark scenario / machine-local paths).
…g or boxed enumeration DefaultShaper accumulated stages in a HashSet rebuilt per section per table per call, with one hash insert per glyph for the per-glyph directional features, and GSUB/GPOS walked it through IEnumerable (interface dispatch + boxed enumerator). Stages are now an insertion-ordered List with linear tag dedup (stage counts are <= ~16, uint compares beat hashing) returned as the concrete List type. Stage application order is now deterministic registration order rather than HashSet implementation order. Tests: 4683 passed, 0 failed. Differential vs HarfBuzzSharp: exact glyph id and advance match on Latin and Arabic scenarios before and after. ShapeTextBenchmark vs previous commit (two runs; allocation figures exact): | Scenario | Mean before | Mean after | Ratio vs HB | Alloc before | Alloc after | |---|---:|---:|---:|---:|---:| | Latin | 30.020 us | 26.816 / 22.606 us | 9.18 -> 8.15 / 6.45 | 28,968 B | 27,352 B (-5.6%) | | Arabic | 31.021 us | 25.816 / 24.408 us | 5.08 -> 5.20 / 5.01 | 24,840 B | 22,040 B (-11.3%) |
… bidi map Two allocation sources with fixed structure removed from every shape call: - GlyphSubstitutionCollection's OffsetGlyphDataPair becomes a readonly struct, eliminating one wrapper object per glyph; the data-shuffling sites (MoveGlyph, Sort) rewrite whole pairs keeping offsets in place. - The codepoint-to-bidi-run map becomes an int[] indexed by codepoint position (-1 = unvisited) instead of a per-codepoint Dictionary insert; consumers already indexed it directly. Tests: 4683 passed, 0 failed. Differential vs HarfBuzzSharp: exact glyph id and advance match on Latin and Arabic scenarios. Benchmark table for this state is carried as the before-column of the immediately following pooled-memory commit.
TextShaper.Shape now rents ShapingScratch from the repository ObjectPool, the hb_buffer_t memory model: the substitution and positioning collections, their shared ShapingFeatureMap, and the GlyphShapingData instances themselves are reused across calls. The positioning collection - sole owner of a pass's final instances after the ownership transfer - returns them to a pool at reset; the substitution collection rents them back, reset in place, as glyphs are added. Scratch storage stays at its high-water mark. Reuse is safe because the public result is materialized by value before the scratch is returned; layout and measure paths are untouched and keep fresh collections pending the shaping pipeline consolidation. Tests: 4683 passed, 0 failed. Pooled-reuse oracle (interleaved Latin/Arabic x3 rounds) reproduces round-0 output exactly; HarfBuzzSharp differential remains an exact glyph id and advance match. ShapeTextBenchmark, 12 iterations, vs previous commit (measured on the thread-static variant; the ObjectPool refit's separate run showed byte-identical allocations, with means noise-dominated - HarfBuzz's own times co-moved ~+50% - and pool overhead of two interlocked ops per call not resolvable above that noise): | Scenario | Mean | Ratio vs HB | Allocated | Gen1 | |---|---:|---:|---:|---| | Latin | 17.226 -> 16.103 us (-6.5%) | 7.59 -> 7.22 | 24,144 -> 9,208 B (-61.9%) | 0 | | Arabic | 18.148 -> 16.897 us (-6.9%) | 4.78 -> 3.83 | 18,400 -> 9,184 B (-50.1%) | 0 |
TextShaper now owns the whole shaping pipeline: font-run itemization (BuildTextRuns), bidi analysis, glyph population and font fallback (DoFontRun), bidi mirroring, and GSUB/GPOS orchestration (ShapeText), relocated verbatim from TextLayout into the TextShaper.Pipeline partial. TextLayout retains only composition, line breaking, and the layout walks. Moved members are internal on the public class, so the public API surface is unchanged. Mechanical code motion: tests 4683 passed, 0 failed; pooled-reuse and HarfBuzzSharp oracles exact; benchmark allocations byte-identical (9,208 / 9,184 B) with means inside the run's noise band.
TextShaper.ShapeText is now the single pooling site: it rents the pipeline scratch, shapes, copies the result out by value, and returns the scratch before any caller sees the result. ShapedText becomes a run table holding run-constant state once (font, point size, text run, placeholder bidi run) plus parallel per-glyph info and position arrays of pure numbers - no metrics, collection, or pooled references survive shaping. Line composition resolves metrics instances from the owning font's cache by the same arguments shaping used; placeholder metrics are recreated from the run entry via PlaceholderGlyphMetrics.Create. TextBlock, TextRenderer, and Shape all call the static shaper API with no pooling knowledge, and the layout path shapes through the pooled pipeline for the first time. Tests: 4683 passed, 0 failed. Pooled-reuse and HarfBuzzSharp oracles exact. ShapeTextBenchmark, 12 iterations, vs previous commit (noisy run: HarfBuzz's own means moved ~+60%; ratios flat within error): | Scenario | Ratio vs HB | Allocated | |---|---:|---:| | Latin | 7.22 -> 7.15 | 9,208 -> 11,984 B | | Arabic | 3.83-4.80 band | 9,184 -> 10,952 B | The allocation increase is the inert result arrays (~24 B info + 24 B pos per glyph plus the run table) that replace escaped references on the Shape path; the layout path sheds its per-offset List allocations in exchange. The arrays are the flat storage the pipeline itself adopts next, at which point the copy-out collapses.
The hb_buffer model applied end to end. One ShapingBuffer type replaces GlyphShapingCollection, GlyphSubstitutionCollection, and GlyphPositioningCollection: glyph state is a struct record (GlyphShapingData, codepoint index folded in) living in one growable array, with a parallel metrics stream seeded after substitution and truncate-only resets that keep storage at the pass high-water mark. The scratch owns two buffer instances - a per-font-run substitution workspace and the accumulated positioning result - distinguished by an explicit role that also replaces the shapers' phase dispatch, which previously pattern-matched on collection type. Element access is by interior reference: buffer indexers return ref, call sites mutate through the indexer expression or explicit ref locals, and helper methods take the record by ref so the shaping-class cache writes land and no ~120-byte copies occur per call. The conversion surfaced one real defect class - reassigning a ref local from the buffer stores a whole record through the reference where the old class code merely rebound a variable - found by bisecting a corrupted Arabic mark test down to FixMarkAttachment, and fixed there and in the fraction scanners. Two source-convention tests now fail the build on value-copy element bindings and on buffer type-test dispatch so both silent-corruption classes are compile-gated. Tests: 5565 passed, 0 failed (includes the new convention scans). Pooled-reuse and HarfBuzzSharp differential oracles exact, including the lam+shadda mark anchoring case used to isolate the rebind defect. ShapeTextBenchmark, 12 iterations, low-noise runs, vs this morning's clean baseline and the previous commit's allocations: | Scenario | Mean | Ratio vs HB | Allocated | |---|---:|---:|---:| | Latin | 17.226 -> 15.483 us (-10.1%) | 7.59 -> 6.96 | 24,144 -> 8,296 B (-65.6%; prev commit 11,984) | | Arabic | 18.148 -> 15.651 us (-13.8%) | 4.78 -> 4.15 | 18,400 -> 7,856 B (-57.3%; prev commit 10,952) |
ShapeTextBenchmark gains a Devanagari scenario (Noto Sans Devanagari, conjunct- and matra-heavy text) so complex-script shaping changes are measured, not assumed. HarfBuzzDifferentialTests shapes Latin, Arabic, and Devanagari cases through both engines and requires exact glyph id and advance equality; it is the standing correctness gate for shaping performance work. All six cases pass at this commit.
Replace the per-glyph IndicShapingEngineInfo/UniversalShapingEngineInfo heap objects and string syllable types with a SyllableInfo struct embedded in GlyphShapingData, matching how HarfBuzz keeps indic_category(), indic_position(), and syllable() in hb_glyph_info_t vars. Classification now allocates nothing and every syllable-type comparison is an integer compare. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari. | Scenario | Metric | Before | After | |------------|-----------|-----------|-----------| | Devanagari | Allocated | 313,024 B | 299,840 B | | Devanagari | Ratio | 9.63x | 7.43x | | Devanagari | Mean | 112.13 us | 91.93 us | | Latin | Allocated | 8,296 B | 8,296 B | | Arabic | Allocated | 7,856 B | 7,856 B | Means were measured under daytime machine load; the HarfBuzz control moved too, so the exact allocation counts and the ratio are the comparable metrics. Latin and Arabic allocations are byte-identical, confirming the shared shaping path is untouched.
Replace the Indic shaper's would-substitute probe, which ran the full GSUB feature-application machinery on a temporary three-glyph buffer, with a direct would-apply query on the feature's lookups: every GSUB lookup subtable now answers whether a glyph id sequence would trigger it by matching its coverage, components, or context rules without substituting. Queries are digest-gated per lookup and per subtable, take raw glyph ids, disallow outside context for new-spec scripts other than Malayalam, and include the vattu variants feature when classifying below-base consonants. The per-call probe buffer and glyph-data scratch array are gone. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari. | Scenario | Metric | Before | After | |------------|-----------|-----------|-----------| | Devanagari | Mean | 91.93 us | 56.23 us | | Devanagari | Ratio | 7.43x | 6.25x | | Devanagari | Allocated | 299,840 B | 288,232 B | | Latin | Allocated | 8,296 B | 8,296 B | | Arabic | Allocated | 7,856 B | 7,856 B | Devanagari RatioSD 0.10 this run; Latin mean 15.55 us matches its low-noise baseline and shared-path allocations are byte-identical.
The generated UniversalShapingData.Decompositions property was expression-bodied, so every access rebuilt the full dictionary and all of its component arrays. The Indic and universal shapers consult it once per glyph when checking for split matra decompositions, which made dictionary construction the single largest cost in complex-script shaping. The property now caches its value; the generator template is fixed to emit the cached form. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari. | Scenario | Metric | Before | After | |------------|-----------|-----------|----------| | Devanagari | Mean | 56.23 us | 21.19 us | | Devanagari | Ratio | 6.25x | 2.34x | | Devanagari | Allocated | 288,232 B | 11,656 B | | Latin | Allocated | 8,296 B | 8,296 B | | Arabic | Allocated | 7,856 B | 7,856 B | Devanagari RatioSD 0.04; Latin 15.50 us and Arabic 16.25 us match their low-noise baselines with byte-identical allocations.
Attribute substitution-phase time and allocation to shaper creation, planning (preprocessing, features, postprocessing, assignment), stage pre/post actions, and stage application. These counters located the decomposition dictionary rebuild; all are inert unless the probe is enabled.
BuildTextRuns counted graphemes twice for the common no-runs path; it now reuses the first count. GetGraphemeCount also drove the full cluster enumerator, which computes terminal width, emoji, and flag metadata for every scalar that a count never reads. The enumerator gains an internal count-only mode that walks the same UAX 29 boundary rules and skips all cluster metadata, so counting stays a single implementation of the boundary algorithm. A count-only walk is 11.2 ns per char across scripts, down from up to 26. Gate: 5,577/0 tests. | Scenario | Mean before | Mean after | Ratio | |------------|-------------|------------|---------------| | Latin | 15.50 us | 13.84 us | 6.78x > 6.14x | | Arabic | 16.25 us | 14.49 us | 4.15x > 3.82x | | Devanagari | 21.19 us | 20.41 us | 2.34x > 2.25x | Allocations byte-identical in all scenarios; RatioSD at or below 0.14. Includes a probe phase around the counting call.
The shaping plan registered ltra/ltrm or rtla/rtlm with a separate single-glyph registration for every glyph, resolving the feature mask per glyph per pass. Direction features now register once per consecutive same-direction span, matching how the shape plan enables a segment direction's features once and producing identical per-glyph masks. Gate: 5,577/0 tests. | Scenario | Mean before | Mean after | Ratio | |------------|-------------|------------|---------------| | Latin | 13.84 us | 12.08 us | 6.14x > 5.24x | | Arabic | 14.49 us | 13.52 us | 3.82x > 3.44x | | Devanagari | 20.41 us | 19.90 us | 2.25x > 2.12x | Allocations byte-identical in all scenarios; RatioSD at or below 0.08.
Byte sampling and per-feature detail cost far more than timestamping and were inflating every enclosing phase's measured time. Both are now switchable so phase timings can be read without them.
Substitution and positioning applied each stage feature separately, in feature registration order, resolving and walking the buffer per feature. Stage actions are the only true synchronization points, so stages between actions now form one group: the group's lookups merge into a single list sorted by lookup index, the order the specification defines within an application pass, and a lookup registered by several features applies once with their glyph masks combined. Merging reuses a scratch list on the pooled buffer, so application still allocates nothing. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari; application order changes are covered by both. | Scenario | Mean before | Mean after | Notes | |------------|-------------|------------|------------------------------| | Latin | 12.08 us | 11.56 us | clean run, RatioSD 0.10 | | Arabic | 13.52 us | 13.92 us | machine loaded, HB elevated | | Devanagari | 19.90 us | 20.46 us | within cross-run spread | Allocations byte-identical in all scenarios. Arabic and Devanagari rows were measured under active machine load (their HarfBuzz controls read 40-60% above idle baselines); both sit inside the spread identical code has shown across runs.
Swept every file on the branch for members declared internal inside internal types and made them public, matching the codebase convention that an internal type's cross-type surface is uniformly public. Members kept internal are those on public types, where the modifier hides real API surface, and overrides bound to a public base class's internal abstract members.
Seeding the positioning buffer resolved every glyph's metrics through the font's concurrent dictionary cache, paying a hash and probe per glyph per shape. The buffer now fronts that resolver with a 256-slot direct-mapped cache whose tag packs the same key fields, so a repeat lookup costs one load and one compare. The cache needs no synchronization because a pooled buffer is exclusively owned for the duration of a shaping pass; seeding from a different font clears it. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari. | Scenario | Mean before | Mean after | |------------|-------------|------------| | Latin | 11.66 us | 11.35 us | | Arabic | 13.26 us | 12.93 us | | Devanagari | 19.93 us | 19.74 us | All rows clean (Fonts StdDev at or below 0.13 us); allocations byte-identical in all scenarios.
Every glyph record carried a sixteen byte bidi run that only inline placeholders ever set and only the copy-out ever read. The run now lives in a small list on the buffer keyed by codepoint offset, which is stable for placeholders because they shape in isolated single-glyph runs. This shrinks the record every walk, copy, and seed touches by sixteen bytes. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari. | Scenario | Mean before | Mean after | Ratio | |------------|-------------|------------|---------------| | Latin | 11.35 us | 11.03 us | 4.94x > 4.74x | | Arabic | 12.93 us | 13.19 us | 3.40x > 3.33x | | Devanagari | 19.74 us | 19.69 us | ~2.1x > 2.16x | All rows clean; the Arabic mean moved with an elevated HarfBuzz control while its ratio improved. Allocations byte-identical in all scenarios.
SyllableInfo carried five ints. The syllable positional classes are now ordinals whose order is the visual order, with zero reserved as the unassigned sentinel, so the classification packs into byte lanes of one word: number, type, category, position, and universal engine category. The trie already stored positions zero-based; the converters map them onto the ordinal enum directly instead of shifting them into flags. The final reorder placed the reph for after-subjoined scripts with a bitmask over three positional flags, the only flag-style use in the repository; it now tests the three members explicitly, preserving the exact prior semantics under both numberings. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari and the Bengali reph cases. Glyph record shrinks from ~120 to ~108 bytes. | Scenario | Mean before | Mean after | |------------|-------------|------------| | Latin | 11.03 us | 11.08 us | | Arabic | 13.19 us | 13.42 us | | Devanagari | 19.69 us | 19.60 us | All rows clean and inside the cross-run spread of identical code; allocations byte-identical in all scenarios.
Six bool auto-properties become single bits of a packed flags field, narrowing the glyph record; the property surface is unchanged. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari. | Scenario | Mean before | Mean after | |------------|-------------|------------| | Latin | 11.08 us | 10.89 us | | Arabic | 13.42 us | 13.35 us | | Devanagari | 19.60 us | 19.63 us | All rows clean; allocations byte-identical in all scenarios.
The text direction stores as a byte and the shaping-class cache key becomes the cached glyph id plus a validity flag bit, with the property surfaces unchanged. The flag encoding also corrects a latent defect: a default record previously carried cache key zero, which read as a valid cached class for glyph id zero, while field initializers only run when a constructor does. A default record now reports an invalid cache. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari. | Scenario | Mean before | Mean after | |------------|-------------|------------| | Latin | 10.89 us | 10.96 us | | Arabic | 13.35 us | 13.38 us | | Devanagari | 19.63 us | 19.19 us | All rows clean; allocations byte-identical in all scenarios.
The packed flags and byte lanes introduced by the record shrink used raw bit numbers; every bit and lane now has a named constant so the property bodies read as intent. All single-line summary documentation across the branch's files is expanded to the multi-line form. Gate: 5,577/0 tests; no functional change.
Every glyph record carried a TextRun reference. Records now store a ushort index into the buffer's run list, assigned to both of a pass's buffers before population so indices agree when records seed across them. Consumers resolve the run through the buffer; the shaped run table keeps the real reference for downstream layout. The record is now free of object references, so the garbage collector no longer scans the pooled glyph arrays, and the record shrinks further toward the flat info layout. Placeholders record the loop position as their run index; the populate tracker can lag it between runs, which the placeholder tests caught. Gate: 5,577/0 tests including exact HarfBuzz differential on Devanagari and all placeholder cases. Allocations fell slightly in every scenario (8,296 to 8,264, 7,856 to 7,824, 11,656 to 11,624 bytes). Means were measured under sustained machine load; ratios, the load-independent metric, read at or better than baseline in both runs (Latin 4.12 vs 4.83, Arabic 3.32 vs 3.50, Devanagari 2.05 vs 2.13). A clean-machine validation follows at the next idle window.
Streaming nested lookups moved two pieces of bookkeeping into paths hot enough to measure: Devanagari lost five percent of its shaping time. The substitution driver had begun deriving the segment's end from the buffer on every glyph it gated, trading a local comparison for two property reads and a subtraction in the innermost loop of the pass. It holds the end as a plain index again and moves it only when the input side actually lengthens, which an in-place mutation and the room a rewind opens both do by the same amount, so the bound stays correct without being recomputed. Contextual application also rebased its matched positions on every rule, though the frames it converts between coincide until a lookup in the pass changes a length, which is most passes and all of Latin. The conversion is skipped when the two sides are level. Profiles before and after are the same shape, with no cost left at either site: substitution application falls from 11.3 to 9.1 percent of samples and sequence matching from 3.2 to 2.5. Measured against the commit before the streaming leg, interleaved, four pairs per scenario: Latin and Arabic at parity, Devanagari 1.5 percent faster. Suite 5,605/0; all differential rows against HarfBuzzSharp unchanged; zero steady-state allocation preserved.
Running HarfBuzz's own in-house shaping corpus against this library found character mapping, not shaping, to be the largest source of disagreement. Three faults, all in cmap. Format 13 was not implemented. It maps a range of characters onto a single glyph, which is how a font answers for characters it has no artwork for, and a font whose only subtable uses it mapped nothing at all: every character came back as notdef. The corpus category built on such a font, 3,825 cases, failed entirely and now passes entirely. Lookup consulted every subtable in turn and took the first answer. A font that carries several encodings intends exactly one of them; the others exist for readers that cannot use it. Consulting them in turn resolves characters through tables the font does not intend for them, which is how a symbol font's characters were being answered from its Macintosh byte table. Selection now picks one subtable in the order the specification's implementations agree on: the symbol encoding, then the encodings that reach beyond the basic multilingual plane, then those that do not, and the Macintosh encoding only when a font offers nothing else. A symbol font addresses its glyphs one private use page up from the characters that reach them, so a lookup that misses is retried there. The reported character coverage follows the same subtable, so a font no longer advertises characters it will not resolve. The sample font's coverage drops from 257 characters to the 7 it maps, which is what HarfBuzz reports for it. Corpus: 10.82 percent of executed cases matching, to 89.49 percent (4,351 of 4,862). Suite 5,605/0 -> 5,607/0.
Add source-derived normalization, script planning, Arabic preprocessing, dedicated Khmer shaping, and GPOS attachment corrections. Add generated shaping data, the pinned reference submodule, and corpus coverage. Correctness (2026-07-27; glyph ID, x/y offset, and x/y advance): | Metric | Before | After | | --- | ---: | ---: | | In-repo corpus failures | 203 | 134 | | Scratch exact cases | 4658/4862 (95.80%) | 4727/4862 (97.22%) | | Khmer exact cases | 89/117 | 117/117 | Verification: - Release non-corpus tests: 5640/5640 passed. - Release corpus tests: 4716/4850 passed. - Reused-buffer allocation tests: 3/3 passed. - Generated output idempotence: no unintended hash changes. The performance gate was not run because active vstest processes and non-idle CPU made a valid measurement impossible.
Scratch corpus comparison: glyph ID + x/y offset + x advance, 2026-07-27. | Corpus cause | Before | After | | --- | ---: | ---: | | Khmer mark order | 24/25 failed | 0/25 failed | | Khmer miscellaneous | 4/92 failed | 0/92 failed | | kern format 2 | 2/3 failed | 0/3 failed | | per-script kern fallback | 3/12 failed | 0/12 failed |
Scratch corpus comparison (glyph ID, X/Y offset, and X advance; 2026-07-27): Metric Before After Output mismatches 150 67 Wrong glyphs 97 55 Placement only 53 12 Arabic fallback shaping is 13/13 exact. The generator source and generated Arabic tables are committed together.
| Contract | Before | After | |---|---|---| | Shape | Forced one direction across the buffer | Resolves and visually orders one unwrapped logical line | | ShapeRun | Not exposed | Preserves explicit single-run shaping and output order | | Layout reordering | Allocated linked run nodes | Uses the shared allocation-free L2 range reversal | | Scratch corpus | 67 mismatches of 4862 executed | 67 mismatches of 4862 executed |
Keep script shaping independent of page orientation, cover forced and mixed vertical layouts, and document the shared UAX #9 L2 reordering utility. Before | After Vertical layout cases: 4 failed | 4 passed All layout-mode golden cases: 2 passed, 4 failed | 6 passed
Implement the complete Universal shaping categories, syllable grammar, feature stages, and reordering behavior. Correct Indic normalization recursion, fallback replacement of consumed source records, vertical shaping selection, positioning ownership, and legacy kerning behavior. Measured 2026-07-27: | Harness and comparison rule | Before | After | | --- | ---: | ---: | | In-repo HarfBuzzCorpusTests failures, glyph id plus x/y offset and x advance | 114 | 104 | | Scratch corpus mismatches of 4,862 executed, glyph id plus x/y offset and x advance | 66 | 56 | | Full-suite non-corpus failures | 0 | 0 |
In-repo corpus comparison (glyph ID, x/y offset, and advance): | Result | Before | After | | --- | ---: | ---: | | Failed | 67 | 62 | | Passed | 4783 | 4788 | Non-corpus Release tests: 5687 passed, 0 failed. Generated Unicode resources are byte-idempotent after regeneration.
Add reverse chaining, deterministic random alternates, script-extension itemization, fallback positioning corrections, and Myanmar behavior required by the shaping corpus. Expose explicit script selection across shaping buffers, text options, and text runs, including Qaag/Zawgyi. Allow text runs to override culture and keep language-system plans isolated. Qaag/API corpus verification (glyph id + x/y offset + x advance): | Result | Before | After | | --- | ---: | ---: | | Executed | 4771 | 4772 | | Passed | 4745 | 4746 | | Loader failures | 26 | 26 | | Glyph or placement mismatches | 0 | 0 | Focused TextShaper and TextOptions tests: 58 passed, 0 failed. Generated data was regenerated with unchanged hashes for existing outputs.
Use reference identity before comparing culture names at substitution and fallback-positioning run boundaries.
Verification (Release, net8.0):
Before After
Full-suite failures 32 26
Targeted visual failures 6 0
Corpus loader failures 26 26
Shape resolves and reorders hard-break-delimited paragraphs independently, while ShapeRun preserves its single directional-run contract.
Release net8.0 EventPipe profile (1 launch, 3 warmups, 5 measurements): | Scenario | Before | After | Allocated | | --- | ---: | ---: | ---: | | Latin | 3.963 us | 3.670 us | 0 B | Correctness: | Suite | Result | | --- | --- | | TextShaperTests | 39 passed | | HarfBuzz corpus | 4,851 passed; same 26 malformed-font loader failures |
Reuse contextual match storage, batch buffer moves, avoid unnecessary run analysis, and hoist invariant shaping work while preserving lookup behavior. Release net8.0 Config.Gate. Before is the mean of two clean pre-optimization samples; after is the first clean checkpoint sample. A second identical-code sample is still required before a final delta claim. | Scenario | Before | After | | --- | ---: | ---: | | Latin | 3.8981 us | 3.6065 us | | Arabic | 6.0792 us | 5.9901 us | | Hebrew | 5.4364 us | 5.4743 us | | Thai | 5.2067 us | 5.1374 us | | Hangul | 1.2455 us | 1.1988 us | | Devanagari | 9.7921 us | 9.4704 us | | Khmer | 6.1475 us | 5.5747 us | | Myanmar | 40.8183 us | 29.4615 us | | MyanmarZawgyi | 1.0594 us | 1.0013 us | | Balinese | 3.2379 us | 3.0510 us | All rows allocated 0 B. TextShaperTests: 39 passed. HarfBuzz corpus: 4,851 passed with the same 26 malformed-font loader failures.
Cover Latin, Arabic, Hebrew, Thai, Hangul, Devanagari, Khmer, Myanmar, Zawgyi, and Balinese side by side with HarfBuzz. The net8.0 Config.Gate run completed all 20 rows with zero managed allocation.
Retain each directional run's shaped glyphs in visual order and consume them as contiguous slices of block-owned storage: line entries hold index ranges, reorder as whole units after line breaking, and render pen plus offset then pen plus advance, so no stage rearranges glyphs inside a run. Replace materialized line-break candidates with a lazy, resumable UAX #14 cursor over retained source text, measuring line fills incrementally. Substitute empty-outline bounds from the positioned advance so invisible default ignorables occupy no measured space. Update test expectations pinned to pre-parity shaping output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prerequisites
Description
This PR makes the shaping engine a first-class public API. Callers can fill a reusable
TextShapingBuffer, shape it against aFont, and pass the same positioned result directly to measurement or rendering without an intermediate glyph-run allocation or caller-side scaling.The result is both accurate and fast. The steady-state shaping path allocates zero managed bytes in every benchmarked script. On .NET 10, seven of the ten cross-script workloads are within 12% of HarfBuzz or faster, including Devanagari at 0.83x, Balinese at 0.88x, and Hebrew at 0.98x HarfBuzz time.
TextShaper.Shapetreats its input as unwrapped logical lines separated by hard breaks. It resolves mixed-direction text independently for each line and preserves the line boundaries needed by direct measurement and rendering.TextShaper.ShapeRunshapes one directional run selected by the caller. It applies the buffer's stated direction to the whole run and returns glyphs in reading order without performing whole-line bidirectional resolution.TextShapingBuffercarries direction, language, and an optional script override. Feature-tag overloads are available for both shaping contracts.TextOptionsandTextRunprovide whole-text and per-run script and culture overrides. Fonts carry their size and variable-axis settings through the existingFontandFontVariationAPIs.Font.Sizeand preserve the shaping coordinate system's Y-up orientation. DPI scaling, device-space Y inversion, and origin placement occur only at the measurement or rendering boundary.TextMeasurerandTextRendereraccept either aTextShapingBufferor positioned glyph-ID and point spans. This provides the same consumption model as positioned-glyph APIs in native backends without forcing callers to allocate or adapt the shaped output.GlyphOptionswhen measuring or rendering a shaped buffer; the reusable buffer itself remains independent of any font.TextLayoutconsumes the same logical-order shaping pipeline. It applies the shared Unicode Bidirectional Algorithm L2 reordering only after line breaking, when the visual extent of each wrapped line is known.TextShapingBufferreaches zero managed allocation per shape. The focused allocation tests and every all-shaper Gate benchmark scenario enforce that contract.Shaping internals
The OpenType layout engine was reworked against the pinned HarfBuzz reference and corpus. Differential tests compare glyph ID, x/y offset, and x advance:
kernand AATkerxformat 2 class tables use their specified subtable-relative offsets, and legacy kerning runs only under the font-table precedence rules.Verification
HarfBuzzCorpusTestspassed 4,851/4,851 locally available cases. The corpus and differential suites compare glyph IDs, advances, and offsets directly against a HarfBuzz font configured to the same scale; there are no assertion-side conversions hiding precision differences.TextShaperandHarfBuzzDifferentialTestspassed in Release.Config.Gatesamples measured zero managed bytes allocated per shape in all ten script scenarios.Shaping performance
ShapeTextBenchmarkshapes the same stated run and font bytes through SixLabors.Fonts and HarfBuzzSharp while reusing one buffer per side.Config.Gateuses three launches, five warmup iterations, and 15 measured iterations per row. Times are arithmetic means in microseconds; ratio is SixLabors.Fonts divided by HarfBuzzSharp. Every scenario measured zero managed bytes allocated per shape.The .NET 10 results are particularly strong: SixLabors.Fonts is faster than HarfBuzz in three workloads and within 12% in another four.
.NET 8.0.29
.NET 10.0.10
The final public coordinate-scaling path was also checked with a short Latin run: SixLabors.Fonts measured 3.411 μs against HarfBuzz at 2.281 μs (1.50x), with zero managed allocations on both sides. This is consistent with the full 1.47x Latin Gate result.
Positioned shaped-run measurement
MeasureShapedRunBenchmarkcompares equivalent work after shaping: exact tight bounds for positioned glyphs. SixLabors.Fonts measures theTextShapingBufferdirectly. The Skia side retrieves each glyph's bounds and unions them at its shaped position becauseSKTextBlob::bounds()is deliberately conservative rather than an exact tight-bounds API.The current .NET 8 sample measures three scripts:
The time ratio for shaping plus exact positioned bounds narrows as shaping complexity grows — 1.75x for Latin, 1.51x for Arabic, and 1.07x for Devanagari — while the reusable SixLabors.Fonts path remains allocation-free in every scenario.
End-to-end text measurement
The checked-in
MeasureTextBenchmarkalso records the broader progress behind this work. Against the original measurements in #268, the three comparable single-line cases are now 7.3-11.9x faster:aHello worldThe quick brown fox jumps over the lazy dogThe issue values were originally reported in nanoseconds and are converted here to microseconds for a direct unit comparison. This historical comparison is directional because the original results used .NET 6 on different hardware. Against the issue's later update, the current results are still 3.48-5.94x faster.