Flat token buffer, part 1: parser cursor, lexer emission, flat mbe transcription, view-backed macro args

[xmakro]

Core slice of #57, re-scoped after measurement. An earlier version of this PR carried only the cursor, lexer emission, and view-backed arguments; a 5-way A/B (below) showed that cut regresses tt-recursive workloads, so flat mbe transcription is now part of this PR. #57 keeps the full per-step history.

What is in this part

• The flat token cursor replaces the tree-walking cursor inside the parser. bump is an index increment, lookahead is direct indexing, parser snapshots and lazy capture replay share the buffer through refcount bumps, and whole delimited sequences skip in one step through the open/close match table. Skipped invisible delimiters stay in the buffer as entries (filtered on consumption), keeping tree-level lookahead and nesting-depth queries byte-exact.
• The lexer emits the flat buffer directly, building through FlatSink like every other producer, so primary parses never construct a token tree. The few cold callers that need a tree rebuild it from the buffer.
• mbe transcription writes expansion output through the same sink, so expansion output parses straight from the buffer with no per-expansion flatten. Captured tt fragments are buffer slices (a refcount bump at capture) and are spliced into expansion output with a depth and match-table rebase; captured non-tt fragments stay on the tree path for now.
• Macro invocation arguments are lazy flat views. parse_delim_args captures arguments as a view of the buffer, parser_from_cx and DeriveProcMacro::from_tts parse straight from the view, and the pre-expansion KeywordIdents lint scans flat entries. Attributes and macro definition bodies stay eager so long-lived nodes do not pin token buffers.
• All the hardening and regression tests from Flat token buffer: lexer-fused parsing, flat mbe transcription, view-backed macro args, sink-replayed nonterminals [-1.14% incl tt-heavy, jemalloc, noise-controlled] #57 that belong to these layers: length-prefixed stable hashing, restored Arc pointer-equality fast paths, the u32::MAX entry guard, lookahead fixes at skipped invisible delimiters, sink-only buffer production, the TokenStream size and iteration fixes, and the six regression tests.

Not in this part: the nonterminal sink replay (#57 commit 12, the next slice: -0.51% geomean on its own and it halves the token-stream-stress residual below) and bridge-side flat emission for proc-macro output (later follow-up).

Why exactly this cut: the 5-way A/B

Measured with the #57 protocol (stage2, CI LLVM, lto=thin, jemalloc, instructions:u, Check+Debug, Full, 3 iterations; same-commit rebuild control pair: geomean -0.003%, worst cell 0.04%, zero cells at the 0.25% threshold):

configuration	geomean vs base	tt-muncher check	deep-vector check	verdict
cursor + lexer only	+2.05%	+23.9%	-1.6%	capture rebuild and arg flatten dominate
cursor + lexer + views	+0.82%	+10.2%	-1.8%	per-expansion flatten of transcription output dominates
this PR (adds flat transcription)	-0.63%	-2.0%	-3.4%	10 cells improved at the 0.25% threshold, none regressed among primaries

The parser reading flat while any other component produces trees creates a conversion tax at the boundary, paid once per expansion level on tt-recursive code. Cursor, lexer, views, and flat transcription are one perf unit; this PR is that unit. The only wrong-way cells are token-stream-stress +0.72% check / +0.40% debug (proc-macro passthrough still pays one flatten per expansion); the nonterminal replay slice more than halves that residual and bridge-side flat emission is the named fix for the rest. The -0.63% geomean also matches the arithmetic from #57's per-commit numbers (-1.14% for the full stack minus -0.51% for the replay commit), so the two measurement rounds cross-validate.

The benchmarked tree for this cut is byte-identical to #57's eleventh commit, which is the state the #57 test battery already exercised.

The old cursor becomes a debug shadow

Instead of deleting the dead TokenCursor, debug builds keep it as a shadow. Parser::new_from_flat rebuilds the token tree from the cursor's own buffer and steps the old cursor alongside the flat one; every bump asserts the two yield identical (Token, Spacing) pairs, and parse_token_tree keeps the shadow in lockstep across the skip-to-group-end fast path. With flat transcription in the tree this now also differentially validates expansion output: the tree the shadow walks is rebuilt from the transcribed buffer, so a splice or match-table bug becomes a divergence on the next bump. Release builds compile all of it out (the Parser size assertion is release-only now). The cursor and shadow get deleted in the final slice once the stack has baked.

Correctness

• x check compiler green at both commits
• unit suites green at stage 1 with debug assertions on (shadow active): rustc_parse 69/69, rustc_ast, rustc_expand, including the six flat-buffer regression tests
• tests/ui/macros + tests/ui/proc-macro + tests/ui/parser: 1673 passed, 0 failed, 14 ignored, stage 1 with debug assertions on, so the shadow cross-checked every parse in those suites, including every rebuilt transcription buffer
• shadow positive control (previous round of this PR, same shadow code): desyncing the shadow by one token makes the rustc_parse suite fail 16 tests with the divergence assert; the check does fire