Skip to content

Eliminate need for forward type declarations in recursive datatypes#443

Open
joehendrix wants to merge 9 commits intomainfrom
jhx/ddm_preregister
Open

Eliminate need for forward type declarations in recursive datatypes#443
joehendrix wants to merge 9 commits intomainfrom
jhx/ddm_preregister

Conversation

@joehendrix
Copy link
Contributor

@joehendrix joehendrix commented Feb 18, 2026

Summary

Mutually recursive datatypes previously required users to manually write
forward type declarations before a mutual block. This PR eliminates
that boilerplate by having the elaborator automatically discover and
pre-register type names before elaborating the block's children.

Key changes

  • Automatic pre-registration replaces forward declarations. The
    @[preRegisterTypes(scope)] annotation on command_mutual triggers
    two-phase elaboration: Phase 1 partially elaborates each child to
    extract type names and params, then pre-registers them in the
    GlobalContext; Phase 2 elaborates the full block with mutual
    references already resolved.
  • GlobalContext simplified. The DeclState enum
    (.forward / .defined) and its bookkeeping are removed. define
    now requires a non-membership proof, making illegal states
    unrepresentable. defineChecked takes a preRegistered flag to
    handle both fresh definitions and pre-registered confirmations.
  • Error propagation instead of silent failure.
    extractConstructorInfo and helpers return Except String instead
    of silently returning empty arrays. Template expansion uses an atomic
    buildAndDefine that checks freshness and defines in one step.
  • Template expansion validated during elaboration.
    evalBindingSpec's .datatype branch calls expandFunctionTemplates
    to catch constructor and template function name collisions early via
    logError, rather than panicking in addCommand. Constructor names
    are no longer required to be globally unique — only the generated
    template function names must be distinct.
  • Elaboration refactors. Extracted elabSyntaxArg, resultContext,
    extractParamNames, and scopeSepFormat from the monolithic
    runSyntaxElaborator. Lifted extractConstructorInfo out of the
    GlobalContext namespace (it only depends on DialectMap).
  • O(1) argument lookup via argElabIndex. A precomputed
    Array (Option Nat) maps each argLevel to its position in
    argElaborators, replacing linear searches.
  • Translation layer adapted. translateMutualBlock allocates
    placeholder entries on the fly when a datatype isn't already
    pre-registered, rather than requiring forward declarations.

Test changes

  • StrataTest/DDM/MutualDatatypes.lean (new): DDM-level test
    dialect with function templates (perConstructor tester,
    perField accessor). Tests mutual block visibility, three-way
    recursion, empty blocks, multi-constructor template expansion, and
    negative cases (undefined type, duplicate type, type clash, duplicate
    constructor name collisions, reused constructor name collisions).
  • StrataTest/Languages/Core/Examples/MutualDatatypes.lean (new):
    Core-level integration tests verifying tester functions, accessor
    functions, equality, polymorphic havoc, and complex Stmt/StmtList
    patterns on mutually recursive types.
  • StrataTest/Languages/Core/TestASTtoCST.lean (new):
    Round-trip tests for AST-to-CST translation.
  • Removed all forward type declarations from existing test files.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@joehendrix joehendrix changed the title Replace forward type declarations with automatic pre-registration WIP: Replace forward type declarations with automatic pre-registration Feb 18, 2026
@joehendrix joehendrix force-pushed the jhx/ddm_preregister branch 2 times, most recently from 4971e76 to 64accb0 Compare February 19, 2026 23:28
@joehendrix joehendrix changed the title WIP: Replace forward type declarations with automatic pre-registration Replace forward type declarations with automatic pre-registration Feb 26, 2026
@joehendrix joehendrix force-pushed the jhx/ddm_preregister branch 2 times, most recently from ab52636 to ad51d1d Compare February 26, 2026 18:04
@joehendrix joehendrix marked this pull request as ready for review February 26, 2026 20:49
@joehendrix joehendrix requested a review from a team February 26, 2026 20:49
Copy link
Contributor

@MikaelMayer MikaelMayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, well-motivated PR that removes a real source of user friction. The two-phase elaboration design in elaborateWithPreRegistration is sound, the GlobalContext simplification is a clear win, and the removal of DeclState/forward declarations is thorough — no dangling references remain.

One minor inconsistency between extractParamNames and the inline code in runSyntaxElaborator (.tvar handling) is worth a look. The rest is solid.

I'd approve after the one question below is addressed.

let newBindings := tpCtx.bindings.toArray.extract inheritedCount tpCtx.bindings.size
let names := newBindings.filterMap fun (b : Binding) =>
match b.kind with
| .type _ [] _ => some b.ident
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extractParamNames matches both .type _ [] _ and .tvar _ _, but the analogous inline code in runSyntaxElaborator (around line 1101) only matches .type _ [] _:

|>.filterMap fun b =>
  match b.kind with
  | .type _ [] _ => some b.ident
  | _ => none

Is the .tvar case intentionally included here but not there? If both should behave the same, consider extracting the shared logic into extractParamNames and calling it from both sites.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've cleaned up this code quite a bit and added additional test cases.

@joehendrix joehendrix changed the title Replace forward type declarations with automatic pre-registration Eliminate need for forward type declarations in recursive datatypes Feb 27, 2026
@joehendrix joehendrix force-pushed the jhx/ddm_preregister branch 2 times, most recently from 6496d19 to d056ba0 Compare March 3, 2026 21:07
joehendrix and others added 9 commits March 3, 2026 16:50
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pass `gctx` directly through the template expansion monad instead of
building an intermediate `Std.HashSet String` of existing names and
folding `ensureDefined` over results afterward. This removes
`TemplateExpansionResult` and the two unnecessary folds in
`addDatatypeBindings`, using `GlobalContext` membership for duplicate
detection and `ensureDefined` as functions are generated.

The template expansion machinery is moved after the `GlobalContext`
definition to resolve forward-reference issues.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change extractFieldsFromBindings, extractSingleConstructor, and
extractConstructorInfo to return Except String instead of silently
returning empty arrays/none on errors. This surfaces clear error
messages for bad arg shapes, missing annotations, and out-of-bounds
indices.

Update callers:
- addDatatypeBindings! (renamed with !) uses panic! on error
- collectConstructorNames returns #[] on error (non-fatal for pre-registration)
- translateConstructorList forwards errors via TransM.error

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add GlobalContext.defineChecked with preRegistered flag for
  explicit membership/freshness checking at define sites
- Replace ensureDefined in addDatatypeBindings!, addCommand.addBinding,
  and preRegisterType with defineChecked or direct define-with-proof
- Add atomic buildAndDefine for template function expansion (no TOCTOU)
- Thread preRegistered : Bool through addCommand → addBinding
- Extract elabSyntaxArg and resultContext from runSyntaxElaborator
- Add docstrings to preRegisterType, runSyntaxElaborator, extractDatatypeInfo

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…review items

- Move constructor info extraction functions out of GlobalContext namespace
  (they only depend on DialectMap, not GlobalContext)
- Move GlobalContext definition after constructor info extraction
- Lift addBinding from where clause to standalone private def
- Move collect helper and checkConstructorUniqueness to Elab.lean
- Add elaboration-time constructor uniqueness check in evalBindingSpec
- Remove callerInfo parameter and FIXME debug diagnostic from
  runSyntaxElaborator
- Fix typos (datype, assignemnt) and stale function name references
- Wrap long lines to stay within 100 char limit
- Make OperationF.foldBindingSpecs public

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rUniqueness

Calls expandFunctionTemplates in evalBindingSpec's .datatype branch so
name collisions (duplicate constructors, duplicate template functions) are
reported via logError during elaboration rather than panicking in
addDatatypeBindings!.

Removes checkConstructorUniqueness from Elab.lean since constructor names
no longer need to be globally unique — collisions are now caught by the
template expansion check which covers both constructor and generated
function names.

Adds function templates (perConstructor tester, perField accessor) to the
DDM MutualDatatypes test dialect and adds Test 5 for complex mutual types
with multiple constructors and fields.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds runChecked to TemplateExpandM to track which constructors failed
registration. Skips template expansion for failed constructors so we
report only the root cause (duplicate constructor name) rather than
cascading into duplicate template function errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add @[prec(30), leftassoc] to constructorListPush so constructor
arguments (at maxPrec) don't get unnecessarily parenthesized.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@joehendrix joehendrix force-pushed the jhx/ddm_preregister branch from a622fb4 to 5347405 Compare March 4, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants