Add comprehensive test suite for template, substitution, and FOL patterns#7
Open
Add comprehensive test suite for template, substitution, and FOL patterns#7
Conversation
…FOL patterns
Tests cover the fragile areas identified around digit-based template processing:
- default_preprocess_template: bare digits → {n}, ← skip, multi-digit, and
the footgun where explicit {0} syntax gets double-wrapped to {{0}}
- Substitution: N[?←X] chaining mechanics, \? escape, fallback to format,
and a confirmed bug where digits in rendered content corrupt wrap()
- Constraint: ∉ substring logic, multi-condition, bidirectional checks
- Rule render integration: terminal/unary/binary rules, reversed digit order,
callable templates
- FOL patterns: entity-into-property substitution, X_quantifier placeholder
chain, full universal/existential quantifier chains, adjective constraint
- FOL generation smoke: 60 seeds × 4 checks (no ? in eng, no (?) in tptp,
no double whitespace, no free X without quantifier binding)
The test_digits_in_rendered_content_crash_substitution test exposes a real
bug: Substitution.wrap() converts ALL bare digits in substituted content to
{n} format placeholders, so content like 'pred5(X)' becomes 'pred{5}(X)'
and format() crashes with IndexError. Currently safe in FOL (predicates use
letters preda–predj) but unprotected at the API level.
https://claude.ai/code/session_01HdENPeusFQ7i6CHZRVJtmr
Bug 1 – Substitution.wrap() corrupts digits in rendered content:
The old implementation applied wrap() (digit→{n}) to the entire substituted
string, so a rendered arg like 'pred5(?)' would become 'pred{5}(X)' and
crash with IndexError when .format() looked for positional arg 5.
Fix: split the template at N[?←X] boundaries with re.split; wrap only the
original template text parts and the replacement string X (e.g. '1'→'{1}'
for format-slot chaining), never the rendered content of the substituted arg.
Bug 2 – default_preprocess_template double-wraps explicit {n} syntax:
The regex r'(\d+)' matched the digit inside '{0}', turning '{0}' into '{{0}}'
which .format() renders back as the literal string '{0}'.
Fix: add negative lookbehind/lookahead (?<!{)(\d+)(?!}) so already-braced
digits are skipped; bare digits are still wrapped as before.
Also adds depth-bound tests (section 7 in the test file):
- FOL generation: assert min_depth ≤ height ≤ max_depth for 60 seeds
- Simple recursive grammar: same bounds check for 30 seeds
- Without min_depth: verify heights vary and stay ≤ max_depth
https://claude.ai/code/session_01HdENPeusFQ7i6CHZRVJtmr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a comprehensive test suite (
tests/test_template_digit_and_constraint.py) covering critical functionality in template preprocessing, substitution mechanics, constraints, and first-order logic (FOL) pattern generation. The tests validate both unit-level behavior and end-to-end integration scenarios.Key Changes
Template Preprocessing Tests (
TestDefaultPreprocessTemplate): Validatesdefault_preprocess_template()behavior including bare-digit wrapping, edge cases with escaped characters, multi-digit numbers, and the double-wrapping footgun when mixing explicit{n}syntax with preprocessing.Substitution Tests (
TestSubstitution): Tests theSubstitutionclass including variable replacement viaN[?←X]patterns, chaining mechanics, handling of escaped question marks, and fallback to.format()when no substitution pattern is present. Includes a documented bug test for digits in rendered content.Constraint Tests (
TestConstraint): Validates substring-based constraint checking (0∉1) for preventing unwanted overlaps between rendered children, including bidirectional and multi-way constraints.Rule Rendering Integration Tests (
TestRuleRenderIntegration): Tests end-to-end rendering of terminal and composite rules with multiple children, including callable templates and digit reordering.FOL Pattern Unit Tests (
TestFOLPatternsUnit): Validates specific FOL template patterns including:?) handling through the substitution chainFOL Generation Smoke Tests: Parametrized tests across 60 random seeds validating:
Notable Implementation Details
_Stub,_make_grammar,_fp,_node) to construct test grammars and parse trees without full grammar initialization overheadhttps://claude.ai/code/session_01HdENPeusFQ7i6CHZRVJtmr