feat: rich semantic validation output with pattern-based formatting#363
Open
aitestino wants to merge 27 commits intonetascode:mainfrom
Open
feat: rich semantic validation output with pattern-based formatting#363aitestino wants to merge 27 commits intonetascode:mainfrom
aitestino wants to merge 27 commits intonetascode:mainfrom
Conversation
Add a dedicated constants module containing the Colors class with ANSI escape codes for terminal colorization. This provides a centralized location for color constants used throughout the validation output formatting. Includes support for basic colors (red, yellow, green, cyan, blue, magenta, white) and text styling (bold, dim, italic, underline).
Introduce OutputFormatter class that uses pattern detection to apply rich terminal formatting to validation output. The formatter is rule-agnostic and automatically colorizes: - ALL_CAPS headers (red, bold) - Section headers ending with colons (cyan, bold) - Separator lines (===, ---, ───) (dim) - Bullet points and list items (appropriate colors) - Key-value pairs and code examples Also adds format_checklist_summary() for generating a remediation checklist showing all failed rules with violation counts. This approach allows each rule to define its own output format while getting consistent colorization without hardcoding rule-specific logic in the validator.
…mary Refactor validator.py to use the new output_formatter module: - Remove embedded Colors class (now imported from constants) - Remove _format_semantic_error method (replaced by output_formatter) - Use format_semantic_error() for rich rule output colorization - Add _count_violations_from_content() helper to extract counts - Collect failed rule info during validation for summary - Display remediation checklist at end of validation showing all failed rules with violation counts The checklist provides a clear action item summary for users to address each validation failure.
Add --list-rules CLI option that prints all available semantic validation rules with their IDs, descriptions, and severity levels. Rules are sorted by ID and color-coded by severity (red=HIGH, yellow=MEDIUM, cyan=LOW). This allows users to discover available rules without running validation, useful for understanding what checks are performed and for documentation purposes. Also makes the paths argument optional when using --list-rules, so users can run: nac-validate --list-rules
This was referenced Jan 26, 2026
Adds --format/-f CLI option with 'text' (default) and 'json' values.
JSON output includes structured syntax and semantic validation errors:
{
"syntax_errors": [{"file": "...", "line": N, "column": N, "message": "..."}],
"semantic_errors": [{"rule_id": "...", "description": "...", "errors": [...]}]
}
Key changes:
- Add SyntaxErrorResult and SemanticErrorResult dataclasses
- Populate structured results in validator for both error types
- Add OutputFormat enum and --format CLI option
- Suppress logging at default verbosity when using JSON format
- Add integration tests for JSON output
Author
|
Author
|
|
Previously, the OutputFormatter only applied pattern-based colorization when a rule returned exactly one result item. Rules like 101 (unique keys) return multiple formatted blocks (one per duplicate type), which were incorrectly treated as simple list items and displayed without colors. Now the formatter checks each result item individually and applies rich content colorization to all items that match the pattern (starts with newline, contains separator characters), while still supporting mixed output where some items are simple strings.
Author
ACI-as-Code-Demo/aac on demo/nac-validate-CVD-best-practices [$!] via 🐍 v3.13.7 (ACI-as-Code-Demo) via 💎 v3.0.0 via 💠 default took 3s
❯ nac-validate data/ -s schemas/apic_schema.yaml -r ../.rules -v DEBUG -f json
INFO - Loading schema
INFO - Loading rules
INFO - Validate file: data/access_policies.nac.yaml
INFO - Validate file: data/node_policies.nac.yaml
INFO - Validate file: data/pod_policies.nac.yaml
INFO - Validate file: data/test_rule_violations.nac.yaml
INFO - Validate file: data/tenant_PRES.nac.yaml
INFO - Validate file: data/test_rule_triggers.nac.yaml
INFO - Validate file: data/tenant_CORE.nac.yaml
INFO - Validate file: data/tenant_ISO.nac.yaml
INFO - Validate file: data/tenant_christopher.nac.yaml
INFO - Validate file: data/tenant_chart2.nac.yaml
INFO - Validate file: data/tenant_mgmt.nac.yaml
INFO - Validate file: data/apic.nac.yaml
INFO - Validate file: data/tenant_infra.nac.yaml
INFO - Validate file: data/tenant_HOST.nac.yaml
INFO - Validate file: data/tenant_TEST.nac.yaml
INFO - Validate file: data/fabric_policies.nac.yaml
INFO - Validate file: data/test_policy_refs.nac.yaml
INFO - Loading yaml files from [PosixPath('data')]
INFO - Verifying rule id 100
INFO - Verifying rule id 101
INFO - Verifying rule id 102
INFO - Verifying rule id 202
INFO - Verifying rule id 204
INFO - Verifying rule id 205
INFO - Verifying rule id 206
INFO - Verifying rule id 301
INFO - Verifying rule id 302
INFO - Verifying rule id 303
INFO - Verifying rule id 304
INFO - Verifying rule id 305
INFO - Verifying rule id 306
INFO - Verifying rule id 307
INFO - Verifying rule id 308
INFO - Verifying rule id 309
INFO - Verifying rule id 310
INFO - Verifying rule id 311
INFO - Verifying rule id 312
INFO - Verifying rule id 401
ERROR - Semantic error, rule 100: Verify unique EPG name:
ERROR - Semantic error, rule 101: Verify Unique Identifier Keys Across ACI Objects:
ERROR - Semantic error, rule 102: Verify Unique EPG Names Within Application Profiles:
ERROR - Semantic error, rule 202: Verify Fabric Leaf Switch Policy Group References Exist:
ERROR - Semantic error, rule 204: Verify Access Leaf Interface Policy Group References Exist:
ERROR - Semantic error, rule 205: Verify Access Spine Interface Policy Group References Exist:
ERROR - Semantic error, rule 206: Verify Object Names Are Not Interpreted as Scientific Notation:
ERROR - Semantic error, rule 301: Verify Infra VLAN Is Defined When Referenced by AAEPs:
ERROR - Semantic error, rule 302: Verify DNS Policy Provider Count Follows Cisco Best Practices:
ERROR - Semantic error, rule 303: Verify Fabric Nodes Are Not Assigned to Reserved Pod 0:
ERROR - Semantic error, rule 305: Verify AAA Password Strength Policy Flags Are Configured:
ERROR - Semantic error, rule 308: Verify VMM Port-Group Names Do Not Exceed vSphere Limits:
ERROR - Semantic error, rule 309: Verify L3Out Route Map Names Do Not Exceed ACI Limits:
ERROR - Semantic error, rule 310: Verify Contract Filter Port Range Configuration Is Valid:
ERROR - Semantic error, rule 311: Verify Contract Filters Specify Protocol for Non-Standard Ports:
ERROR - Semantic error, rule 312: Verify VPC and PC Channel Node IDs Can Be Resolved:
{
"syntax_errors": [],
"semantic_errors": [
{
"rule_id": "100",
"description": "Verify unique EPG name",
"errors": [
"Regular EPG and uSeg EPG have duplicated name - Conflicting_EPG in application profile Rule102_Test_ANP of tenant RuleTrigger_Tenant",
"Regular EPG and uSeg EPG have duplicated name - WebServer_EPG in application profile Rule102_Test_ANP_2 of tenant RuleTrigger_Tenant"
]
},
{
"rule_id": "101",
"description": "Verify Unique Identifier Keys Across ACI Objects",
"errors": [
"apic.tenants.name - Duplicate Tenant Names: 'RuleTrigger_Tenant' appears 2 times in apic.tenants",
"apic.tenants.vrfs.name - Duplicate VRF Names: 'Duplicate_VRF' appears 2 times in apic.tenants.vrfs",
"apic.tenants.bridge_domains.name - Duplicate Bridge Domain Names: 'Duplicate_BD' appears 2 times in apic.tenants.bridge_domains"
]
},
{
"rule_id": "102",
"description": "Verify Unique EPG Names Within Application Profiles",
"errors": [
"apic.tenants[name=RuleTrigger_Tenant].application_profiles[name=Rule102_Test_ANP] - EPG 'Conflicting_EPG' exists as both regular and uSeg EPG in RuleTrigger_Tenant/Rule102_Test_ANP",
"apic.tenants[name=RuleTrigger_Tenant].application_profiles[name=Rule102_Test_ANP_2] - EPG 'WebServer_EPG' exists as both regular and uSeg EPG in RuleTrigger_Tenant/Rule102_Test_ANP_2"
]
},
{
"rule_id": "202",
"description": "Verify Fabric Leaf Switch Policy Group References Exist",
"errors": [
"apic.node_policies.nodes[id=901].fabric_policy_group - Node 901 (TEST-LEAF-INVALID-POLICY) references undefined policy group 'NONEXISTENT_LEAF_POLGRP'"
]
},
{
"rule_id": "204",
"description": "Verify Access Leaf Interface Policy Group References Exist",
"errors": [
"apic.interface_policies.nodes[id=902].interfaces[port=1].policy_group - Node 902 interface Eth1/1 references undefined policy group 'NONEXISTENT_LEAF_INTF_POLGRP'",
"apic.interface_policies.nodes[id=902].interfaces[port=10].policy_group - Node 902 interface Eth1/10 references undefined policy group 'ANOTHER_MISSING_LEAF_POLGRP'",
"apic.interface_policies.nodes[id=902].interfaces[port=11].policy_group - Node 902 interface Eth1/11 references undefined policy group 'ANOTHER_MISSING_LEAF_POLGRP'",
"apic.interface_policies.nodes[id=902].interfaces[port=12].policy_group - Node 902 interface Eth1/12 references undefined policy group 'ANOTHER_MISSING_LEAF_POLGRP'"
]
},
{
"rule_id": "205",
"description": "Verify Access Spine Interface Policy Group References Exist",
"errors": [
"apic.interface_policies.nodes[id=903].interfaces[port=1].policy_group - Spine 903 interface Eth1/1 references undefined policy group 'NONEXISTENT_SPINE_INTF_POLGRP'",
"apic.interface_policies.nodes[id=903].interfaces[port=5].policy_group - Spine 903 interface Eth1/5 references undefined policy group 'ANOTHER_MISSING_SPINE_POLGRP'",
"apic.interface_policies.nodes[id=903].interfaces[port=6].policy_group - Spine 903 interface Eth1/6 references undefined policy group 'ANOTHER_MISSING_SPINE_POLGRP'"
]
},
{
"rule_id": "206",
"description": "Verify Object Names Are Not Interpreted as Scientific Notation",
"errors": [
"apic.access_policies.vlan_pools.9.name - Value '421e714314321443' contains pattern '421e714314321443' that may be converted to a number"
]
},
{
"rule_id": "301",
"description": "Verify Infra VLAN Is Defined When Referenced by AAEPs",
"errors": [
"apic.access_policies.aaeps[name=TEST_INFRA_AAEP_VIOLATION].infra_vlan - AAEP 'TEST_INFRA_AAEP_VIOLATION' has infra_vlan enabled but global infra_vlan is not defined"
]
},
{
"rule_id": "302",
"description": "Verify DNS Policy Provider Count Follows Cisco Best Practices",
"errors": [
"apic.fabric_policies.dns_policies[name=default] - Policy 'default' has 4 providers (recommended maximum: 2)"
]
},
{
"rule_id": "303",
"description": "Verify Fabric Nodes Are Not Assigned to Reserved Pod 0",
"errors": [
"apic.node_policies.nodes[id=999].pod - Node 999 (TEST-INVALID-POD-NODE) is assigned to reserved Pod 0"
]
},
{
"rule_id": "305",
"description": "Verify AAA Password Strength Policy Flags Are Configured",
"errors": [
"apic.fabric_policies.aaa.management_settings.password_strength_profile.password_class_flags - Only 2 password class(es) configured; minimum 3 required"
]
},
{
"rule_id": "308",
"description": "Verify VMM Port-Group Names Do Not Exceed vSphere Limits",
"errors": [
"apic.tenants[name=TEST].application_profiles[name=Enterprise_Production_Datacenter_Virtualization_ANP].endpoint_groups[name=VMware_Guest_Virtual_Machine_Servers_Primary_EPG] - EPG 'TEST/Enterprise_Production_Datacenter_Virtualization_ANP/VMware_Guest_Virtual_Machine_Servers_Primary_EPG' generates 103-char port-group name (exceeds limit by 24)"
]
},
{
"rule_id": "309",
"description": "Verify L3Out Route Map Names Do Not Exceed ACI Limits",
"errors": [
"apic.tenants[name=TEST].l3outs[name=Enterprise_External_Internet_Provider_Edge_L3OUT] - Route control 'TEST/Enterprise_External_Internet_Provider_Edge_L3OUT/Inbound_Route_Filtering_Policy_For_External_Routes' (Import): 184 chars (exceeds by 49)"
]
},
{
"rule_id": "310",
"description": "Verify Contract Filter Port Range Configuration Is Valid",
"errors": [
"apic.tenants[name=TEST].filters[name=INVALID_PORT_RANGE_FILTER].entries[name=MISSING_FROM_PORT] - Filter 'INVALID_PORT_RANGE_FILTER' entry 'MISSING_FROM_PORT': destination_to_port=443 specified without destination_from_port",
"apic.tenants[name=TEST].filters[name=INVALID_PORT_RANGE_FILTER].entries[name=INVERTED_PORT_RANGE] - Filter 'INVALID_PORT_RANGE_FILTER' entry 'INVERTED_PORT_RANGE': destination_from_port (8080) > destination_to_port (80)"
]
},
{
"rule_id": "311",
"description": "Verify Contract Filters Specify Protocol for Non-Standard Ports",
"errors": [
"apic.tenants[name=PRES].filters[name=BACKUP_PORTS].entries[name=TCP_8400] - Filter 'BACKUP_PORTS' entry 'TCP_8400': non-well-known ports (destination_from_port=8400) without explicit protocol",
"apic.tenants[name=PRES].filters[name=BACKUP_PORTS].entries[name=TCP_8401] - Filter 'BACKUP_PORTS' entry 'TCP_8401': non-well-known ports (destination_from_port=8401) without explicit protocol",
"apic.tenants[name=PRES].filters[name=BACKUP_PORTS].entries[name=TCP_8408] - Filter 'BACKUP_PORTS' entry 'TCP_8408': non-well-known ports (destination_from_port=8408) without explicit protocol",
"apic.tenants[name=PRES].filters[name=BACKUP_PORTS].entries[name=TCP_2987] - Filter 'BACKUP_PORTS' entry 'TCP_2987': non-well-known ports (destination_from_port=2987) without explicit protocol",
"apic.tenants[name=PRES].filters[name=BACKUP_PORTS].entries[name=TCP_8403] - Filter 'BACKUP_PORTS' entry 'TCP_8403': non-well-known ports (destination_from_port=8403) without explicit protocol",
"apic.tenants[name=PRES].filters[name=BACKUP_PORTS].entries[name=TCP_8405] - Filter 'BACKUP_PORTS' entry 'TCP_8405': non-well-known ports (destination_from_port=8405) without explicit protocol",
"apic.tenants[name=PRES].filters[name=SMB].entries[name=TCP_445] - Filter 'SMB' entry 'TCP_445': non-well-known ports (destination_from_port=445) without explicit protocol",
"apic.tenants[name=PRES].filters[name=SMB-S].entries[name=TCP_445] - Filter 'SMB-S' entry 'TCP_445': non-well-known ports (source_from_port=445) without explicit protocol",
"apic.tenants[name=PRES].filters[name=ESX_PORTS].entries[name=TCP_2049] - Filter 'ESX_PORTS' entry 'TCP_2049': non-well-known ports (destination_from_port=2049) without explicit protocol",
"apic.tenants[name=PRES].filters[name=ESX_PORTS].entries[name=TCP_902] - Filter 'ESX_PORTS' entry 'TCP_902': non-well-known ports (destination_from_port=902) without explicit protocol",
"apic.tenants[name=PRES].filters[name=RDP-S].entries[name=TCP_3389] - Filter 'RDP-S' entry 'TCP_3389': non-well-known ports (source_from_port=3389) without explicit protocol",
"apic.tenants[name=PRES].filters[name=NetBIOS].entries[name=TCP_139] - Filter 'NetBIOS' entry 'TCP_139': non-well-known ports (destination_from_port=139) without explicit protocol",
"apic.tenants[name=PRES].filters[name=NetBIOS-S].entries[name=TCP_139] - Filter 'NetBIOS-S' entry 'TCP_139': non-well-known ports (source_from_port=445) without explicit protocol",
"apic.tenants[name=PRES].filters[name=RDP].entries[name=RDP-TCP] - Filter 'RDP' entry 'RDP-TCP': non-well-known ports (destination_from_port=3389) without explicit protocol",
"apic.tenants[name=ISO].filters[name=PERMIT_TCP_81].entries[name=TCP-81] - Filter 'PERMIT_TCP_81' entry 'TCP-81': non-well-known ports (destination_from_port=81) without explicit protocol"
]
},
{
"rule_id": "312",
"description": "Verify VPC and PC Channel Node IDs Can Be Resolved",
"errors": [
".apic.tenants[12].application_profiles[1].endpoint_groups[1].static_ports[0] - Channel 'VPC_TEST_UNRESOLVABLE': not found in interface_policies and node_id/node2_id not specified"
]
}
]
}
|
Introduce Violation, RuleContext, RuleResult, and GroupedRuleResult dataclasses that rules use to report validation issues. This enables clean separation between detection (rules) and presentation (formatter). Export these types from the package's public API for rule authors.
Rewrite OutputFormatter to render RuleResult and GroupedRuleResult objects as colored terminal output with rich context sections. Update format_json_result to output errors as simple "path - message" strings for consistent API contract with downstream consumers.
Update SemanticErrorResult.errors to be a simple list[str] type containing "path - message" formatted strings, matching the simplified JSON output contract.
Update validate_semantics to work with RuleResult and GroupedRuleResult objects. Add helper methods for violation counting and detection. Also sort rules by filename during loading for consistent, predictable execution order (ascending by rule ID prefix).
Add 15 unit tests covering format_json_result function: - RuleResult with violations produces path-message strings - GroupedRuleResult flattens all groups into single errors array - String list input passes through directly - Edge cases: empty paths, special characters, multiline messages
Enhance test_json_format_semantic_errors to verify that errors are strings containing the " - " separator, not dicts.
- Add explicit dict[str, Any] type annotation to format_json_result - Use X | Y syntax in isinstance calls per ruff UP038 - Remove unreachable code branches - Apply ruff formatter changes
Use 'raise from None' to avoid exposing internal exception details when rule loading fails.
- Add ExitCode IntEnum for distinct CLI exit codes (SUCCESS=0, SEMANTIC_ERROR=1, SYNTAX_ERROR=2, CONFIG_ERROR=3) - Add default path constants (DEFAULT_SCHEMA, DEFAULT_RULES) - Add formatting width constants (HEADER_SEPARATOR_WIDTH, etc.) - Add rule loading constants (RULE_MODULE_NAME, etc.) - Add severity sorting constants (SEVERITY_SORT_ORDER, etc.) - Add Colors.for_severity() staticmethod for severity-to-color mapping - Remove unused color codes (BLUE, WHITE) and styles (ITALIC, UNDERLINE) - Remove unused colorize() method
Move CLI option types and type aliases to options.py for better separation of concerns. This makes main.py a thin orchestration layer focused on command logic rather than type definitions. - Add VerbosityLevel and OutputFormat enums - Add version_callback function - Add annotated type aliases (Verbosity, Schema, Rules, etc.)
- Replace magic exit code numbers with ExitCode enum values - Import option types from options.py module - Import defaults from constants.py instead of cli/defaults.py - Add format_validation_summary() calls for text output - Add file counting for validation summary display - Use Validator.from_paths() factory method - Add module docstring describing CLI architecture
Delete cli/defaults.py as it is no longer imported anywhere. Default path constants are now defined in constants.py.
- Add from_paths() classmethod for creating Validator instances - Extract _load_schema() and _load_rules() as static methods - Add _get_data() with cache invalidation on path changes - Validate match method parameter count during rule loading - Use Path methods instead of os.path functions - Use rglob() instead of os.walk() for directory traversal - Import constants (RULE_MODULE_NAME, YAML_SUFFIXES, etc.) - Remove unused import (import importlib) - Move re import to module level
- Add format_validation_summary() for CLI validation status display - Add format_rules_list() for --list-rules output formatting - Use Colors.for_severity() instead of inline color selection - Use constants for separator widths and magic values - Remove DOUBLE_SEP (unused separator type) - Move re import to module level
Remove GroupedRuleResult.all_violations property as it is not referenced anywhere in the codebase.
Remove # type: ignore from importlib.metadata import as it is no longer needed with current Python version requirements.
Add comprehensive unit tests for: - _get_data() caching behavior with cache invalidation - _get_violation_count() for different result types - _get_named_path() path transformation utility Tests verify cache hits return same object reference and cache misses return distinct objects, ensuring proper caching behavior.
Add tests for: - --list-rules functionality with valid/empty/malformed rules - Missing required arguments handling - Configuration error JSON output - Unexpected error handling in JSON format Tests use real rule files to trigger error paths rather than mocking, ensuring actual code paths are exercised.
Add test classes for: - format_violation() and format_violations_list() - format_context() with and without references - format_rule_result() for rich and simple output - format_validation_summary() status display - format_rules_list() rule listing output - format_checklist_summary() sorting and display
Replace magic exit code numbers with ExitCode enum values for clearer test assertions and consistency with CLI changes.
- Document exit codes (0=success, 1=semantic, 2=syntax, 3=config) - Add --format and --list-rules to CLI help section - Add structured rules documentation with RuleContext example - Document Violation, RuleContext, RuleResult data classes - Add JSON output format documentation - Fix YAML syntax highlighting in pre-commit examples
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces significant improvements to the semantic validation output, making it more human-readable and actionable:
--list-rulesCLI option: Discover available validation rules without running validationArchitecture
The implementation follows the Single Responsibility Principle:
constants.py: Color definitionsoutput_formatter.py: Pattern detection and colorization logicvalidator.py: Validation orchestration (now uses output_formatter)cli/main.py: CLI interface with new optionsRelated Issues
This PR addresses the goals of:
Screenshots
Screenshots will be added in comments to demonstrate the improved output.