Demo For Common Ansible Action-Plugin Based Testing Strategy for ND 4…

[astawast-cisco]

Current Progress

This repository contains a proof of concept (POC) for the proposed common Ansible action-plugin based testing strategy for ND 4.x integration tests. The goal is to demonstrate how common integration test logic can be centralized while allowing module-specific validations to remain within individual test playbooks.

Objectives Achieved

1. Shared Action Plugin
   Implemented a reusable action plugin: " plugins/action/nd4x_module_test.py "

The plugin standardizes the common integration test workflow by executing:

• Check mode execution
• Normal module execution
• Idempotency verification
• Expected changed/unchanged validation
• Expected failure validation
• Debug and result collection

This removes the need for every module test to duplicate the same execution logic.

2. Proof of Concept Using nd_interface_loopback
   The shared action plugin has been demonstrated using the nd_interface_loopback module.

Demo state playbooks have been created for:

• merged
• replaced
• overridden
• deleted

This validates that the same execution framework can support multiple module states without changing the core testing logic.

3. Smaller Sanity Test Flow

A lightweight sanity workflow has been introduced for faster validation during pull requests.

The demo flow consists of:

• Environment setup
• Module execution
• Idempotency verification
• Validation
• Cleanup

This provides a faster alternative to executing the complete integration test suite while preserving confidence in basic functionality.

4. Standardized Expected Result Contract

The plugin now supports a simplified expected-result schema:

expected:
check_mode_changed: true
apply_changed: true
idempotency: true
failed: false

This improves readability compared to previous run-specific expectations and provides a consistent contract across modules.

5. ND API Query and Validation Framework

ND API validation has been added to the action plugin as an optional validation step after module execution.

The plugin supports:

Executing ND REST API queries through cisco.nd.nd_rest
Validating API responses using JSONPath expressions
Comparing actual API response values with expected values
Supporting expected HTTP status codes, including 404 for deleted resources
Reporting ND API validation failures as part of the same test execution flow

This allows a test to verify not only the module return result, but also the actual state present in the ND controller.

Example use cases include:

Validating that an interface was created successfully
Validating that interface attributes were updated correctly
Validating that a deleted resource is absent by expecting an HTTP 404 response

Current implementation supports both positive ND API validation and deleted-resource validation.

6. Separation of Generic and Module-Specific Validation

The testing framework intentionally separates responsibilities.

The shared action plugin handles generic testing behavior such as:

• Module execution
• Idempotency checks
• Expected result validation
• ND API query execution

Individual module playbooks continue to contain feature-specific validations, such as verifying interface configuration, descriptions, IP addresses, or resource absence.

This keeps the framework reusable across different ND modules.

7. Cleanup Strategy

Cleanup is performed through Ansible always blocks to ensure test resources are removed regardless of test success or failure, preventing leftover configuration from affecting subsequent test runs.

8. Environment and Integration Improvements

During development, several integration issues were resolved, including:

• Generalizing ansible_python_interpreter
• Correcting ansible_network_os configuration
• Resolving dependency issues (including Pydantic)
• Resolving visibility issues with ansible.netcommon and ansible.utils
• Validating required fabric and switch variables
• Adding switch ID support for ND API endpoint validation
• Successfully executing the proof-of-concept test flow

Current Status

The proof of concept demonstrates that a shared action-plugin based framework can successfully centralize common ND 4.x integration test behavior while keeping module-specific assertions independent.

Steps to run the ND4.x demo playbook

1. Go to the collection root:

   cd /Users//ansible_collections/cisco/nd

2. Install/check required collections:

   ansible-galaxy collection install ansible.netcommon ansible.utils

3. Create local inventory :

   cp tests/integration/inventory.networking tests/integration/inventory.LOCAL.networking

4. Update local inventory values:

   ansible_host=
   ansible_user=
   ansible_password=
   nd_test_fabric_name=
   nd_test_switch_ip=
   nd_test_switch_id=<switch id/serial>

5. Run the demo sanity flow:

   ANSIBLE_CONFIG=ansible.cfg
   ANSIBLE_COLLECTIONS_PATH=/Users/:/Users//.ansible/collections:/usr/share/ansible/collections
   python -m ansible_test._util.target.cli.ansible_test_cli_stub
   network-integration nd_interface_loopback
   --inventory tests/integration/inventory.LOCAL.networking
   --tags nd4x_demo
   -vvv

6. To run one state only, replace the tag:

   --tags nd4x_demo_merged
   --tags nd4x_demo_replaced
   --tags nd4x_demo_overridden
   --tags nd4x_demo_deleted

Expected result:

PLAY RECAP should show failed=0

[sivakasi-cisco]

Can we add unit tests for the action plugin?

[mikewiebe]

Medium: predictive check mode does not verify that controller state stayed unchanged

The harness currently trusts the target module’s check-mode implementation.

During predictive check mode, it:

1. Runs the target module with check mode enabled.
2. Checks only the returned changed and failed values.
3. Proceeds to the real apply when those expectations pass.

It does not:

• Capture controller state before check mode.
• Query the controller immediately afterward.
• Compare before and after controller state.
• Detect an unintended mutation elsewhere in the fabric.

The target module normally performs GET requests to calculate its predicted changes, but that does not prove it avoided a write.

Also, the configured nd_queries do not help here—they run only after the real apply and idempotency phases.

To verify check-mode safety, the sequence should be:

GET relevant controller state
        ↓
run target module in check mode only
        ↓
GET the same controller state again
        ↓
normalize and compare before == after
        ↓
only then run the real apply

For an interface test, the snapshot should cover all relevant managed interfaces on the test switch—not just the requested interface—so an unintended mutation to another interface is also detected.

The cleanest enhancement would be for the harness to support something like:

check_mode_queries:
  - name: Snapshot managed interfaces
    path: >-
      /api/v1/manage/fabrics/{{ fabric_name
      }}/switches/{{ switch_id }}/interfaces

The harness would run that query before and after predictive check mode and require equivalent normalized controller state before proceeding.

The current phase verifies the module’s reported prediction; it does not verify that predictive execution was mutation-free.

[mikewiebe]

Medium: add a contributor guide for writing integration tests with this framework

The PR explains the framework’s purpose and demonstrates it with loopback and Ethernet-access playbooks, but it does not provide a clear authoring guide for a contributor migrating another module.

The nd4x_module_test documentation stub describes the top-level options, while the target-local demos show selected implementations. A contributor must still infer the framework’s structure, responsibilities, safety requirements, and migration process from the source.

Could we add an authoritative guide such as:

tests/integration/playbooks/nd4x_module_tests/README.md

At minimum, it should document:

1. Responsibility split

   • What the action plugin handles.
   • What remains in the module-specific playbook.
   • That setup, feature-specific assertions, and cleanup are not performed automatically by the harness.

2. Complete input contract

   • module, state, config, common_args, and module_args.
   • The nested expected structure.
   • The nested nd_queries and JSONPath expectation structure.
   • Defaults, supported values, and phase-skipping behavior.

3. Canonical directory and file structure

   • Where a migrated module’s state files should live.
   • Whether the shared hierarchy or target-local hierarchy is authoritative.
   • A copyable template for adding another module.

4. Scenario patterns

   • Simple create/delete tests.
   • Multi-step create → update → delete workflows.
   • Multi-resource, fan-out, and split-configuration tests.
   • Passing module-specific options such as deploy: false.

5. Safety and lifecycle

   • Pre-test cleanup and block/always cleanup.
   • Destructive-test opt-in and preflight requirements.
   • ND-version gating.
   • Resolving current switch IDs from management IPs instead of maintaining two independent identities.
   • Querying controller state before and after predictive check mode to prove that no mutation occurred.

6. Assertions and negative tests

   • Checking phase results and module-specific returned fields.
   • REST validation and normalized controller-state comparisons.
   • Verifying the intended failure reason rather than accepting any failed: true result.

7. Tags and execution

   • Safe aggregate, state-specific, fine-grained, and destructive tags.
   • Exact ansible-test commands for each supported execution mode.

8. Migration acceptance checklist

   • Inventory every original test scenario.
   • Map each scenario to its harness replacement.
   • Confirm parity before removing or retiring the original suite.

Without this guide, the PR demonstrates the framework but does not yet make it straightforward for another contributor to use it consistently or confirm that a module’s original integration coverage has been preserved and it will help our team review it's capabilities.

[mikewiebe]

Medium: demonstrate full test parity for all four migrated modules

Before treating this framework as complete, could we migrate every existing integration-test scenario for:

• nd_interface_loopback
• nd_interface_ethernet_access
• nd_manage_switches
• nd_manage_vrf

The current demos prove that the framework supports selected workflows, but they do not yet demonstrate that it can preserve the modules’ complete existing test coverage. Exercising all four full suites would also expose framework gaps that simplified examples may not reveal.

For each module, please:

1. Inventory every scenario in the original integration suite.
2. Map each original scenario to its framework replacement.
3. Preserve the original module-result and controller-state assertions.
4. Run the original and replacement suites against the same environment.
5. Document any scenario that cannot be represented by the framework.
6. Retain the original tests until complete parity has been demonstrated.

The migrated coverage should include, where applicable:

• Every supported state.
• Create → update → delete workflows.
• Multiple resources and split or fan-out configurations.
• Module-specific options such as deploy: false.
• Predictive check mode with controller queries before and after execution.
• Idempotency.
• Negative tests that verify the intended failure reason.
• ND-version and destructive-test safety gates.
• Guaranteed cleanup, including after failures.
• Controller-state validation rather than only checking returned changed and failed values.

Setup, feature-specific assertions, and cleanup can remain in the module-specific playbooks where appropriate; the framework does not need to internalize every responsibility. However, every original behavior should have an identifiable replacement.

If an existing scenario cannot be represented cleanly, that should be treated as a framework gap to address rather than dropping or weakening the test. The original suites should only be retired after the parity mapping and replacement runs are complete.

[nikhilsrikrishna]

I was trying to use this framework for the Interface Groups integration tests and noticed one possible improvement around idempotency. The second execution currently verifies changed: false, but that does not prove it avoided redundant mutation or deployment API calls.

Could you please evaluate supporting an optional expectation such as:

expected:
  idempotency:
    changed: false
    failed: false
    non_get_call_count: 0

The harness can calculate this from the idempotency result fields api_verbs and api_paths: pair the lists, exclude GET, and count the remaining calls. If the expected count does not match, the assertion should report the non-GET verbs and their corresponding paths.

This would detect redundant POST, PUT, PATCH, DELETE, deployment, configuration-save, or other action calls even when the module reports changed: false.

[nikhilsrikrishna]

I was trying to use this framework for the Interface Groups integration tests and noticed another area that could improve negative-test validation. Currently, phase expectations validate only changed and failed. A test expecting failed: true could therefore pass even if the module failed for an unrelated reason.

Could you please evaluate supporting a per-phase message assertion, for example:

expected:
  apply:
    changed: false
    failed: true
    msg_contains: "does not exist"

This could validate the msg returned by the target module, whether it originated from local validation or an ND API failure, and avoid separate assertion tasks in every module test suite.