Add opt-in gathered-state filtering for loopback and local users by deekpand-cisco · Pull Request #391 · CiscoDevNet/ansible-nd

deekpand-cisco · 2026-07-07T16:27:16Z

Related Issue(s)

Related to #340.

This PR builds on the foundational framework established in
#312
including the state machine, base model infrastructure, orchestrator pattern,
and secrets handling.

Proposed Changes

This PR introduces an opt-in gathered-state filtering framework for modules
using NDStateMachine and enables it for:

cisco.nd.nd_interface_loopback
cisco.nd.nd_local_user

Shared gathered-filtering framework

The shared implementation:

runs filtering only when state: gathered;
requires each model to opt in using supports_gathered_filtering;
treats config as partial filter criteria for supported gathered operations;
uses AND semantics between fields within one filter item;
uses OR semantics between separate filter items;
converts API responses into normalized Ansible configuration before final matching;
allows models to declare supported filter properties through gathered_filter_properties and rejects unsupported properties before any API request;
allows models to normalize filter values through normalize_gathered_filter;
allows models to customize matching through matches_gathered_filter;
removes duplicate matches using the model identifier;
keeps gathered filters out of proposed configuration;
prevents gathered filters from entering create, update, or delete diff processing;
preserves existing behavior for non-gathered states;
preserves existing behavior for models that have not opted in.

The generic local filter remains the final correctness layer. Endpoint-specific filtering is used only to reduce the number of API candidates where the endpoint safely supports it.

Loopback-interface filtering

The loopback module supports gather-all and partial gathered filters across switches in a fabric.

Gathered loopback queries use the interface-list endpoint and its Lucene filter query parameter:

every query includes interfaceType:loopback and policyType:loopback;
switch_ip is resolved to a switch serial number and limits the query to that switch;
omitting switch_ip queries all switches in the fabric;
interface_name is translated into an exact-match Lucene term;
separate filter items are queried independently to preserve OR semantics;
an unfiltered loopback query for a switch supersedes narrower interface-name queries for that switch;
pagination uses max and offset until the API reports no remaining results;
responses from multiple switches, filter items, and pages are combined and deduplicated.

Lucene expressions are built only from structured module input. Raw Lucene expressions are not accepted from users.

The module continues to enforce its resource-ownership boundary:

interfaceType must be loopback;
policyType must be loopback;
system-managed underlayLoopback interfaces are excluded;
non-standard ipfmLoopback and userDefined policies are excluded.

Nested criteria such as IP address, administrative state, IPv6 address, and VRF are evaluated by the generic final local filter.

Interface names supplied as filters are normalized to lowercase to match the existing model and Nexus Dashboard API behavior.

Local-user filtering

The local-user endpoint does not expose documented server-side Lucene filtering. Gather-all retrieves the available local users, and optional filter criteria are evaluated through the shared final local-matching layer.

Supported gathered filter properties for local users are login_id, email, first_name, and last_name. Other configuration fields (such as user_password, reuse_limitation, or security_domains) are rejected with a validation error before any API request.

Multiple filter items use OR semantics. String matching is case-sensitive.

Local-user passwords are not returned by Nexus Dashboard and remain excluded from gathered output. Gathered output for an existing user can be reused as configuration, but recreating a deleted user still requires a new password.

Argument validation and gathered output

For both modules:

config can be omitted for gather-all operations;
config remains required for write states through required_if;
the underlying Pydantic resource identifiers remain required for complete resources;
gathered output is pruned to the module argument specification so it can be reused as config;
gathered operations remain read-only and return changed: false.

For loopback gathered operations, switch_ip, interface_name, and nested policy fields (admin_state, ip, ipv6, vrf) can be supplied as partial filtering criteria.

For local-user gathered operations, login_id, email, first_name, and last_name can be supplied as partial filtering criteria.

Compatibility and rollout

Gathered filtering is disabled by default in the base model.
Existing modules are unaffected until they explicitly opt in.
Lucene filtering is disabled by default in the base orchestrator.
Existing orchestrators are unaffected until they explicitly opt in.
Custom modules using specialized runners are unaffected.
Non-gathered state behavior remains unchanged.
Gathered filters do not enter write-state processing.
Gathered operations do not modify Nexus Dashboard resources.

Test Notes

Unit coverage includes Lucene expression construction and escaping, query planning, pagination, deduplication, local matching, gathered/write-state isolation, local-user filtering, property validation, and password secrecy.

Integration coverage was added for gather-all, exact and multiple filters, OR semantics, no-match behavior, unsupported filter rejection, empty filter rejection, password secrecy, and gathered-output reuse.

Cisco Nexus Dashboard Version

Cisco Nexus Dashboard 4.2 lab environment.

Related ND API Resource Category

manage
infra

The loopback-interface module uses the Manage API, while the local-user module uses the Infra AAA API.

Checklist

Latest commit is rebased from develop with merge conflicts resolved
New or updated documentation has been added
Assigned the proper reviewers

sivakasi-cisco · 2026-07-20T13:30:56Z

+    Returned objects remain in API shape for NDConfigCollection.from_api_response().
+    """
+
+    if not filters:


Should we still deduplicate the results when no filters are provided? Right now this returns before the deduplication logic runs.

Removed the early return so deduplication now runs unconditionally. The to_config() function is only called when active filters exist to avoid unnecessary work.

sivakasi-cisco · 2026-07-20T13:31:38Z

+                query_kwargs["gathered_filters"] = raw_config
+
+            response_data = self.model_orchestrator.query_all(**query_kwargs)
+


Can we validate the gathered filters before calling query_all()? Otherwise, an invalid filter may make unnecessary API calls before it fails.

Added a validate_gathered_filters() function as a pre-flight check in the state machine, called before query_all(). Invalid filters now fail fast without making API calls.

sivakasi-cisco · 2026-07-20T13:40:28Z

+            meta = result.get("meta") or {}
+            counts = meta.get("counts") or {}
+            try:
+                remaining = int(counts.get("remaining", 0))


If remaining is missing or invalid, this treats it as zero and stops pagination. Could this return incomplete gathered results when the first page is full? Or, may be continue based on the page size or fail instead of returning partial data?

Add gathered_all state to the state machine for modules that retrieve all instances, mask_secrets helper for sensitive field redaction, and associated unit tests. Note: gathered-all framework cherry-picked from PR CiscoDevNet#312

Add server-side Lucene candidate filtering for gathered loopback interfaces with pagination, deduplication, and final local matching. Support gather-all and login-ID filtering for local users while rejecting unsupported gathered criteria. Add unit and integration coverage for filtering, validation, secrecy, and reusable output.

- Remove early return in filter_gathered_response() so deduplication runs unconditionally regardless of whether filters are provided - Add validate_gathered_filters() pre-flight check called before query_all() to reject invalid filters without wasted API calls

- Add gathered_filter_properties ClassVar to NDBaseModel for declarative filter whitelisting per module - Add pre-flight property validation (_extract_active_leaf_paths, _reject_unsupported_filter_properties) that rejects unsupported filter fields before any API call - Replace local_user custom normalize_gathered_filter validation with shared gathered_filter_properties tuple (login_id, email, first_name, last_name) - Add gathered_filter_properties to loopback model (switch_ip, interface_name, admin_state, ip, ipv6, vrf) - Pass supported_properties from model to validate_gathered_filters in state machine - Add gathered_transform support in NDOutput for modules with input/output shape differences - Fix loopback pagination: add max_pages safety cap, handle missing or invalid remaining metadata gracefully - Update nd_local_user DOCUMENTATION with supported filter properties - Update local_user unit tests to use shared validation path

- Apply black formatting to all gathered filtering module and test files - Add gathered state as no-op pass in manage_state() so modules can call manage_state unconditionally without raising InvalidState - Address review formatting feedback (trailing whitespace, blank lines)

- Extract gathered query logic into _query_existing() private helper - Move user-input validation outside try block for clean error messages - Build proposed before querying ND (fail-fast on bad config) - Declare get_argument_spec on NDBaseModel and gathered_transform on NDBaseOrchestrator; remove getattr probes for discoverability - Rename supports_gathered_lucene_filtering to supports_gathered_server_filtering (mechanism-neutral) - Normalize filters once in state machine; pass normalize_filter=None downstream to eliminate triple normalization - Return models from filter_gathered_response to avoid double Pydantic validation; use pre-built models for NDConfigCollection construction - Cap Lucene query fan-out at 3 expressions per switch; collapse to base expression beyond threshold - Raise ValueError for unknown switch_ip in gathered filters instead of silently returning empty results - Raise RuntimeError on pagination limit exhaustion instead of silently truncating gathered results - Update unit tests for new return types, renamed flags, and pre-normalized filter inputs

deekpand-cisco changed the title ~~[minor_change] Add opt-in gathered-state filtering for loopback and local users~~ Add opt-in gathered-state filtering for loopback and local users Jul 7, 2026

allenrobel assigned deekpand-cisco Jul 7, 2026

deekpand-cisco force-pushed the feature/pr312-gathered-module-filtering branch from 7a31104 to 3a3e41b Compare July 9, 2026 08:38

sivakasi-cisco added the nac01 NaC ND release 0.0.1 label Jul 14, 2026

sivakasi-cisco reviewed Jul 20, 2026

View reviewed changes

deekpand-cisco mentioned this pull request Jul 29, 2026

Track state: gathered support for NDStateMachine/orchestrator-backed modules #340

Open

deekpand-cisco added 4 commits August 3, 2026 11:09

deekpand-cisco force-pushed the feature/pr312-gathered-module-filtering branch 2 times, most recently from b2f84a4 to 687baef Compare August 3, 2026 09:39

deekpand-cisco force-pushed the feature/pr312-gathered-module-filtering branch from 687baef to 8d216fa Compare August 3, 2026 10:07

Fix implicit-str-concat pylint errors

f640d5e

deekpand-cisco force-pushed the feature/pr312-gathered-module-filtering branch 2 times, most recently from 6c0f646 to da41a29 Compare August 5, 2026 09:31

deekpand-cisco force-pushed the feature/pr312-gathered-module-filtering branch from da41a29 to 01537ab Compare August 6, 2026 05:23

deekpand-cisco marked this pull request as ready for review August 7, 2026 08:43

deekpand-cisco requested review from akinross, allenrobel, anvitha-jain, gmicol, lhercot, mikewiebe, mtarking and sajagana as code owners August 7, 2026 08:43

deekpand-cisco requested review from samiib and shrsr as code owners August 7, 2026 08:43

Navigation Menu

Uh oh!

Add opt-in gathered-state filtering for loopback and local users - #391

Add opt-in gathered-state filtering for loopback and local users#391
deekpand-cisco wants to merge 7 commits into
CiscoDevNet:developfrom
deekpand-cisco:feature/pr312-gathered-module-filtering

deekpand-cisco commented Jul 7, 2026 •

edited

Loading

Uh oh!

sivakasi-cisco Jul 20, 2026

Uh oh!

deekpand-cisco Jul 23, 2026

Uh oh!

sivakasi-cisco Jul 20, 2026

Uh oh!

deekpand-cisco Jul 23, 2026

Uh oh!

sivakasi-cisco Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		query_kwargs["gathered_filters"] = raw_config

		response_data = self.model_orchestrator.query_all(**query_kwargs)

Uh oh!

Conversation

deekpand-cisco commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issue(s)

Proposed Changes

Shared gathered-filtering framework

Loopback-interface filtering

Local-user filtering

Argument validation and gathered output

Compatibility and rollout

Test Notes

Cisco Nexus Dashboard Version

Related ND API Resource Category

Checklist

Uh oh!

sivakasi-cisco Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

deekpand-cisco Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

sivakasi-cisco Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

deekpand-cisco Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

sivakasi-cisco Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

deekpand-cisco commented Jul 7, 2026 •

edited

Loading