New validation for missing vnsRsCIfAttN#394
Conversation
| relation_dn = relation_attributes["dn"].strip() | ||
| if not relation_dn: | ||
| continue | ||
| target_dn = get_target_dn(relation_attributes) |
There was a problem hiding this comment.
dn is the full path of the relationship object, and tDn is the path of the target interface object it is attached to.
No need of check for tDN.
There was a problem hiding this comment.
i agree. its not resolved. at all.
There was a problem hiding this comment.
Comparison now uses full relation DN instead of tDn and handles rscIfAtt vs rscIfAttN differences before matching.
| continue | ||
| relation_attributes = vnsRsCIfAttN["vnsRsCIfAttN"]["attributes"] | ||
| if "dn" not in relation_attributes: | ||
| continue |
There was a problem hiding this comment.
when mo is not configured, vnsRsCIfAttNs will be empty.. above if conditions not needed.
There was a problem hiding this comment.
Removed if conditions already when mo is not configured
|
|
||
|
|
||
| @check_wrapper(check_title="Cleanup vnsRsCIfAtt usage in services") | ||
| def vns_rscifatt_cleanup_check(tversion, **kwargs): |
There was a problem hiding this comment.
change the proper function name and title
There was a problem hiding this comment.
Updated name and title
|
|
||
| Impact: | ||
|
|
||
| If any `vnsRsCIfAtt` relation exists without a matching `vnsRsCIfAttN` for the same concrete interface target (`tDn`), the upgrade is outage-risky and should be treated as affected. |
There was a problem hiding this comment.
please update the document as per impact and requirement.
| continue | ||
| missing_dn = old_relation_dn_by_key[relation_key] | ||
| tenant_name, device_name, logical_interface, concrete_interface = parse_relation_context(missing_dn) | ||
| data.append([tenant_name, device_name, logical_interface, concrete_interface, missing_dn]) |
There was a problem hiding this comment.
where is that fault case handled for post upgrade case..
fault : F1690
uni/tn-CSCwj49418/lDevVip-test/lIf-intf-cons/vnsConfIssue-lif-invalid-CIf
Configuration is invalid due to LIf has an invalid CIf
There was a problem hiding this comment.
As per below comment added by Lovkesh,
"After doing upgarde from 605h to 615e.
fab3-apic# moquery -c vnsRsCIfAtt | grep dn
fab3-apic# moquery -c vnsRsCIfAttN | grep dn
dn : uni/tn-CSCwj49418/lDevVip-test/lIf-intf-prov/rscIfAttN-[uni/tn-CSCwj49418/lDevVip-test/cDev-cdev/cIf-[prov]]
fab3-apic#
Gui shows cIf-[prov] deleted from concrete device interface.
and fault raised
F1690
Minor
2026-05-29T13:37:55.525+00:00
Raised
uni/tn-CSCwj49418/lDevVip-test/lIf-intf-cons/vnsConfIssue-lif-invalid-CIf
Configuration is invalid due to LIf has an invalid CIf
Config
configuration-failed
ignorable:no, issue:lif-invalid-CIf. this Fault removed asap i attached the cluster interface."
It'll be seen after upgrade to 6.1(5e). But if older mo config is not there in the new mo config, we are recommending cu to add the missing cluster interface in the lower version itself before doing upgrade. Then we will not see the fault post upgradation and that's the reason current check exists what I feel.
There was a problem hiding this comment.
do not consider any fault in this case. You can ignore that. please do repro and test the pre-post upgrade and script.
There was a problem hiding this comment.
Yes, have recreated the issue in 5.2(4d) and 5.3(2f) and upgraded to 6.0(5h) and validated the script. I can do the validation one more time by upgrading to 6.1(5e)
| [N9K-C9408 with more than 5 N9K-X9400-16W LEMs][d31] | CSCws82819 | :white_check_mark: | :no_entry_sign: | ||
| [Multi-Pod Modular Spine Bootscript File][d32] | CSCwr66848 | :white_check_mark: | :no_entry_sign: | ||
| [Inband Management Policy Misconfiguration][d33]| CSCwd40071 | :white_check_mark: | :no_entry_sign: | ||
| [Cleanup vnsRsCIfAtt usage in services][d34] | CSCwr51759 | :white_check_mark: | :no_entry_sign: |
There was a problem hiding this comment.
"Check missing vnsRsCIfAttN" ---> change the name
| [d31]: #n9k-c9408-with-more-than-5-n9k-x9400-16w-lems | ||
| [d32]: #multi-pod-modular-spine-bootscript-file | ||
| [d33]: #inband-management-policy-misconfiguration | ||
| [d34]: #cleanup-vnsrscifatt-usage-in-services |
|
|
||
| This check will verify the count of the `svccoreCtrlr` Managed Object and raise and alarm with the bug if object count found more than 240. Remove the content or objects of `svccoreCtrlr` or `svccoreNode`. Contact Cisco TAC or upgrade to a release containing the fix for CSCws84232 before proceeding with an upgrade. | ||
|
|
||
| ### Cleanup vnsRsCIfAtt usage in services |
|
|
||
| ### Cleanup vnsRsCIfAtt usage in services | ||
|
|
||
| Due to [CSCwr51759][70], when targeting 6.0(3)+, having only `vnsRsCIfAtt` without the corresponding `vnsRsCIfAttN` under the same `vnsLIf` can leave service graph interface attachment in an inconsistent state. |
There was a problem hiding this comment.
wrong buug mentioned. Please do the cleanup for bogus info.
There was a problem hiding this comment.
when upgrading to 6.0(3) and above, 'vnsRsCIfAtt' get deleted and without creating the corresponding 'vnsRsCIfAttN' under the same vnsLIf. this will leave service graph interface attachment in an inconsistent state
|
|
||
| Impact: | ||
|
|
||
| If any `vnsRsCIfAtt` relation exists without a matching `vnsRsCIfAttN` for the same concrete interface target (`tDn`), the upgrade is outage-risky and should be treated as affected. |
There was a problem hiding this comment.
no need for this ---> If any vnsRsCIfAtt relation exists without a matching vnsRsCIfAttN for the same concrete interface target (tDn), the upgrade is outage-risky and should be treated as affected.
|
|
||
| Suggestion: | ||
|
|
||
| Before the upgrade, add the missing `vnsRsCIfAttN` relation under the same cluster interface (`vnsLIf`) with the same concrete interface target (`tDn`). |
There was a problem hiding this comment.
specify the GUI path.. as discussed earlier
| return Result(result=ERROR, msg="Error occurred while fetching svccore object counts: {}".format(str(e)), doc_url=doc_url) | ||
|
|
||
|
|
||
| @check_wrapper(check_title="Cleanup vnsRsCIfAtt usage in services") |
| headers = ["Tenant", "Device Name", "Cluster Interface", "Missing Concrete Interface", "vnsRsCIfAtt DN"] | ||
| data = [] | ||
| recommended_action = ( | ||
| "Mo vnsRsCIfAtt is deprecated >=6.0(3d). Before upgrade, under Services, add the missing concrete interface as vnsRsCIfAttN under the same cluster interface" |
There was a problem hiding this comment.
this is not explainatery enough. please work on it.
There was a problem hiding this comment.
Updated recommended action with additional details.
|
|
||
| vnsRsCIfAttNs = icurl("class", "vnsRsCIfAttN.json?rsp-prop-include=config-only") | ||
|
|
||
| def get_target_dn(relation_attributes): |
There was a problem hiding this comment.
look for whole DN, not just Tdn. . this seems wrong to me.
you will see one small diff there for RsCifAtt/RsCifAttN --> make sure you cover this while comparing DN.
There was a problem hiding this comment.
Updated by considering full DN.
Harinadh-Saladi
left a comment
There was a problem hiding this comment.
Addressed all the comments
| [N9K-C9408 with more than 5 N9K-X9400-16W LEMs][d31] | CSCws82819 | :white_check_mark: | :no_entry_sign: | ||
| [Multi-Pod Modular Spine Bootscript File][d32] | CSCwr66848 | :white_check_mark: | :no_entry_sign: | ||
| [Inband Management Policy Misconfiguration][d33]| CSCwd40071 | :white_check_mark: | :no_entry_sign: | ||
| [Cleanup vnsRsCIfAtt usage in services][d34] | CSCwr51759 | :white_check_mark: | :no_entry_sign: |
| [d31]: #n9k-c9408-with-more-than-5-n9k-x9400-16w-lems | ||
| [d32]: #multi-pod-modular-spine-bootscript-file | ||
| [d33]: #inband-management-policy-misconfiguration | ||
| [d34]: #cleanup-vnsrscifatt-usage-in-services |
|
|
||
| This check will verify the count of the `svccoreCtrlr` Managed Object and raise and alarm with the bug if object count found more than 240. Remove the content or objects of `svccoreCtrlr` or `svccoreNode`. Contact Cisco TAC or upgrade to a release containing the fix for CSCws84232 before proceeding with an upgrade. | ||
|
|
||
| ### Cleanup vnsRsCIfAtt usage in services |
|
|
||
| ### Cleanup vnsRsCIfAtt usage in services | ||
|
|
||
| Due to [CSCwr51759][70], when targeting 6.0(3)+, having only `vnsRsCIfAtt` without the corresponding `vnsRsCIfAttN` under the same `vnsLIf` can leave service graph interface attachment in an inconsistent state. |
|
|
||
| Impact: | ||
|
|
||
| If any `vnsRsCIfAtt` relation exists without a matching `vnsRsCIfAttN` for the same concrete interface target (`tDn`), the upgrade is outage-risky and should be treated as affected. |
|
|
||
|
|
||
| @check_wrapper(check_title="Cleanup vnsRsCIfAtt usage in services") | ||
| def vns_rscifatt_cleanup_check(tversion, **kwargs): |
There was a problem hiding this comment.
Updated name and title
| headers = ["Tenant", "Device Name", "Cluster Interface", "Missing Concrete Interface", "vnsRsCIfAtt DN"] | ||
| data = [] | ||
| recommended_action = ( | ||
| "Mo vnsRsCIfAtt is deprecated >=6.0(3d). Before upgrade, under Services, add the missing concrete interface as vnsRsCIfAttN under the same cluster interface" |
There was a problem hiding this comment.
Updated recommended action with additional details.
|
|
||
| vnsRsCIfAttNs = icurl("class", "vnsRsCIfAttN.json?rsp-prop-include=config-only") | ||
|
|
||
| def get_target_dn(relation_attributes): |
There was a problem hiding this comment.
Updated by considering full DN.
| relation_dn = relation_attributes["dn"].strip() | ||
| if not relation_dn: | ||
| continue | ||
| target_dn = get_target_dn(relation_attributes) |
There was a problem hiding this comment.
Comparison now uses full relation DN instead of tDn and handles rscIfAtt vs rscIfAttN differences before matching.
| continue | ||
| missing_dn = old_relation_dn_by_key[relation_key] | ||
| tenant_name, device_name, logical_interface, concrete_interface = parse_relation_context(missing_dn) | ||
| data.append([tenant_name, device_name, logical_interface, concrete_interface, missing_dn]) |
There was a problem hiding this comment.
As per below comment added by Lovkesh,
"After doing upgarde from 605h to 615e.
fab3-apic# moquery -c vnsRsCIfAtt | grep dn
fab3-apic# moquery -c vnsRsCIfAttN | grep dn
dn : uni/tn-CSCwj49418/lDevVip-test/lIf-intf-prov/rscIfAttN-[uni/tn-CSCwj49418/lDevVip-test/cDev-cdev/cIf-[prov]]
fab3-apic#
Gui shows cIf-[prov] deleted from concrete device interface.
and fault raised
F1690
Minor
2026-05-29T13:37:55.525+00:00
Raised
uni/tn-CSCwj49418/lDevVip-test/lIf-intf-cons/vnsConfIssue-lif-invalid-CIf
Configuration is invalid due to LIf has an invalid CIf
Config
configuration-failed
ignorable:no, issue:lif-invalid-CIf. this Fault removed asap i attached the cluster interface."
It'll be seen after upgrade to 6.1(5e). But if older mo config is not there in the new mo config, we are recommending cu to add the missing cluster interface in the lower version itself before doing upgrade. Then we will not see the fault post upgradation and that's the reason current check exists what I feel.
| fabric_link_redundancy_check, | ||
| apic_downgrade_compat_warning_check, | ||
| svccore_excessive_data_check, | ||
| vns_rscifattn_missing_check, |
There was a problem hiding this comment.
vns_rscifattn_missing_check --> should be vnsrscifattn_missing_check correct it everywhere.
| continue | ||
| missing_dn = old_relation_dn_by_key[relation_key] | ||
| tenant_name, device_name, logical_interface, concrete_interface = parse_relation_context(missing_dn) | ||
| data.append([tenant_name, device_name, logical_interface, concrete_interface, missing_dn]) |
There was a problem hiding this comment.
do not consider any fault in this case. You can ignore that. please do repro and test the pre-post upgrade and script.
|
|
||
| ### Check missing vnsRsCIfAttN | ||
|
|
||
| When upgrading to 6.0(3) and above, 'vnsRsCIfAtt' get deleted and without creating the corresponding 'vnsRsCIfAttN' under the same vnsLIf will leave the service graph interface attachment in an inconsistent state |
There was a problem hiding this comment.
When upgrading to 6.0(3) and above, 'vnsRsCIfAtt' get deleted without creating 'vnsRsCIfAttN' under 'vnsLIf' . this leave the service graph interface attachment in an inconsistent state
There was a problem hiding this comment.
Updated by correcting a few grammatical errors.
|
|
||
| When upgrading to 6.0(3) and above, 'vnsRsCIfAtt' get deleted and without creating the corresponding 'vnsRsCIfAttN' under the same vnsLIf will leave the service graph interface attachment in an inconsistent state | ||
|
|
||
| Before the upgrade, in APIC GUI navigate to Tenant > Services > L4-L7 > Device and open cluster interface `intf-prov`. If concrete interface `cons` is missing, re-add concrete interface `cons` under the same cluster interface. so the corresponding `vnsRsCIfAttN` relation exists. |
There was a problem hiding this comment.
For all impacted Dn in this check, re-attach the Concrete interfaces associated to cluster interface under Devices in Services L4-L7 tab.
Tenant --> Services --> L4-L7 --> Devices (Device_name) --> cluster interface --> Concrete interfaces
There was a problem hiding this comment.
Updated by correcting a few grammatical errors.
lovkeshsharma702
left a comment
There was a problem hiding this comment.
please work on given logic.,. validate this in lab again.
Harinadh-Saladi
left a comment
There was a problem hiding this comment.
Yesterday have recreated the issue in 5.3(2f) and validated the script. Enclosed updated script validation logs, pytest and full script run logs
|
|
||
| ### Check missing vnsRsCIfAttN | ||
|
|
||
| When upgrading to 6.0(3) and above, 'vnsRsCIfAtt' get deleted and without creating the corresponding 'vnsRsCIfAttN' under the same vnsLIf will leave the service graph interface attachment in an inconsistent state |
There was a problem hiding this comment.
Updated by correcting a few grammatical errors.
|
|
||
| When upgrading to 6.0(3) and above, 'vnsRsCIfAtt' get deleted and without creating the corresponding 'vnsRsCIfAttN' under the same vnsLIf will leave the service graph interface attachment in an inconsistent state | ||
|
|
||
| Before the upgrade, in APIC GUI navigate to Tenant > Services > L4-L7 > Device and open cluster interface `intf-prov`. If concrete interface `cons` is missing, re-add concrete interface `cons` under the same cluster interface. so the corresponding `vnsRsCIfAttN` relation exists. |
There was a problem hiding this comment.
Updated by correcting a few grammatical errors.
| continue | ||
| missing_dn = old_relation_dn_by_key[relation_key] | ||
| tenant_name, device_name, logical_interface, concrete_interface = parse_relation_context(missing_dn) | ||
| data.append([tenant_name, device_name, logical_interface, concrete_interface, missing_dn]) |
There was a problem hiding this comment.
Yes, have recreated the issue in 5.2(4d) and 5.3(2f) and upgraded to 6.0(5h) and validated the script. I can do the validation one more time by upgrading to 6.1(5e)
| fabric_link_redundancy_check, | ||
| apic_downgrade_compat_warning_check, | ||
| svccore_excessive_data_check, | ||
| vns_rscifattn_missing_check, |
|
|
||
| To avoid this issue, change the target version to another version. Or verify that the `bootscript` file exists in the bootflash of each modular spine switch prior to upgrading to 6.1(4h). If the file is missing, you have to do clean reboot on the impacted spine to ensure that `/bootflash/bootscript` gets created again. In case you already upgraded your spine and you are experiencing the traffic impact due to this issue, clean reboot of the spine will restore the traffic. | ||
|
|
||
|
|
There was a problem hiding this comment.
Removed unwanted lines at line numbers 43 and 210
| ( | ||
| { | ||
| vnsRsCIfAtt_api: read_data(dir, "vnsRsCIfAtt_match.json"), | ||
| vnsRsCIfAttN_api: read_data(dir, "vnsRsCIfAttN_missing_cons.json"), |
There was a problem hiding this comment.
Added test case for empty vnsRsCIfAttN scenario
| old_relation_dn_by_key = {} | ||
| for vnsRsCIfAtt in vnsRsCIfAtts: | ||
| if "vnsRsCIfAtt" not in vnsRsCIfAtt: | ||
| continue |
There was a problem hiding this comment.
Can you try this optimized script.
new_dn_keys = set()
for new_no in vnsRsCIfAttNs:
try:
dn = new_no["vnsRsCIfAttN"]["attributes"]["dn"].strip()
except (KeyError, TypeError, AttributeError):
continue
if dn:
new_dn_keys.add(dn.replace("/rscIfAttN-[", "/rscIfAtt-[", 1))
for old_mo in vnsRsCIfAtts:
try:
old_dn = old_mo["vnsRsCIfAtt"]["attributes"]["dn"].strip()
except (KeyError, TypeError, AttributeError):
continue
if not old_dn:
continue
if old_dn.replace("/rscIfAttN-[", "/rscIfAtt-[", 1) in new_dn_keys:
continue
match = re.search(
r"uni/tn-(?P<tenant>[^/]+)/lDevVip-(?P<device>[^/]+)/lIf-(?P<lif>[^/]+)/"
r"rscIfAtt-\[.*?/cIf-\[(?P<cif>[^\]]+)\]\]",
old_dn,
)
data.append([
match.group("tenant") if match else "",
match.group("device") if match else "",
match.group("lif") if match else "",
match.group("cif") if match else "",
old_dn,
])
There was a problem hiding this comment.
The above suggested optimized loop logic works functionally, but without sorting it can return rows in different orders based on API/input order. Our Pytest case compares as an ordered list, so it fails intermittently on order only. I added sorting to make pytest stable and output consistent.
Harinadh-Saladi
left a comment
There was a problem hiding this comment.
Addressed the comments and tested in the lab with code changes and enclosed the latest logs
|
|
||
| To avoid this issue, change the target version to another version. Or verify that the `bootscript` file exists in the bootflash of each modular spine switch prior to upgrading to 6.1(4h). If the file is missing, you have to do clean reboot on the impacted spine to ensure that `/bootflash/bootscript` gets created again. In case you already upgraded your spine and you are experiencing the traffic impact due to this issue, clean reboot of the spine will restore the traffic. | ||
|
|
||
|
|
There was a problem hiding this comment.
Removed unwanted lines at line numbers 43 and 210
| ( | ||
| { | ||
| vnsRsCIfAtt_api: read_data(dir, "vnsRsCIfAtt_match.json"), | ||
| vnsRsCIfAttN_api: read_data(dir, "vnsRsCIfAttN_missing_cons.json"), |
There was a problem hiding this comment.
Added test case for empty vnsRsCIfAttN scenario
| old_relation_dn_by_key = {} | ||
| for vnsRsCIfAtt in vnsRsCIfAtts: | ||
| if "vnsRsCIfAtt" not in vnsRsCIfAtt: | ||
| continue |
There was a problem hiding this comment.
The above suggested optimized loop logic works functionally, but without sorting it can return rows in different orders based on API/input order. Our Pytest case compares as an ordered list, so it fails intermittently on order only. I added sorting to make pytest stable and output consistent.
| if not tversion: | ||
| return Result(result=MANUAL, msg=TVER_MISSING, doc_url=doc_url) | ||
|
|
||
| if tversion.older_than("6.0(3d)"): |
There was a problem hiding this comment.
Add cversion check
current code is pre-6.0.3
There was a problem hiding this comment.
As discussed, cversion check is not required. When we are providing tversion as 6.0(3) and above if both old mo and new mo are not configured then we are alerting the customer with recommended action. This is equivalent by checking the cversion <6.0(3) and verifying the old and new mo config unavailability and alerting the customer.
|
|
||
| if old_dn.replace("/rscIfAttN-[", "/rscIfAtt-[", 1) in new_dn_keys: | ||
| continue | ||
|
|
There was a problem hiding this comment.
add this check
if none of vnsRsCIfAtt and vnsRsCIfAttN present. only in case Both of Mo missing attached to given vnsLif—> vnsLIf has NEITHER relation - service graph broken, ask Cu to do manual check warning only
There was a problem hiding this comment.
Added manual check warning when both old and new mo config is unavailable.
| old_dn, | ||
| ]) | ||
|
|
||
| data.sort(key=itemgetter(-1)) |
There was a problem hiding this comment.
what is the need of this sort and new library imported?
There was a problem hiding this comment.
The above suggested optimized loop logic works functionally, but without sorting it can return rows in different orders based on API/input order. Our Pytest case compares as an ordered list, so it fails intermittently on order only. I added sorting to make pytest stable and output consistent.
Have removed new library imported and replaced with lambda instead.
| [d31]: #n9k-c9408-with-more-than-5-n9k-x9400-16w-lems | ||
| [d32]: #multi-pod-modular-spine-bootscript-file | ||
| [d33]: #inband-management-policy-misconfiguration | ||
|
|
Harinadh-Saladi
left a comment
There was a problem hiding this comment.
Addressed the comments and tested the code with changes in lab and enclosed comprehensive logs.
| continue | ||
| relation_attributes = vnsRsCIfAttN["vnsRsCIfAttN"]["attributes"] | ||
| if "dn" not in relation_attributes: | ||
| continue |
There was a problem hiding this comment.
Removed if conditions already when mo is not configured
| old_dn, | ||
| ]) | ||
|
|
||
| data.sort(key=itemgetter(-1)) |
There was a problem hiding this comment.
The above suggested optimized loop logic works functionally, but without sorting it can return rows in different orders based on API/input order. Our Pytest case compares as an ordered list, so it fails intermittently on order only. I added sorting to make pytest stable and output consistent.
Have removed new library imported and replaced with lambda instead.
| if not tversion: | ||
| return Result(result=MANUAL, msg=TVER_MISSING, doc_url=doc_url) | ||
|
|
||
| if tversion.older_than("6.0(3d)"): |
There was a problem hiding this comment.
As discussed, cversion check is not required. When we are providing tversion as 6.0(3) and above if both old mo and new mo are not configured then we are alerting the customer with recommended action. This is equivalent by checking the cversion <6.0(3) and verifying the old and new mo config unavailability and alerting the customer.
|
|
||
| if old_dn.replace("/rscIfAttN-[", "/rscIfAtt-[", 1) in new_dn_keys: | ||
| continue | ||
|
|
There was a problem hiding this comment.
Added manual check warning when both old and new mo config is unavailable.
Added a new pre-upgrade validation to detect missing vnsRsCIfAttN relations for existing vnsRsCIfAtt entries under the same cluster interface and target concrete interface.
Enclosed comprehensive logs for the same.
vns_rscifattn_missing_check_Test_Logs.txt
vns_rscifattn_missing_check_Pytest_Logs.txt
Full_script_run_logs.txt