Skip to content

Fix rsrc_static LIKE mismatch in snowflake hub & link - #492

Open
jhue-dvelop wants to merge 4 commits into
ScalefreeCOM:mainfrom
jhue-dvelop:main
Open

Fix rsrc_static LIKE mismatch in snowflake hub & link#492
jhue-dvelop wants to merge 4 commits into
ScalefreeCOM:mainfrom
jhue-dvelop:main

Conversation

@jhue-dvelop

@jhue-dvelop jhue-dvelop commented Jul 20, 2026

Copy link
Copy Markdown

EDIT: All of this also applies to links.

Description

Background: what is rsrc_static?
When declaring a hub's source models, datavault4dbt lets you optionally set an rsrc_static parameter per source — a literal value that identifies where records from that source came from, used purely to speed up the incremental (High-Water-Mark) lookup instead of scanning the full hub for MAX(ldts). The docs and most examples use a plain literal (e.g. a source system name), but the macro also accepts SQL LIKE patterns (e.g. %some/file/path/%) for cases where the actual rsrc column contains variable values that only share a common substring/pattern — e.g. when rsrc is populated with a file path, URL, or other per-batch identifier rather than a fixed constant.

What is the bug?
In the incremental HWM path of the macro, the CTEs that look up the existing max load-timestamp per source (rsrc_static_<n>, max_ldts_per_rsrc_static_in_target) correctly compare rsrc against rsrc_static using LIKE:

src.<rsrc> LIKE '<rsrc_static>'

But the src_new CTE, which selects the actual new rows to load, filters with equality instead:

src.<rsrc> = '<rsrc_static>'

This inconsistency only matters when rsrc_static is defined as a wildcard pattern rather than a literal value. In that case, a real rsrc value is never = equal to a %...% pattern — the comparison is always false. So on any incremental run after a source has been loaded at least once (source_included_before = true), src_new selects zero rows for that source, and new business keys silently stop reaching the hub. Full/initial loads are unaffected, since this code path only runs is_incremental().

When does this occur, and why does it matter?

  • Applies only to hubs where rsrc_static is configured as a LIKE-style pattern rather than an exact literal — this is a supported, documented use case of the macro, not a misconfiguration.
  • Because the bug only manifests on the second and later incremental runs per source (the first run still has source_included_before = false), it is easy to miss in initial testing or a quick CI check that only exercises a fresh/first load.
  • Confirmed present in datavault4dbt 1.17.0, v1.18.x, v2.0.1, and main — anyone relying on wildcard rsrc_static values is affected regardless of package version, and simply upgrading the package does not resolve it.

Fix:
Changed the src_new filter from = to LIKE, making it consistent with the HWM lookup CTEs above it. The unrelated WHERE rsrc_static = '...' comparison in the lookup subqueries was intentionally left as =, since there it compares a pattern literal against itself, not against an rsrc value.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce.

  • Tests you ran:

Did test this in our data platform project where the incremental run had the issues described above. Now it works correctly.

Checklist:

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas (not neccessary from my pov)
  • I have made corresponding changes to the documentation or included information that needs updates (e.g. in the Wiki) (not neccessary from my pov)

@jhue-dvelop jhue-dvelop changed the title Fix rsrc_static LIKE mismatch in snowflake hub Fix rsrc_static LIKE mismatch in snowflake hub & link Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant