Fix rsrc_static LIKE mismatch in snowflake hub & link - #492
Open
jhue-dvelop wants to merge 4 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Background: what is
rsrc_static?When declaring a hub's source models, datavault4dbt lets you optionally set an
rsrc_staticparameter per source — a literal value that identifies where records from that source came from, used purely to speed up the incremental (High-Water-Mark) lookup instead of scanning the full hub forMAX(ldts). The docs and most examples use a plain literal (e.g. a source system name), but the macro also accepts SQLLIKEpatterns (e.g.%some/file/path/%) for cases where the actualrsrccolumn contains variable values that only share a common substring/pattern — e.g. whenrsrcis populated with a file path, URL, or other per-batch identifier rather than a fixed constant.What is the bug?
In the incremental HWM path of the macro, the CTEs that look up the existing max load-timestamp per source (
rsrc_static_<n>,max_ldts_per_rsrc_static_in_target) correctly comparersrcagainstrsrc_staticusingLIKE:But the
src_newCTE, which selects the actual new rows to load, filters with equality instead:This inconsistency only matters when
rsrc_staticis defined as a wildcard pattern rather than a literal value. In that case, a realrsrcvalue is never=equal to a%...%pattern — the comparison is alwaysfalse. So on any incremental run after a source has been loaded at least once (source_included_before = true),src_newselects zero rows for that source, and new business keys silently stop reaching the hub. Full/initial loads are unaffected, since this code path only runsis_incremental().When does this occur, and why does it matter?
rsrc_staticis configured as aLIKE-style pattern rather than an exact literal — this is a supported, documented use case of the macro, not a misconfiguration.source_included_before = false), it is easy to miss in initial testing or a quick CI check that only exercises a fresh/first load.main— anyone relying on wildcardrsrc_staticvalues is affected regardless of package version, and simply upgrading the package does not resolve it.Fix:
Changed the
src_newfilter from=toLIKE, making it consistent with the HWM lookup CTEs above it. The unrelatedWHERE rsrc_static = '...'comparison in the lookup subqueries was intentionally left as=, since there it compares a pattern literal against itself, not against anrsrcvalue.Type of change
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce.
Did test this in our data platform project where the incremental run had the issues described above. Now it works correctly.
Checklist: