Spark: [3/N] Support sending referenced-by to all endpoints by singhpk234 · Pull Request #13979 · apache/iceberg

singhpk234 · 2025-09-03T00:54:15Z

About the change

This provides a reference implementation for passing the view name that table is referenced in as part of which the loadTable call is being made.

Details on the spec change proposal and motivation here

SPEC: Add referenced-by in loadTable API #13810

gaborkaszab

Thanks for the PR @singhpk234 !
I took a look for my own benefit, and left some observations and questions meanwhile. Let me know what you think!

gaborkaszab · 2025-09-05T08:13:36Z


+  @Override
+  public Table loadTableViaView(TableIdentifier identifier, Map<String, Object> viewContext) {
+    if (delegate instanceof ContextAwareTableCatalog) {


delegate is a BaseSessionCatalog.AsCatalog that now implements ContextAwareTableCatalog. Is there a use-case or configuration where delegate doesn't implement the new interface?

I see the concept in this class is rather to have a new member of type ContextAwareTableCatalog that is converted from delegate and this new function could directly call that. Similarly as nsCatalog.

Would this also work here? Do I miss something?

github-actions · 2025-12-04T00:18:43Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

gaborkaszab

I went through the api/ and core/ part for now (haven't checked Spark). Left some questions and nits. I also found some of my previous comments so I'm not sure this was ready for review.
Anyway, some test coverage in TestRESTCatalog would be great!

chris922 · 2026-02-20T16:02:45Z

Hi all - the Iceberg REST spec was merged 😃 This PR is now the next step in getting this referenced-by feature into the different Iceberg clients, right? Looking forward to get it merged & released

Resolve the chain of views a relation was reached through and pass it to the catalog as a LoadContext, so a REST catalog can forward it to the server. SparkSupportsLoadContext extends the loading contract with context-carrying overloads of loadTable and loadView, including the time-travel forms, and SparkCatalog and SparkSessionCatalog implement it. ResolveViews records the chain while resolving nested views, with UnResolvedRelationFromView carrying it through analysis. Cross-catalog view references fail rather than silently dropping the chain. Gated behind spark.sql.iceberg.referenced-by-enabled, off by default.

Generated-by: Codex

TestSparkCatalog only overrode loadTable(Identifier), so once the read path started resolving through loadRelation -> loadTable(Identifier, LoadContext), the double was no longer consulted and lookups fell through to the real catalog. Override the contextual variant to delegate to the existing one. The double ignores the context; TestReferencedByViewChain covers the referenced-by behaviour against real catalogs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Restore the Spark 4.1-style analyzer path for referenced-by propagation in Spark 4.2. ResolveViews now expands Iceberg views, carries the view chain through UnResolvedRelationFromView, and loads nested relations with an explicit LoadContext instead of using SQLConf-backed view metadata. Generated-by: Codex

Preserve existing cross-catalog view behavior by dropping catalog-relative referenced-by entries that cannot be represented for the target catalog. Bypass CachingCatalog for non-empty LoadContext loads so referenced-by reaches catalogs such as REST, and add regression coverage for cache-enabled Spark reads, time-travel views, and cross-catalog view references. Generated-by: Codex

Add a default-off Spark SQL flag for referenced-by view-chain propagation. When disabled, Spark keeps its native view resolution behavior, including cross-catalog views. When enabled, Iceberg uses the explicit analyzer path and rejects cross-catalog chains because LoadContext carries catalog-relative identifiers. Generated-by: Codex

Build the referenced-by view chain from the catalog and identifier Spark resolved when loading the view. Keep view metadata current catalog and namespace only for resolving unqualified references inside the view SQL. Generated-by: Codex

Keep the view chain on Spark's pre-execution table refresh by binding it to the catalog of relations resolved through a view, so a table read through a view is never reloaded as direct access. Convert view time travel with Spark's TimeTravelSpec so string and integer timestamps work and pinned snapshots are not refreshed. Leave V1 and delegating tables read through a view to Spark, and alias view-resolved tables so table-qualified columns resolve. Remove loadRelation overrides that duplicate BaseCatalog.

Move referenced-by view expansion out of ResolveViews into ResolveReferencedByViews, which does nothing when referenced-by is disabled. ResolveViews is back to its main behavior. With referenced-by enabled, BaseCatalog.loadRelation returns tables only, so Spark resolves tables itself and leaves Iceberg views unresolved for the new rule. This removes the hint-batch registration, which Spark's Hive session state ignores, and keeps a table read to a single load. Expanded views are wrapped in Spark's View node so cycle detection and cache invalidation keep working, and parse, streaming and not-found errors match Spark's. LoadContext gains equals and hashCode so cached plans over a view are reused by later queries. SparkSessionCatalog fails a load with a referenced-by chain when its Iceberg catalog cannot pass the context, instead of dropping it. Generated-by: Claude Code

…extual loads Fail referenced-by view expansion past spark.sql.view.maxNestedViewDepth with the same error Spark raises when referenced-by is disabled, so a view cycle no longer overflows the stack during analysis. CachingCatalog now sends contextual loads the same canonical identifier as cached loads, so a table read through a view and read directly reach the delegate under one name when the cache is case-insensitive. Drop two Spark tests covered by more specific ones. Generated-by: Claude Code

huaxingao

LGTM

github-actions Bot added API spark core labels Sep 3, 2025

singhpk234 mentioned this pull request Sep 3, 2025

SPEC: Add referenced-by in loadTable API #13810

Merged

singhpk234 marked this pull request as ready for review September 5, 2025 01:04

gaborkaszab reviewed Sep 5, 2025

View reviewed changes

nandorKollar reviewed Sep 5, 2025

View reviewed changes

Comment thread api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java Outdated

gaborkaszab reviewed Sep 5, 2025

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java Outdated

Comment thread core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java Outdated

singhpk234 force-pushed the feature/ref-imp-referenced-by branch 2 times, most recently from 97296ab to bcec3f2 Compare September 8, 2025 00:22

nastra reviewed Sep 22, 2025

View reviewed changes

Comment thread api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java Outdated

nastra reviewed Sep 22, 2025

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java Outdated

nastra reviewed Sep 22, 2025

View reviewed changes

Comment thread ...ain/scala/org/apache/spark/sql/catalyst/plans/logical/views/UnResolvedRelationFromView.scala Outdated

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from a34c0f2 to fb6e824 Compare November 3, 2025 12:12

singhpk234 changed the title ~~Spark, Rest: Reference implementation of referenced-by in the loadTable call~~ [REST | SPARK]: Reference implementation of referenced-by in the loadTable call Nov 3, 2025

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 20d30b3 to 737fc50 Compare November 3, 2025 12:31

github-actions Bot added the stale label Dec 4, 2025

singhpk234 added not-stale and removed stale labels Dec 4, 2025

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch 3 times, most recently from f368355 to 57e3e30 Compare January 11, 2026 05:39

gaborkaszab reviewed Jan 13, 2026

View reviewed changes

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 57e3e30 to a300631 Compare February 21, 2026 19:05

github-actions Bot added AWS GCP AZURE labels Feb 21, 2026

singhpk234 changed the title ~~[REST | SPARK]: Reference implementation of referenced-by in the loadTable call~~ [REST | SPARK]: Reference implementation of referenced-by to all endpoints for DEFINER views Feb 21, 2026

nssalian added this to the Iceberg 1.13.0 milestone Sep 17, 2026

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 684da4f to acc960c Compare September 17, 2026 18:04

singhpk234 requested a review from huaxingao September 18, 2026 00:31

huaxingao reviewed Sep 18, 2026

View reviewed changes

Comment thread spark/v4.2/spark/src/main/scala/org/apache/spark/sql/catalyst/analysis/ViewUtil.scala

Comment thread spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java

Comment thread spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java

huaxingao reviewed Sep 21, 2026

View reviewed changes

Comment thread ....2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala Outdated

singhpk234 requested a review from huaxingao September 23, 2026 18:17

huaxingao reviewed Oct 3, 2026

View reviewed changes

huaxingao reviewed Oct 8, 2026

View reviewed changes

Comment thread ....2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala Outdated

Comment thread ....2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala Outdated

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 2479b08 to be411cc Compare October 8, 2026 23:13

singhpk234 requested a review from huaxingao October 8, 2026 23:58

findinpath mentioned this pull request Oct 10, 2026

Send referenced-by when loading Iceberg REST tables & views trinodb/trino#31542

Open

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from be411cc to bce1ced Compare October 12, 2026 01:30

huaxingao reviewed Oct 14, 2026

View reviewed changes

github-actions Bot added the API label Oct 16, 2026

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch 3 times, most recently from 4693c3f to e8b01f5 Compare October 21, 2026 17:32

huaxingao reviewed Oct 22, 2026

View reviewed changes

Comment thread ...ensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveReferencedByViews.scala Outdated

Comment thread core/src/main/java/org/apache/iceberg/CachingCatalog.java Outdated

sfc-gh-prsingh and others added 11 commits October 22, 2026 12:20

Spark: Move referenced-by context to Spark 4.2

c0a391f

Generated-by: Codex

Spark: Fix ViewUtil Scala import grouping

759a96a

Generated-by: Codex

huaxingao approved these changes Oct 22, 2026

View reviewed changes

Conversation

singhpk234 commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About the change

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaborkaszab Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Dec 4, 2025

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chris922 commented Feb 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huaxingao left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

singhpk234 commented Sep 3, 2025 •

edited

Loading