Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@
## Admin / Setup

- [Private repo access in Fast Bakery](docs/metaflow/private_repo_access.md)
- [Table-ownership registry view](docs/metaflow/table_ownership_registry.md)
- [Table-ownership registry](docs/metaflow/table_ownership_registry.md)

5 changes: 3 additions & 2 deletions docs/metaflow/publish.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,9 @@ Notes:
(the RFC `CREATE TAG` setup). Until then, tagging is **skipped with a warning** — the publish
still succeeds.
- Invalid `status`/`sla` values raise `ValueError` before any data is written.
- Tagged tables surface in the `TABLE_OWNERSHIP_REGISTRY` view (see
[Table-ownership registry view](table_ownership_registry.md)).
- Tagged tables surface in the `TABLE_OWNERSHIP_REGISTRY` table, which is
refreshed daily by a Metaflow flow (see
[Table-ownership registry](table_ownership_registry.md)).
- The underlying Snowflake session always has `QUOTED_IDENTIFIERS_IGNORE_CASE = TRUE`
set (`snowflake_connection.py`), so quoted identifiers in `query`/`audits` resolve
case-insensitively.
3 changes: 3 additions & 0 deletions docs/metaflow/publish_pandas.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ publish_pandas(
- Tag *definitions* must first be created by a Snowflake admin (RFC `CREATE TAG` setup);
until then tagging is **skipped with a warning** and the publish still succeeds.
- Invalid `status`/`sla` values raise `ValueError` before any data is written.
- Tagged tables surface in the `TABLE_OWNERSHIP_REGISTRY` table, which is
refreshed daily by a Metaflow flow (see
[Table-ownership registry](table_ownership_registry.md)).

## Notes

Expand Down
71 changes: 44 additions & 27 deletions docs/metaflow/table_ownership_registry.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,45 @@
# Table-ownership registry view
# Table-ownership registry

The central **table-ownership registry view**,
`PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY`, pivots the object tags on each table
into one row per table, exposing `owner`, `team`, `domain`, `project`, `status`, `sla`,
`contact` and `last_updated`.
The central **table-ownership registry**,
`PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY`, is a Snowflake **table** that
pivots the object tags on each table into one row per table, exposing `owner`,
`team`, `domain`, `project`, `status`, `sla`, `contact` and `last_updated`.

It reads the tags straight from `SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES`, so it is
**source-agnostic**: it surfaces both tables tagged automatically by
It is **refreshed daily** by
[`TableOwnershipRegistryFlow`](https://ui.pattern.obp.outerbounds.com/dashboard/p/prod/j/table_ownership_registry/b/main/workflows/tableoregistry.prod.tableowstryflow-3vt56?tab=recentRuns)
in the **prod** perimeter on Outerbounds. Query the table directly.

The refresh reads tags from `SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES`, so the
registry is **source-agnostic**: it surfaces both tables tagged automatically by
[`publish`](publish.md) / [`publish_pandas`](publish_pandas.md) *and* tables tagged
manually (e.g. by someone using Claude or plain `ALTER TABLE ... SET TAG` who doesn't use
`ds-platform-utils`). Any table carrying at least one of these tags appears, no matter how
it was tagged.
manually (e.g. by someone using Claude or plain `ALTER TABLE ... SET TAG` who
doesn't use `ds-platform-utils`). Any table carrying at least one of these tags
appears, no matter how it was tagged.

## Query

```sql
SELECT * FROM PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY
ORDER BY team, table_name;
```

This is a **one-time admin setup step**, not part of the Python API. A Snowflake admin
runs the SQL below once; the view is live thereafter (see [Notes](#notes)).
## Columns

## Create (or replace) the view
| Column | Source tag |
| -------------- | --------------- |
| `table_name` | object name |
| `owner` | `TABLE_OWNER` |
| `team` | `TABLE_TEAM` |
| `domain` | `TABLE_DOMAIN` |
| `project` | `TABLE_PROJECT` |
| `status` | `TABLE_STATUS` |
| `sla` | `TABLE_SLA` |
| `contact` | `TABLE_CONTACT` |
| `last_updated` | `LAST_UPDATED` |

The daily refresh rebuilds the table with this pivot:

```sql
CREATE OR REPLACE VIEW PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY AS
SELECT
tr.object_name AS table_name,
MAX(CASE WHEN tr.tag_name = 'TABLE_OWNER' THEN tr.tag_value END) AS owner,
Expand All @@ -40,22 +61,18 @@ WHERE tr.object_database = 'PATTERN_DB'
GROUP BY tr.object_name;
```

Then query it:

```sql
SELECT * FROM PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY
ORDER BY team, table_name;
```

## Notes

- **No refresh needed.** A view is not materialized — it re-runs its query on every read,
so it is always live.
- **~2h lag.** The view reads `SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES`, which itself lags
up to ~2 hours. For the current value of a single table's tag, use
- **Daily refresh.** The registry is a table, not a live view.
[`TableOwnershipRegistryFlow`](https://ui.pattern.obp.outerbounds.com/dashboard/p/prod/j/table_ownership_registry/b/main/workflows/tableoregistry.prod.tableowstryflow-3vt56?tab=recentRuns)
rebuilds it once a day, so newly tagged (or retagged) tables can take up to
~24 hours to appear or update.
- **~2h lag on top of that.** The refresh reads `SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES`,
which itself lags up to ~2 hours. For the current value of a single table's tag, use
`SYSTEM$GET_TAG('PATTERN_DB.DATA_SCIENCE.TABLE_OWNER', '<table>', 'table')` instead.
- **Adoption-based.** Only tables that have at least one of these tags appear in the view,
regardless of whether the tags were applied by `ds-platform-utils` or manually.
- **Adoption-based.** Only tables that have at least one of these tags appear in the
registry, regardless of whether the tags were applied by `ds-platform-utils` or
manually.
- **`last_updated`.** For tables published via `publish` / `publish_pandas` this is stamped
automatically (UTC, `YYYY-MM-DD HH:MI:SS`) on each publish. For manually tagged tables it
reflects whatever value the person set, and may be absent if the `LAST_UPDATED` tag was
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "ds-platform-utils"
version = "0.6.0"
version = "0.6.1"
description = "Utility library for Pattern Data Science."
readme = "README.md"
authors = [
Expand Down
2 changes: 1 addition & 1 deletion uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading