diff --git a/README.md b/README.md index f65531b..54d8565 100644 --- a/README.md +++ b/README.md @@ -13,5 +13,5 @@ ## Admin / Setup - [Private repo access in Fast Bakery](docs/metaflow/private_repo_access.md) -- [Table-ownership registry view](docs/metaflow/table_ownership_registry.md) +- [Table-ownership registry](docs/metaflow/table_ownership_registry.md) diff --git a/docs/metaflow/publish.md b/docs/metaflow/publish.md index b5f044e..b67d945 100644 --- a/docs/metaflow/publish.md +++ b/docs/metaflow/publish.md @@ -104,8 +104,9 @@ Notes: (the RFC `CREATE TAG` setup). Until then, tagging is **skipped with a warning** — the publish still succeeds. - Invalid `status`/`sla` values raise `ValueError` before any data is written. -- Tagged tables surface in the `TABLE_OWNERSHIP_REGISTRY` view (see - [Table-ownership registry view](table_ownership_registry.md)). +- Tagged tables surface in the `TABLE_OWNERSHIP_REGISTRY` table, which is + refreshed daily by a Metaflow flow (see + [Table-ownership registry](table_ownership_registry.md)). - The underlying Snowflake session always has `QUOTED_IDENTIFIERS_IGNORE_CASE = TRUE` set (`snowflake_connection.py`), so quoted identifiers in `query`/`audits` resolve case-insensitively. diff --git a/docs/metaflow/publish_pandas.md b/docs/metaflow/publish_pandas.md index 869d0f3..0e3a343 100644 --- a/docs/metaflow/publish_pandas.md +++ b/docs/metaflow/publish_pandas.md @@ -79,6 +79,9 @@ publish_pandas( - Tag *definitions* must first be created by a Snowflake admin (RFC `CREATE TAG` setup); until then tagging is **skipped with a warning** and the publish still succeeds. - Invalid `status`/`sla` values raise `ValueError` before any data is written. +- Tagged tables surface in the `TABLE_OWNERSHIP_REGISTRY` table, which is + refreshed daily by a Metaflow flow (see + [Table-ownership registry](table_ownership_registry.md)). ## Notes diff --git a/docs/metaflow/table_ownership_registry.md b/docs/metaflow/table_ownership_registry.md index 0b5e92f..30bbb99 100644 --- a/docs/metaflow/table_ownership_registry.md +++ b/docs/metaflow/table_ownership_registry.md @@ -1,24 +1,45 @@ -# Table-ownership registry view +# Table-ownership registry -The central **table-ownership registry view**, -`PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY`, pivots the object tags on each table -into one row per table, exposing `owner`, `team`, `domain`, `project`, `status`, `sla`, -`contact` and `last_updated`. +The central **table-ownership registry**, +`PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY`, is a Snowflake **table** that +pivots the object tags on each table into one row per table, exposing `owner`, +`team`, `domain`, `project`, `status`, `sla`, `contact` and `last_updated`. -It reads the tags straight from `SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES`, so it is -**source-agnostic**: it surfaces both tables tagged automatically by +It is **refreshed daily** by +[`TableOwnershipRegistryFlow`](https://ui.pattern.obp.outerbounds.com/dashboard/p/prod/j/table_ownership_registry/b/main/workflows/tableoregistry.prod.tableowstryflow-3vt56?tab=recentRuns) +in the **prod** perimeter on Outerbounds. Query the table directly. + +The refresh reads tags from `SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES`, so the +registry is **source-agnostic**: it surfaces both tables tagged automatically by [`publish`](publish.md) / [`publish_pandas`](publish_pandas.md) *and* tables tagged -manually (e.g. by someone using Claude or plain `ALTER TABLE ... SET TAG` who doesn't use -`ds-platform-utils`). Any table carrying at least one of these tags appears, no matter how -it was tagged. +manually (e.g. by someone using Claude or plain `ALTER TABLE ... SET TAG` who +doesn't use `ds-platform-utils`). Any table carrying at least one of these tags +appears, no matter how it was tagged. + +## Query + +```sql +SELECT * FROM PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY +ORDER BY team, table_name; +``` -This is a **one-time admin setup step**, not part of the Python API. A Snowflake admin -runs the SQL below once; the view is live thereafter (see [Notes](#notes)). +## Columns -## Create (or replace) the view +| Column | Source tag | +| -------------- | --------------- | +| `table_name` | object name | +| `owner` | `TABLE_OWNER` | +| `team` | `TABLE_TEAM` | +| `domain` | `TABLE_DOMAIN` | +| `project` | `TABLE_PROJECT` | +| `status` | `TABLE_STATUS` | +| `sla` | `TABLE_SLA` | +| `contact` | `TABLE_CONTACT` | +| `last_updated` | `LAST_UPDATED` | + +The daily refresh rebuilds the table with this pivot: ```sql -CREATE OR REPLACE VIEW PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY AS SELECT tr.object_name AS table_name, MAX(CASE WHEN tr.tag_name = 'TABLE_OWNER' THEN tr.tag_value END) AS owner, @@ -40,22 +61,18 @@ WHERE tr.object_database = 'PATTERN_DB' GROUP BY tr.object_name; ``` -Then query it: - -```sql -SELECT * FROM PATTERN_DB.DATA_SCIENCE.TABLE_OWNERSHIP_REGISTRY -ORDER BY team, table_name; -``` - ## Notes -- **No refresh needed.** A view is not materialized — it re-runs its query on every read, - so it is always live. -- **~2h lag.** The view reads `SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES`, which itself lags - up to ~2 hours. For the current value of a single table's tag, use +- **Daily refresh.** The registry is a table, not a live view. + [`TableOwnershipRegistryFlow`](https://ui.pattern.obp.outerbounds.com/dashboard/p/prod/j/table_ownership_registry/b/main/workflows/tableoregistry.prod.tableowstryflow-3vt56?tab=recentRuns) + rebuilds it once a day, so newly tagged (or retagged) tables can take up to + ~24 hours to appear or update. +- **~2h lag on top of that.** The refresh reads `SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES`, + which itself lags up to ~2 hours. For the current value of a single table's tag, use `SYSTEM$GET_TAG('PATTERN_DB.DATA_SCIENCE.TABLE_OWNER', '', 'table')` instead. -- **Adoption-based.** Only tables that have at least one of these tags appear in the view, - regardless of whether the tags were applied by `ds-platform-utils` or manually. +- **Adoption-based.** Only tables that have at least one of these tags appear in the + registry, regardless of whether the tags were applied by `ds-platform-utils` or + manually. - **`last_updated`.** For tables published via `publish` / `publish_pandas` this is stamped automatically (UTC, `YYYY-MM-DD HH:MI:SS`) on each publish. For manually tagged tables it reflects whatever value the person set, and may be absent if the `LAST_UPDATED` tag was diff --git a/pyproject.toml b/pyproject.toml index be797b3..e551d22 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "ds-platform-utils" -version = "0.6.0" +version = "0.6.1" description = "Utility library for Pattern Data Science." readme = "README.md" authors = [ diff --git a/uv.lock b/uv.lock index edfd3a9..ec64521 100644 --- a/uv.lock +++ b/uv.lock @@ -571,7 +571,7 @@ wheels = [ [[package]] name = "ds-platform-utils" -version = "0.6.0" +version = "0.6.1" source = { editable = "." } dependencies = [ { name = "jinja2" },