Skip to content

chore: back-merge main into development - #699

Open
github-actions[bot] wants to merge 53 commits into
developmentfrom
main
Open

chore: back-merge main into development#699
github-actions[bot] wants to merge 53 commits into
developmentfrom
main

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Automated back-merge to keep development in sync with main, triggered by a push to main. Merge this with a merge commit (not squash) so the merge-base advances and this PR does not regenerate. If the ruleset blocks merge commits, an admin must allow it or add the back-merge token to the ruleset bypass list. If GitHub reports conflicts, resolve them on this PR before merging. (Workflow: .github/workflows/backmerge-main-to-development.yml.)

will-yuponce-db and others added 30 commits June 19, 2026 08:44
Add a merge_group trigger to the Test Coverage workflow so the five
required status checks (Backend Tests, Frontend Tests, TypeScript Type
Check, Requirements Lockfile Check, Frontend Build Check) report on the
gh-readonly-queue branch GitHub builds for each queued batch. Without it,
PRs entering a merge queue on main/develop would hang waiting on checks
that never run.

Also extend the lockfile drift gate to hard-fail under merge_group: the
auto-commit fixup is PR-only (it cannot push to the read-only queue
branch), so the queue must fail on drift rather than skip the check.

Co-authored-by: Isaac

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
The collection picker was disabled whenever a default collection prop was
passed, which the business terms view always does. As a result users could
never switch away from the first editable collection. Disable the picker
only when there is at most one editable collection, treating the prop as a
default seed value.
A data contract created via POST /api/data-contracts was saved with no
owner: create_contract_with_relations never set draft_owner_id and only
set owner_team_id when an owner was supplied. Under the PRD #442
role-aware visibility filter a family is "elevated" only for its
draft_owner or owner_team members, so an owner-less draft was invisible
to its own creator (and the create dialog showed no row in the list).

Additionally GET /api/data-contracts had no `status` query parameter, so
`?status=draft` was silently ignored by FastAPI.

Changes:
- Stamp the creator as draft_owner_id when a draft is created without an
  owning team, mirroring the existing personal-draft clone path.
- Clear draft_owner_id on the draft->proposed transition so a contract
  under steward review is promoted from creator-only to org visibility.
- Add a real `status` filter to GET /api/data-contracts, applied after
  the visibility filter so it can only narrow what a caller may see.
- Unit tests for the status filter and personal-draft owner visibility.

Fixes regression ONT-CUJ-006.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
Quality rules authored on a schema column (the schema editor's per-column
"Quality" tab) were accepted in the dialog but silently dropped on save:
GET /api/data-contracts/{id} came back with schema.quality=[] and the
column carried no quality field, so ODCS export omitted the rules.

Root cause: column rules arrive nested under properties[].quality, but
_create_schema_objects (used by both create and update) only ever
persisted the property row itself and never read its quality list. The
only quality persistence path was the separate object-level qualityRules
field. ColumnProperty also didn't declare `quality`, relying on
extra="allow" to keep it.

Changes:
- Declare `quality: List[QualityRule]` on the ColumnProperty API model.
- Extract the rule->DataQualityCheckDb mapping into a shared
  _build_quality_check_db helper that sets property_id for column rules.
- Persist properties[].quality in _create_schema_objects, bound to both
  object_id and property_id.
- On update, scope the object-level qualityRules replacement to
  property_id IS NULL so it no longer deletes the column rules persisted
  by the schema recreation in the same save.
- Unit tests for column-rule persistence and object/property coexistence.

Fixes regression ONT-CUJ-008.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…iew stub (#529)

request_steward_review constructed an in-memory ReviewedAsset, logged
"Created asset review record", and never persisted it — misleading dead
code that made it look like a steward review record was created when none
was. A proposed contract was therefore only discoverable by direct URL.

Proposed contracts actually reach stewards through two real paths that
already exist:
  - GET /api/approvals/queue lists contracts in proposed/under_review
    status, where the steward approves/rejects them.
  - The ON_REQUEST_REVIEW workflow trigger runs configured review
    workflows.
The /data-asset-reviews surface targets Unity Catalog assets and requires
an explicit reviewer; routing contracts there needs a steward-assignment
policy that does not exist yet.

Changes:
- Remove the unsaved ReviewedAsset stub and its false success log (and the
  now-unused AssetType/ReviewedAssetStatus import).
- Document the two real surfacing paths inline.
- Add tests: requesting review transitions draft->proposed and the
  contract then appears in the approvals queue; review from a non-draft
  status is rejected.

Addresses regression ONT-CUJ-012 (with contract-list discoverability also
improved in the draft-owner-visibility fix).

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
POST /api/data-contracts/{id}/approve returned HTTP 500 ("Failed to
approve contract") for a proposed contract, leaving it stuck in
"proposed". The approve endpoint is documented as accepting
PROPOSED/UNDER_REVIEW -> APPROVED and pre-checks that source status, but
DATA_CONTRACT_TRANSITIONS only listed ["draft", "under_review",
"deprecated"] for "proposed" — "approved" was missing. transition_status
therefore raised ValueError("Invalid status transition"), which the route
caught with a bare `except Exception` and reported as a 500.

Changes:
- Add "approved" to the allowed transitions from "proposed" so a steward
  can approve directly (under_review stays an optional intermediate step).
- In the approve and reject routes, catch ValueError and return 409 with
  the message instead of an opaque 500, so any invalid transition is a
  clean client error.
- Tests: lifecycle map allows proposed->approved; transition_status
  proposed->approved succeeds; a genuinely invalid jump still raises.

Fixes regression ONT-CUJ-013.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…ins (#531)

Adding a deliverable to a data product failed with PUT 403 for both the
Producer who created it and an Admin, because:

1. create_product never assigned an owner. A new product had no
   project_id, owner_team_id, or draft_owner_id, so the update
   authorization cascade failed for everyone except a workspace admin —
   the creator couldn't edit their own product, and it could never be
   assigned an owner since that also requires PUT.
2. The update ownership cascade only short-circuited on workspace-admin
   (group based), ignoring the caller's feature-level data-products
   permission. An in-app role override granting data-products=Admin was
   therefore denied when the caller wasn't the object owner.
3. GET /api/delivery-methods required the delivery-methods feature, which
   a Producer lacks, so the Add Deliverable delivery-method dropdown was
   empty (403).

Changes:
- create_product stamps the creator as draft_owner_id when no project,
  team, or draft owner is supplied (mirrors the personal-draft clone).
- update_product_with_auth accepts is_feature_admin and short-circuits on
  it; the update route resolves the caller's effective data-products
  permission (honoring in-app role overrides) and passes it in.
- Gate the delivery-methods read endpoints on data-products:READ_ONLY
  (reading the catalog is a prerequisite for authoring deliverables);
  writes stay on delivery-methods:READ_WRITE.
- Tests: creator becomes draft owner; explicit team owner not overwritten;
  feature-admin can edit orphan/others' products; non-admin still
  fail-closed.

Fixes regression ONT-CUJ-015.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…tion (#532)

POST /api/data-products/{id}/request-certify returned 202 Accepted for a
product with zero deliverables (no output ports), letting an empty product
into the certification workflow. The documented "at least one deliverable
required" guard was never enforced.

Changes:
- Block request-certify with 409 when the product has no output ports.
- Tests: request-certify without a deliverable returns 409; with a
  deliverable returns 202.

Fixes regression ONT-CUJ-019 and ONT-NEG-008.

Note: the request-certify endpoint intentionally uses a request-based
model (it records a certification request and fires the on_request_certify
workflow) rather than directly transitioning Draft->Proposed. The separate
/submit-certification endpoint performs the draft->proposed transition.
This change only adds the missing deliverable precondition; it does not
alter the request-vs-transition model.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…in (#533)

Creating a data contract whose name duplicated an existing contract in the
same domain succeeded silently (POST 200, duplicate persisted) — no 409,
no inline error.

create_contract_with_relations always creates a *new* version family (new
versions go through create_version), so any existing contract sharing the
same name+domain is a genuine duplicate rather than another version of the
same family. Add a case-insensitive name + domain check that raises
ConflictError (surfaced as HTTP 409 by the create route, matching the
existing duplicate-id behavior). Same name in a different domain is still
allowed.

Tests: duplicate name in the same domain raises ConflictError (incl.
case-insensitive); a distinct name still creates successfully.

Fixes regression ONT-NEG-002.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…1) (#535)

GET /api/data-products/{id} returned the full product to any caller with
data-products:READ_ONLY, with no status or ownership check. A Data
Consumer could therefore read a draft (uncertified) product directly by
id, even though the marketplace listing correctly hides drafts.

Add a read gate to the get-by-id route:
- Published products (active/deprecated) remain readable by anyone with
  READ_ONLY — the catalog/marketplace contract.
- Unpublished products (draft/proposed/under_review/approved/...) are
  readable only by data-products admins (incl. via in-app role override)
  and owners (draft_owner / owning team / project member), reusing the
  same ownership scope as the listing.
- A denied read returns 404 (not 403) so the draft's existence isn't
  disclosed.

Tests cover the gate directly: published readable by all; draft denied
for a non-owner consumer; draft allowed for a feature admin and for an
owner in the accessible set.

Fixes regression ONT-NEG-011.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…2F%2F path collapse (#536)

* fix(concepts): add /by-iri query-param routes to survive %2F%2F path collapse

The Databricks Apps proxy decodes and collapses `%2F%2F` to a single `/`
in path segments, mangling IRIs like `http://ontos.example.org/...` into
`http:/ontos.example.org/...` before they reach FastAPI. This broke every
concept detail page (`/concepts/browser/{iri}`) with "Concept not found"
regardless of which concept was clicked.

Query-string values survive that transformation unchanged. Existing
sibling endpoints (`/concepts/hierarchy?iri=`, `/neighbors?iri=`) already
work via the query-param shape — this PR aligns the rest of the
`{concept_iri:path}` route family on the same pattern.

Approach:
- Add new `/concepts/by-iri?iri=<urlencoded>` (semantic-models) and
  `/knowledge/concepts/by-iri?iri=<urlencoded>` (knowledge) route families.
- Keep the existing path-form routes as deprecated aliases for one
  migration window so external bookmarks don't 404. Both shapes share
  module-level `_*_payload` helpers — no duplicated logic.
- Frontend (`concept-detail.tsx`, `node-links-panel.tsx`,
  `concepts-search.tsx`) switched to the query-param form.
- Backend integration tests cover both shapes for the read path and at
  least one mutating path.
- Drive-by fix: `get_concept_hierarchy` referenced an undefined name
  `concept_iri` in its exception logger (parameter is named `iri`).

Routes declared in `by-iri`-first order — FastAPI matches in declaration
order and the path-form parameter is greedy, so the reverse order would
shadow the new routes.

Co-authored-by: Isaac

* test(concepts): correct lifecycle smoke test for /by-iri routes

Direct draft → published transition is rejected by the manager (must go
through under_review first), so swap the lifecycle smoke test to use the
submit-review action — which is the right shape for proving the
/by-iri/<action> routes are wired without re-testing the state machine.

Co-authored-by: Isaac
POST /api/compliance/policies was sharing the same Pydantic CompliancePolicy
model with PUT, which marks id as required. Pydantic rejected every UI-driven
create with 422 "field required" before the route handler could even generate
a UUID.

This commit splits the create contract from the read/update model:

- New CompliancePolicyCreate model with only the fields a client must supply
  (name, description, rule, failure_message, is_active, severity, category).
- POST /api/compliance/policies now accepts CompliancePolicyCreate.
- ComplianceManager.create_policy generates the UUID server-side via uuid4().
- Defensive secondary fix per #235: the read-side CompliancePolicy.compliance
  field gets default=0.0 so stale rows and migrations cannot 500 on read.
- PUT contract is unchanged on purpose; the bug was create-only.

No DB schema change is required — compliance_policies.id is already a string
primary key with no DB-level default, and the manager continues to supply the
UUID explicitly.

Unit + integration tests updated to match the new contract.

Closes #235

Co-authored-by: Isaac
Internal/computed named graphs (urn:app-entities, urn:demo, urn:semantic-links,
urn:meta:sources, the rdflib default graph) were leaking into the RDF Sources
list once the in-memory graph contained triples (e.g. after the first ontology
upload with a Data Contract present). These pseudo-rows had no backing file, so
their Preview action 404'd with "Failed to load content."

Filter internal contexts at the /api/semantic-models boundary and derive the
frontend table filter from the canonical SYSTEM_RDF_NAMESPACE_KEYS map so the
two stay in sync. Caching and Knowledge Graph stats are intentionally left
untouched.
…525)

Expand src/backend/src/tests/data with a broader set of ODCS contract and
ODPS data product YAML fixtures so tests can cover Ontos-specific extensions
(polymorphic owners, namespaced unified tags, semantic assignments, v3.1.0
Team-as-object, ports, consumer principals) on top of the baseline Bitol specs.

- Add canonical odcs/full-example.odcs.yaml referenced by the existing
  roundtrip and export-validation integration tests (was previously missing).
- Add ODCS fixtures: ontos-extensions superset, team-as-object, minimal,
  composite-key relationships, and a quality-rules matrix.
- Add new odps/ folder with full example, ontos-extensions superset, minimal,
  ports matrix, and a mixed valid/invalid multi-product batch.
- Document folder layout and the three Ontos encoding conventions in
  tests/data/README.md.
…ew (ONT-NEG-005) (#534)

* fix(contracts): require a schema before a contract can be proposed for review

An empty draft contract (no schema objects) could be proposed for review:
POST /api/data-contracts/{id}/request-review returned 200 with
status -> proposed. There is nothing for a steward to review on an empty
draft, and the documented "schema required before proposal" guard was not
enforced.

Changes:
- request_steward_review raises ValueError when the contract has zero
  schema objects, leaving the contract in draft.
- The request-review route now surfaces the validation message on the 409
  (previously masked as a generic "Invalid review request") so the UI can
  explain why the request was blocked; "not found" still maps to 404.
- Tests: an empty draft cannot be proposed (and stays draft); a draft with
  a schema object can.

Fixes regression ONT-NEG-005.

* test(contracts): seed schema object in steward-review happy-path fixture

The new ONT-NEG-005 guard in request_steward_review blocks DRAFT→PROPOSED
when a contract has no schema objects. The happy-path test's draft_contract
fixture created a schema-less contract, so it began failing with
"A schema is required before a contract can be proposed for review."
Seed one SchemaObjectDb so the fixture represents a reviewable draft.

Co-authored-by: Isaac

---------

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
Estate Manager 'View Details' built the target URL from the current
location (/estate-manager/${id}) instead of the registered route
(/estates/:estateId), producing /estate-manager/1 which rendered the
404 page. Navigate to /estates/${id} via an exported buildEstateDetailPath
helper and add a unit test guarding the path contract.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
The Add Connection dialog hardcoded a config block only for BigQuery, so
selecting Snowflake rendered only the generic Name/Description/Enabled/Default
fields with no credential/config inputs. The backend already exposes Snowflake
config field hints (account, user, warehouse, database, default_schema, role)
via /api/connections/types, but the frontend form ignored them.

Add a Snowflake config block mirroring the BigQuery pattern, extend the form
schema and reset logic, and extract the per-connector config-field mapping into
a pure, unit-tested helper (buildConnectionConfig / CONNECTOR_CONFIG_FIELDS).

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
The Catalog Commander Info panel surfaced only notes/links/documents
(EntityMetadataPanel) for a selected table, with no way to apply a tag.
The backend already exposes a generic, free-form entity-tag assignment
API (GET/POST /api/entities/{entity_type}/{entity_id}/tags[:set]) used
elsewhere, so this is a frontend surfacing gap rather than a missing
feature.

Add a focused EntityTagsPanel that reads assigned tags and applies them
via tags:set using the existing TagSelector/TagChip components, gated on
the tags READ_WRITE permission. Wire it into the Info panel for the
selected catalog object (entity_type 'catalog-object', matching the
Comments panel).

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
The Data Contracts list page rendered the New Contract and Upload create
controls for every persona. A Data Consumer (READ_ONLY) could open the
create dialog and only hit a 403 from POST /api/data-contracts on save.

Gate the create/upload controls behind the same permission check the Data
Products list uses: usePermissions().hasPermission('data-contracts',
READ_WRITE). When the persona lacks write access the controls are not
rendered, so the unauthorized action is never surfaced. Backend 403
enforcement is unchanged.

Adds a focused vitest (data-contracts.permissions.test.tsx) proving the
control is hidden for read-only personas and shown for write-capable ones.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
Adds a workflow that, on every push to main (and via workflow_dispatch),
ensures an open PR exists merging main -> development, keeping the
development branch in sync with main. PR-based rather than a direct push
so it respects branch protection on development and surfaces merge
conflicts for manual resolution. No-ops when development already contains
main, and reuses an existing open back-merge PR instead of creating
duplicates.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
On a successful certification request, a pre-review data product
(draft/sandbox) now transitions to 'proposed' so it surfaces in the
review/approvals queue. Previously request-certify returned 202 but the
product stayed in 'draft' (the documented Draft -> Proposed transition
never occurred). The ONT-NEG-008 zero-deliverable guard is preserved and
still runs before the transition.

Fixes ONT-CUJ-019.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
Custom Properties created via the concept editor were always stored as
skos:Concept and rendered as "Concept" in the UI because the backend
ignored concept_type, property_type, domain, and range on write and had
no way to recover them on read. The fix wires those fields end-to-end so
every dialog option (concept, class, property/{object,datatype,annotation},
individual, term) round-trips correctly.

- models/ontology.py: expose property_type/domain/range on
  OntologyConcept, ConceptCreate, and ConceptUpdate (incl. concept_type
  on update for draft retypes).
- controller/semantic_models_manager.py:
  * _resolve_concept_rdf_types / _detect_concept_type centralise the
    mapping between UI types and rdf:type triples (owl:Class,
    owl:ObjectProperty, owl:DatatypeProperty, owl:AnnotationProperty,
    owl:NamedIndividual, skos:Concept + ontos:conceptType for "term").
  * create_concept writes the resolved rdf:type set plus
    rdfs:domain/rdfs:range when supplied.
  * update_concept allows draft retyping and partial domain/range
    updates without clobbering unspecified fields.
  * Read paths (_extract_concept_metadata, get_concept,
    get_concept_details, _compute_all_concepts) surface
    property_type/domain/range from the graph.
  * promote_concept / migrate_concept forward the new fields.
- routes/semantic_models_routes.py: pass property_type/domain/range
  through POST /knowledge/concepts and PATCH
  /knowledge/concepts/{iri}.
- tests/integration/test_knowledge_routes.py: add
  TestKnowledgeConceptTypes covering create+read for all type
  combinations and partial PATCH updates of property fields.

Verified end-to-end via Playwright for all 7 dialog options
(concept, class, property{object,datatype,annotation}, individual, term).
)

The back-merge workflow (#548) used actions/checkout@v4, which fails the
databrickslabs/ontos policy requiring all actions be pinned to a
full-length commit SHA. Pin to the same SHA the other workflows in this
repo already use (df4cb1c069e1874edd31b4311f1884172cec0e10 # v6).

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
Hide several not-ready surfaces from the default V1 experience.

Navigation surfaces (maturity gating, default showAlpha=false hides alpha):
- Catalog Commander: flip maturity beta -> alpha so it is hidden by default
  like the other three Deploy/Govern surfaces.
- Estate Manager, Entitlements, Security Features: already alpha; now also
  hidden from home-page Quick Actions.

Quick Actions previously filtered only by permission, not maturity, so they
still linked to alpha surfaces even when those were hidden from the sidebar.
quick-actions.tsx now also gates on allowedMaturities from
useFeatureVisibilityStore, mirroring navigation.tsx, while keeping the
existing permission filtering.

Mockup connectors:
- list_connector_types() now filters out connectors whose is_available is
  False (the Snowflake/PowerBI/Kafka stubs hard-code False), so they no longer
  appear in the Add Connection dropdown. The default connector type is always
  retained so the primary connection path is never hidden.

Added a focused unit test for the connector-type availability filtering.

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…cts (#554)

PR #535 added a direct-read gate to GET /api/data-products/{id} but it was
ineffective for read-only consumers (ONT-NEG-011 still failing on deployed
main). A Data Consumer with data-products READ_ONLY could still fetch a
draft product by id and get 200.

Root cause: the prior gate resolved the caller's project scope via
projects_manager.get_user_projects, then checked whether the product was in
manager.list_products(...). But get_user_projects treats ANY group whose
name merely contains the substring "admin" (e.g. the common workspace
"account-admins" group, distinct from the configured app-admin "admins"
group) as a global admin and returns EVERY project. Such a consumer got
caller_project_ids = all projects, which matched the draft's project_id in
list_products, so the gate returned True and leaked the draft. Verified by
a real-DB repro: with the old gate a consumer in "account-admins" reads a
draft in a project they do not belong to; with the new gate they get 404.

Fix: stop trusting the broad listing scope. The gate now decides from the
product's own ownership facts using membership-based checks only:
  * published (active/deprecated) stays readable by everyone (catalog
    contract);
  * data-products feature admins (incl. in-app role override) see all;
  * otherwise grant only genuine ownership of THIS product —
    draft_owner_id == caller, owner_team_id in the caller's real team
    memberships (teams_manager.get_teams_for_user), or project membership
    via projects_manager.is_user_project_member (which uses configured
    admin groups, not a substring match).
A denied read returns 404 so the draft's existence isn't disclosed.

Tests: real-DB regression proving a consumer in an "admin"-substring group
is denied a draft in an unowned project (and published stays readable, and
the draft owner is allowed); updated the existing mock-based gate tests to
the new ownership-fact logic. Validated via unit/manager tests; the
test_data_*_routes.py integration suites have a pre-existing audit-service
fixture gap unrelated to this change (same 7 manager failures on clean
main).

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…ueue (#556)

The approvals queue (GET /api/approvals/queue) listed data products with
status == 'draft' and so missed exactly the products awaiting steward
review. Since #549 (ONT-CUJ-019), request-certify transitions a pre-review
product (draft/sandbox) to 'proposed', so a submitted product is never
'draft' anymore and never surfaced to the steward ('No results') while
not-yet-submitted drafts were shown instead.

Mirror the contracts filter and query products in 'proposed'/'under_review'
status so submitted-for-review products appear in the steward review queue.

Fixes ONT-CUJ-020 (part a).
…557)

`PUT /api/settings` logged the inbound payload at INFO level via
`logger.info(f"Received settings update request: {settings_payload}")`,
so any GitHub PAT, MCP token, or future secret field in the request
body would be written to plaintext logs (and potentially picked up by
log shippers).

Today the schema doesn't include a secret-shaped field, so the leak is
forward-looking. But the log line is type-agnostic — the moment a
`github_token` / `git_token` / OAuth-secret sibling is added to the
settings payload, it leaks. This adds a small, well-scoped redactor so
the gate is in place before that happens.

Changes:
- New helper `redact_sensitive_keys()` in `common/logging.py`.
  Case-insensitive substring match on a curated set
  (`secret`, `token`, `password`, `credential`, `api_key`, `private_key`,
  `client_secret`, `bearer`, `github_pat`, `mcp_token`, …). Recurses
  into nested dicts and lists. Never mutates the input — the route
  still hands the real payload to `manager.update_settings`. The set
  intentionally excludes `path` so settings like
  `WORKSPACE_DEPLOYMENT_PATH` are not clobbered.
- `routes/settings_routes.py`: wraps the leaky log line and adds an
  allowlist-discipline comment on the audit-log `details` dict so
  future contributors do not mirror the full payload there either.
- 11 unit tests in `tests/unit/test_log_redaction.py` covering
  pass-through, known-secret keys, case-insensitivity, substring match,
  nested dicts, lists of dicts, the `path`-not-redacted regression,
  non-string keys, and the no-mutation contract.

## Test plan

- [x] `hatch -e dev run pytest backend/src/tests/unit/test_log_redaction.py -v --no-cov` → 11 passed
- [x] Confirmed via `git stash` round-trip that the two pre-existing
  failures in `test_settings_manager::test_get_settings_returns_dict`
  and `test_settings_routes::test_get_settings` reproduce on clean
  `upstream/main` HEAD and are unrelated to this change (stale mock /
  stale response-shape assertion).

Co-authored-by: Isaac
Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
…ok payloads (#581)

Three related fixes for Daimler Truck process-workflow webhook templates:

1. **workflow_executor.py** – strip the `user:` prefix that PrincipalPicker
   serialises approver specs with (e.g. `user:foo@bar.com`).  Without this,
   `_resolve_role_to_users` stored the prefixed string as the notification
   recipient so approval tasks were invisible in the admin UI.

2. **access_grants_manager.py** – add `consumer_principals_value` scalar to
   `entity_data` (first CP group's `value` field).  Allows webhook templates to
   use `${entity.consumer_principals_value}` to embed the UC access-group name
   directly without quoting a JSON array.

3. **approval-wizard-dialog.tsx** – thread the `on_behalf_of` step payload
   into `collectedFieldsRef` / `wizard_data` so the process workflow receives
   it in `entity_data` and `${context.on_behalf_of.value}` resolves correctly
   in webhook body templates.

Fixes #579.
When a user clicks "Assign" next to a team member in the Copy from Team
dialog, the Assign Owner dialog now opens with email and display name
already populated from that member's data.

Co-authored-by: Isaac
mvkonchits-db and others added 23 commits July 6, 2026 10:29
- Backend: add ReferenceDocumentConfig model (label + url) and optional
  references field to UserActionStepConfig, LegalDocumentStepConfig, and
  AcknowledgementChecklistStepConfig
- Frontend: extend simpleMarkdown() to convert [text](url) to safe <a>
  tags (target=_blank, rel=noopener); render step descriptions via
  dangerouslySetInnerHTML so links are clickable; render checklist item
  labels the same way
- Add ReferenceDocuments sub-component: renders a border-separated list
  of labelled ExternalLink icons; wired into user_action, legal_document,
  and acknowledgement_checklist step renderers
- Types: add ReferenceDocumentConfig interface; add references field to
  LegalDocumentStepConfig and AcknowledgementChecklistStepConfig

Co-authored-by: Isaac
… Actions inline (#572)

Three connected changes that make the approval-response surface consistent
across all workflow trigger types (not just access_grant):

**backend** (`workflow_executor.py`):
- `_extract_approval_facets` now has a "universal extras" block that runs for
  every trigger type: surfaces `on_behalf_of`, `step_results`, and the full
  entity dict as `full_payload`. Previously only `access_grant` populates
  structured facets; now any paused approval step passes its context through.

**frontend** (`workflow-approval-response-dialog.tsx`):
- `WorkflowApprovalResponseDialogPayload` extended with `on_behalf_of` and
  `full_payload`; `buildDetailRows` surfaces both plus any unknown keys from
  `full_payload` (SKIP_KEYS list guards internal IDs).
- Approver can adjust `granted_duration_days` and `permission_level` before
  submitting (fields shown only when the original request carries those values).
- `handleSubmit` passes the adjusted values in the POST body.

**frontend** (`required-actions-section.tsx`):
- `workflow_approval` notifications now appear in the My Actions unified table
  with an amber "Approval" badge and an inline CheckSquare button that opens
  `WorkflowApprovalResponseDialog` directly — no navigation required.

Closes #572
…sible_asset_ids

list_products(is_admin=False) with no caller scope intentionally returns
empty (fail-closed). Switch to is_admin=True + is_visible_consumer filter
so Data Consumers see assets from any active/deprecated DP without
requiring ownership.
…ter test

_mock_dpm omitted `status` from the SimpleNamespace it returned, so
`is_visible_consumer` saw an empty string and filtered every product out,
causing two test failures after the is_admin=True + is_visible_consumer fix.

Adds a dedicated test asserting that draft-status DPs returned by
list_products(is_admin=True) do not contribute assets to a consumer's
visible set.

Co-authored-by: Isaac
… sent

Two defects made 'Create Genie Space' fail silently end to end:

1. genie_client sent {display_name, tables[].full_name, instructions} to
   POST /api/2.0/genie/spaces, but the API requires title/description/
   warehouse_id at the top level plus the space definition as a JSON
   string in serialized_space ({version: 2, data_sources.tables[]
   .identifier, instructions.text_instructions[].content}). The request
   400'd on the missing warehouse_id before anything else. warehouse_id
   now falls back to settings.DATABRICKS_WAREHOUSE_ID (the app already
   has the sql-warehouse resource attached). Schema verified against the
   live API.

2. NotificationsManager defines create_notification twice; the later
   object-based definition (notification, db) shadows the async kwargs
   version, so the Genie flow's three notifications (started / ready /
   failed) all raised TypeError (unexpected keyword argument 'user_id')
   and were swallowed: the user got zero feedback. The kwargs version is
   renamed create_user_notification; its four call sites (Genie x3,
   projects_manager) are migrated, and the Genie sites now pass
   type=NotificationType.* instead of the nonexistent status= kwarg.

Same defect family as the shadowed duplicate MCP tool classes (87224e1).

Co-authored-by: Isaac
…nd notifications

- collect_datasets_from_products fed port.asset_identifier straight to the
  Genie API, but that field is often schema-level (catalog.schema), which
  Genie rejects ('Invalid table identifier'). Table FQNs are now resolved
  from the port contract's schema objects (physical_name); asset_identifier
  is used only when it is a 3-part table name.
- The Ready/Failed notifications created inside the background task were
  silently rolled back: notification_repo.create only flushes, and the
  task's session closes without commit. Explicit commits added.

Co-authored-by: Isaac
…tings; bump to v1.0.1

iam.current-user:read and iam.access-control:read were added in PR #351
to enable OBO group resolution in /api/user/details. However, the
Marketplace listing manifest is fixed at publish time — adding scopes to
manifest.yaml does not propagate to any existing or new installs from the
current listing; it only takes effect after a new listing version is
published.

Because these scope names are also rejected as invalid on several
Databricks workspaces (fevm-valcon-demo), the safest path is to remove
them from the manifest until a new listing version can be prepared. For
direct-deploy customers who need group resolution, the scopes must be
added manually via `databricks apps update`.

Version bumped to 1.0.1 to produce a clean Marketplace patch release.
fix(genie): space creation from data products was broken end to end
…604)

* ci: fix development branch filters and back-merge runner IP failure

Two fixes for the stalled development merge flow:

- test-coverage.yml listened on 'develop', a branch that does not exist;
  the working branch is 'development'. PRs targeting development got no
  required checks, so they could never enter the merge queue.
- backmerge-main-to-development.yml ran on stock ubuntu-latest runners,
  whose IPs are not on the org IP allow list, so every GitHub API call
  failed and main never synced back into development. Moved to the
  protected runner group and switched the PR step to plain REST via
  gh api instead of the GraphQL-backed gh pr subcommands.

Co-authored-by: Isaac

* docs: fix stale develop branch name in testing plan example

Co-authored-by: Isaac

---------

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
* chore(deps): bump pytz from 2026.1.post1 to 2026.2 in /src

Bumps [pytz](https://github.com/stub42/pytz) from 2026.1.post1 to 2026.2.
- [Release notes](https://github.com/stub42/pytz/releases)
- [Commits](stub42/pytz@release_2026.1.post1...release_2026.2)

---
updated-dependencies:
- dependency-name: pytz
  dependency-version: '2026.2'
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: regenerate requirements lockfiles

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
#638)

publish_product accesses port.data_contract_id in three places, but the
OutputPortDb column is contract_id; data_contract_id exists nowhere on the
model (no column, property, or alias). Publishing any product that has at
least one output port therefore raises AttributeError, which the route's
generic exception handler surfaces as HTTP 500 ("Failed to publish
product"), hiding the cause.

Impact: the canonical publish action (approved -> active + publication
scope, WITH validation that every output port has an approved contract) is
unusable for real products. The alternate paths that still reach the
marketplace (change-status to active, then set-publication-scope or
handle-publish) skip that contract validation entirely, so the only
governance-checked route to publication is the broken one.

Introduced in 0b3920d (route-to-manager refactor), where the manager was
written against a field name the model never had.

Verified on a live deployment: publish returned 500 for a product with
three contracted output ports before the change and succeeds after it,
with publication_scope set and the product visible in the marketplace.

Co-authored-by: Isaac
(cherry picked from commit fe1326c)
Readiness check #2 read op.customProperties.get("contract_id"), but
customProperties on the OutputPort API model is a List[CustomProperty]:
.get() on a non-empty list raises AttributeError, and no code path ever
writes contract_id into customProperties anyway. The linked contract
lives on OutputPort.contractId (ORM column contract_id), which is what
the rest of the codebase reads (e.g. data_products_manager). As a
result the check could never count a single contract and products
never passed this readiness item even with contracts linked on every
port.

Co-authored-by: Isaac
(cherry picked from commit a561fcf2ca1a5311f972e6c5c54e71949c2f81e4)
…osed (#640)

* fix(compliance): app-entity loader never yielded anything

AppEntityLoader called repo.list() but repositories expose get_multi()
(CRUDBase), imported two repositories by nonexistent module names
(domains_repository, data_asset_review_repository), and read attributes
the Db models do not have (owner, tags, description on products). The
blanket try/except around the whole generator swallowed the resulting
AttributeError/ImportError, so every compliance rule over data_product,
data_contract, domain, or review evaluated zero entities and scored 0.

Fix: use get_multi(), correct the imports, read the real columns
(owner_team_id, domain_id, description_purpose) with safe getattr, and
isolate each entity type in its own try block so one failure cannot
suppress the other types.

Co-authored-by: Isaac
(cherry picked from commit 63249d1a1f90601a3cb68a6e4467337e4560a6a7)

* fix(compliance): products repo is fail-closed; evaluate system-wide with is_admin

Co-authored-by: Isaac
(cherry picked from commit 4d025d714c0e245a40e093f9e2852405098d8223)
* fix(mcp): tools/list crashed for scoped tokens on required_scope=None tools

get_app_state and search_ontos_handbook declare required_scope=None (no
scope needed), and _has_scope evaluated `":" in None` -> TypeError for any
token without the "*" wildcard. Scoped (least-privilege) tokens could never
list tools.

Co-authored-by: Isaac
(cherry picked from commit 0b61baeb187d6fc11366622873a714174daa9597)

* fix(tools): add missing required_scope on duplicate tool class definitions

data_products.py and data_contracts.py each define get/delete (and
search_data_contracts) twice; the later definition wins at import time and
lacked required_scope, silently inheriting the admin-only '*' default. Any
least-privilege token (e.g. read-only) was denied get_data_product /
get_data_contract even with the right read scope.

Co-authored-by: Isaac
(cherry picked from commit e242e8759a6a0e6a831cb08f3a0a69f9f368d00e)
…lients (#642)

* fix(mcp): JSON-RPC spec compliance for strict clients (MCP Python SDK)

- Never reply to notifications: unknown notifications (e.g.
  notifications/cancelled from CrewAI) previously got a METHOD_NOT_FOUND
  error with id=null, which the MCP Python SDK cannot parse — the SSE
  stream crashed and Kasal reconnect-looped. Notifications now return
  202 Accepted with no body per the streamable HTTP spec.
- Serialize exactly one of result/error: model_dump() emitted both keys
  (one null), rejected by strict JSONRPCMessage validation.
- Parse the body before auth so auth/validation errors echo the request id
  instead of id=null.

Co-authored-by: Isaac
(cherry picked from commit 6d2a33d921e156412a3e1b0a68f62cdf72d11e2a)

* fix(mcp): release DB session before long-lived SSE stream

FastAPI holds route dependencies until the response finishes; for the GET
SSE stream that pinned one pooled DB connection per open stream for the
stream's whole lifetime. MCP clients that auto-reconnect (Kasal / MCP
Python SDK) exhausted the pool within seconds, after which token lookups
failed (requests got 401 'Invalid or missing API key') and all other
requests hung on the pool timeout. The stream loop only uses the in-memory
session store, so the DB session can be returned to the pool up front.

Co-authored-by: Isaac
(cherry picked from commit 18e18c79e11796846ecb238e33a58a8e6ceb2827)
- app.state.workspace_client was never set, so the analytics MCP tools
  (get_table_schema, execute_analytics_query, explore_catalog_schema)
  always failed with 'Workspace client not available' — agents could not
  query any data through Ontos.
- get_data_product ignored Description model objects (description came
  back null) and returned no output ports — agents saw a bare product
  with no pointers to the physical tables.
- search_data_contracts read DataContractsManager._contracts, a legacy
  in-memory dict that is never populated — always zero results. Query
  the contracts table directly.

Co-authored-by: Isaac
(cherry picked from commit cd809ac25ef1b5b080b3bbd50c402ead30682236)
…ts (#644)

Databricks foundation-model endpoints for Claude (e.g.
us.anthropic.claude-opus-4-8) reject the temperature parameter with 400
BAD_REQUEST, so generation always failed on those endpoints. Retry the
request without temperature when the endpoint rejects it.

Co-authored-by: Isaac
(cherry picked from commit c65dfd50dac69ea4f7d63584aaab3222d9ae8652)
* chore: migrate frontend and docs site to npm-only

Replace yarn (frontend) and the mixed yarn/pnpm/npm lockfiles (docs site)
with npm exclusively across the whole repo.

- Lockfiles: delete src/frontend/yarn.lock and .npmrc (package-lock=false);
  delete website/ontos yarn.lock + pnpm-lock.yaml; keep npm package-lock.json
  in both projects.
- CI: setup-node-jfrog action now uses npm caching and a lock-path input, and
  drops the yarn.lock registry sed rewrite (npm's replace-registry-host default
  rewrites resolved-URL hosts to the JFrog registry at install time); convert
  test-coverage.yml to npm ci / npm run; add a dependabot npm entry for the
  docs site.
- Scripts/hooks: build_static.sh, src/package.json license-check scripts,
  pre-commit hooks, pyproject dev-frontend, playwright webServer, and script
  message/comment strings.
- Docs/rules: cursor rule 09, CLAUDE.md, README, CONTRIBUTING (incl. simplified
  private-mirror guidance), docs/*, planning docs, i18n README, and the
  website docs.

Co-authored-by: Isaac

* fix(frontend): regenerate package-lock.json with all platform deps

The initial lockfile was generated on top of a yarn-populated node_modules
on macOS, so npm only recorded the host platform's optional deps (1 @esbuild
entry). CI's Linux npm ci then failed with EUSAGE (missing @esbuild/linux-*
etc. from lock file). Regenerated cleanly from an empty node_modules so all
26 @esbuild platform variants are present.

Co-authored-by: Isaac

* fix(frontend): point package-lock resolved URLs at registry.npmjs.org

The lockfile was generated on a machine whose ~/.npmrc points at an internal
npm proxy, so all 1087 resolved URLs pointed at npm-proxy.dev.databricks.com.
npm's replace-registry-host (default "npmjs") only rewrites registry.npmjs.org
hosts to the configured registry, so in CI the proxy URLs were left as-is and
npm ci hung trying to reach an unreachable host until the job timed out.

Rewrite the resolved-URL host to registry.npmjs.org (the proxy mirrors npmjs
with an identical path layout), so npm rewrites it to the JFrog registry at
install time in CI. Verified npm ci + build + type-check locally against a
proxy-only egress environment, exercising the same host-rewrite path.

Co-authored-by: Isaac

---------

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
Release the npm-only frontend build (#657) so the Databricks Apps
Marketplace listing and git-install path can point at a tagged version.

Also fixes a long-standing bump_version.py bug: the pyproject.toml regex
was unanchored and matched `target-version` under [tool.ruff], writing a
semver ("1.0.1", "1.0.2", ...) where ruff expects a Python target like
py311. Anchored the pattern to line start and corrected target-version
to py311 (matches requires-python >=3.11).

Co-authored-by: Isaac
…ions idle (#596)

* fix(dfra): concept ownership + property links + RDF dedup + notifications idle

Batch of fixes from customer report ("dfra"):

- Owners on concept/term/property didn't render: business_owners `by-object` /
  `history` and metadata `/entities/{type}/{id}/{rich-texts,links,documents,
  attachments}` and semantic-links `/iri/…` used path params for IRI-valued
  ids, which the Databricks Apps proxy mangles by collapsing `%2F%2F` in path
  segments (see #536). Switched to `?iri=` / `?object_id=` / `?entity_id=`
  query params — inert to that transformation — matching the pattern #536
  established for the concepts routes. Frontend callers (`ownership-panel`,
  `entity-metadata-panel`, `linked-objects-panel`) updated accordingly.
- Business concepts/properties disappeared from contract columns because
  `data-contract-details.tsx` declared `propertyLinks` without a setter and
  never populated it. Added `fetchPropertySemanticLinks` (schema-level, one
  call per schema via new `/semantic-links/entity-prefix/{type}/{prefix}`
  endpoint) that replaces the slice per schema so removed assignments no
  longer linger.
- RDF triples duplicated unboundedly on re-import. `_skolemize_bnode`
  embedded rdflib's random per-parse bnode id, so re-importing an ontology
  with blank nodes (OWL restrictions, SHACL shapes) minted brand-new URIs
  that never hit the `uq_rdf_triple` constraint. Now canonicalise the graph
  (RGDA1 `to_canonical_graph`) before persisting, so identical content
  produces identical rows and re-imports are true no-ops.
- Added threshold-gated bloat diagnostics to `RdfTriplesRepository`: when the
  table exceeds `RDF_TRIPLES_DIAGNOSTIC_THRESHOLD` (default 30000) rows, log a
  one-shot forensic snapshot (blank-node count, constraint-bypass check,
  source_type + context breakdown, per-(subject,predicate) churn) — needed
  because the customer's 64k-with-no-file case is not locally reproducible.
- `_register_sources_as_collections` was registering the dynamic recomputed
  contexts `urn:app-entities` / `urn:semantic-links` as KnowledgeCollections
  (spurious timestamped metadata, polluted Collections list). Added them to
  the skip set.
- `list_for_iri` returned explicit DB links + inferred graph links with no
  dedup ("assignment shown twice" report). Now deduped on `(entity_type,
  entity_id)`, explicit wins over the synthetic inferred link.
- Data domains stuck loading in Home Discovery when no domain named "Core"
  existed. Now falls back to first root domain, then first domain.
- Notifications polling kept Lakebase compute warm 24/7 because there was no
  tab-visibility handling. Store now pauses the interval on
  `visibilitychange` and resumes (with an immediate fetch) on visible.
- Made `vite.config.ts` proxy target env-configurable via `VITE_PROXY_TARGET`
  (dev-only ergonomics; defaults to `http://localhost:8000`).

Tests: added `test_rdf_bnode_dedup.py` (re-import idempotency, stable bnode
ids, threshold-gated diagnostic firing + throttle + no-op paths) and a
`list_for_iri` dedup case in `test_semantic_links_manager.py`.

Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9

* docs(dev): env-driven ports for parallel worktree dev servers

Make frontend/backend dev ports overridable via env vars so multiple
git worktrees can run isolated servers without colliding on 3000/8000:

- vite.config.ts: VITE_PORT (default 3000) drives server.port
- pyproject.toml: dev-backend honors BACKEND_PORT/BACKEND_HOST (default 8000/0.0.0.0)
- CONTRIBUTING.md: "Running Multiple Worktrees Side-by-Side" section
- .cursor/rules/08-testing-and-deployment.mdc + CLAUDE.md: LLM-agent guidance

VITE_PROXY_TARGET (already present) points a worktree's frontend at its
own backend. Worktrees share the local app_ontos DB; only ports differ.

Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9

* fix(semantic): repopulate caches so dashboard stops recomputing per request

Customer redeploy was still slow. Prod logs showed every request logging
"Persistent cache not found for {stats,taxonomies,concepts}, computing live"
against a 68,684-triple graph — the caches were destroyed and never rebuilt.

Two compounding defects:

1. on_models_changed() ran rebuild_graph_from_enabled() (which rebuilds BOTH
   the in-memory snapshots and the persistent JSON files) and then immediately
   called _invalidate_cache(), deleting exactly what it had just built. Any
   concept/link mutation wiped every cache tier.
2. The read paths (get_taxonomies / get_taxonomy_stats / get_grouped_concepts)
   recomputed live on a miss but never repopulated, so the miss recurred on
   every subsequent request until the next full rebuild.

Fix:
- Drop the redundant _invalidate_cache() in on_models_changed; the rebuild is
  already the authoritative clear-then-rebuild.
- Add _ensure_caches_warm(): on a cold read, recompute all three tiers from the
  current singleton graph via the existing atomic writer, populating memory +
  files. Read paths call it on miss and return the warmed value.
- File-cache-hit branches now also populate the in-memory snapshot to avoid
  re-parsing JSON every request.

Coherence: warming recomputes from self._graph, which every mutation keeps
fresh (full rebuild, or incremental graph edit + invalidate), so a warmed cache
is never staler than the graph. Verified single-worker deploy + singleton read
manager, and audited all 12 graph mutators satisfy the invalidate-or-rebuild
invariant — so this does not reintroduce the historical "changes don't show"
staleness.

Adds test_semantic_cache_warm.py covering both defects + invalidate→rewarm.

Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9

* fix(jobs): stop polling thread from keeping Lakebase warm when idle

The background job-polling thread opened a fresh DB session every
JOB_POLLING_INTERVAL_SECONDS (default 300s) and queried
workflow_installation_repo.get_all_installed unconditionally — even with
zero installed workflows (customer log: "Polling 0 installed workflows...").
On Databricks Apps that query every 5 minutes never lets Lakebase reach its
idle window, so compute stayed permanently warm (the reported "compute always
active"). This is server-side and independent of the client notification-poll
visibility fix.

Fix: cache an in-memory presence flag (_has_installations). Once a cycle
confirms zero installations, subsequent cycles skip the DB session entirely and
just wait the interval, letting Lakebase idle down. install_workflow() resets
the flag to None so a newly installed workflow is re-detected on the next cycle.
Authoritative because JobsManager is a single app-state singleton on a
single-worker deployment.

Adds test_jobs_polling_idle.py: skip-after-zero, None-forces-recheck,
non-empty-keeps-polling.

Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9

* test(semantic): update linked-objects-panel mock to match by-iri endpoint

The panel's fetch moved from /api/semantic-links/iri/{iri} to
/api/semantic-links/by-iri?iri= in this branch, but the test mock
still matched the old path, so the panel rendered the empty state.

Co-authored-by: Isaac

* perf(jobs): incremental poll window + change-gated writes + adaptive backoff

Even with workflows installed, the background job poll kept Lakebase warm and
busy: every cycle it re-fetched the last 7 days of runs per job and re-committed
every row unconditionally. A job scheduled every 10 min = ~1000 historical runs
re-written every 5 min, forever — so Lakebase never idled and did constant
redundant work.

Three changes so a quiet cycle performs zero Lakebase writes and a fully idle
system stops waking frequently:

- Incremental window: query runs since (last_polled_at - overlap) instead of a
  fixed 7-day lookback, capped by JOB_POLLING_BACKFILL_DAYS for cold-start /
  post-downtime catch-up. Steady state fetches a handful of recent runs, not the
  whole history.
- Change-gated writes: upsert_run skips its commit when the run row is unchanged;
  update_last_polled gains only_if_changed=True so an unchanged job state is not
  re-persisted. (list_runs still hits the Databricks control plane, not Lakebase.)
- Adaptive backoff: when a cycle sees no active (non-terminal) runs, the interval
  doubles toward a cap (4x base); any active run or state change snaps it back to
  base so live jobs are still tracked promptly.

Builds on the earlier skip-when-zero-installations fix. Net: an app with idle or
quiet workflows lets Lakebase scale down; an actively running job is still
tracked at the base cadence. Full scale-to-zero while a scheduled job runs is
inherently not possible without dropping proactive tracking — that trade-off
(activity-gating / push-based) is left as a follow-up.

Tests: upsert skip-unchanged/commit-on-change; update_last_polled
skip-unchanged/write-on-change; existing polling-idle tests still pass.

Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9

* fix(dev): revert broken BACKEND_PORT hatch-script indirection

The dev-backend hatch script used --port=${BACKEND_PORT:-8000}, but hatch
parses ${...}/{...} as its own template syntax and fails to launch with
"Unknown context field 'BACKEND_HOST'" — so BACKEND_PORT/BACKEND_HOST never
worked. Reverted the script to a hard-coded --port=8000.

For a secondary worktree, invoke uvicorn directly with the desired port;
updated CONTRIBUTING.md, .cursor/rules/08-testing-and-deployment.mdc, and
CLAUDE.md to show that command instead of the non-working BACKEND_PORT form.
Frontend VITE_PORT / VITE_PROXY_TARGET are real process.env and unaffected.

Session: claude -r e6659b77-708d-48af-8781-d2c8323f62b9

---------

Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
… request (#674)

Users could click "Profile with DQX" while the backing background job was
disabled, producing a generic error toast. The button now reflects the job's
state, explains why it is disabled, and can optionally notify admins.

Backend:
- Add ALLOW_JOB_ENABLEMENT_REQUESTS setting (persisted via app_settings,
  mirroring LLM_ENABLED) controlling whether users may ask admins to enable
  a background job.
- Add GET /api/jobs/capabilities: reports install state for an allowlist of
  feature-gating workflows. The existing job endpoints require the 'jobs'
  permission, which no non-admin default role has, so the features gated by
  those jobs could not read their state. Returns booleans only -- no job
  IDs, cluster IDs or run history.
- Add POST /api/jobs/workflows/{id}/request-enablement: sends a deduplicated
  ACTION_REQUIRED notification to the Admin role. Not gated by the 'jobs'
  permission by design -- the point is to let users who cannot manage jobs
  ask someone who can -- but guarded by the setting and the allowlist.
- Add NotificationsManager.has_unhandled_actionable_notification() so repeat
  clicks do not flood the admin bell. Fails open: a duplicate notification
  beats silently dropping a request.

Two bugs found while wiring this up, both in the profiling start path:
- start_profiling() committed the profiling-run record before checking that
  the workflow was installed, so every failed click left a dangling 'pending'
  run that the UI then polled indefinitely ("DQX profiling in progress..."
  with no job to advance it). The check now runs first.
- The route collapsed the sanitized ValueError into a generic 400 detail, so
  the actionable "job not enabled" message never reached the client. It now
  raises ConflictError (409) and re-raises HTTPException.

Frontend:
- Add job-capabilities store; assumes available until proven otherwise so a
  failed fetch cannot disable a working feature.
- Disable the button when the job is not installed, with the hint in a
  tooltip and the "Notify admin" action inside it, so the request does not
  compete visually with the primary schema actions.
- Treat a lingering pending/running record as stale when the job is not
  installed, fixing the false in-progress spinner.
- Add the admin toggle to Settings > General and translations for all 7
  locales.
* chore(deps): bump astral-sh/setup-uv from 8.2.0 to 8.3.2 (#612)

Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 8.2.0 to 8.3.2.
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@fac544c...11f9893)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 8.3.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/checkout from 6.0.3 to 7.0.0 (#559)

Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.3 to 7.0.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@df4cb1c...9c091bb)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump @radix-ui/react-popover in /src/frontend (#560)

Bumps [@radix-ui/react-popover](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/popover) from 1.1.15 to 1.1.19.
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/popover/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/popover)

---
updated-dependencies:
- dependency-name: "@radix-ui/react-popover"
  dependency-version: 1.1.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump pydantic-settings from 2.14.1 to 2.14.2 in /src (#565)

* chore(deps): bump pydantic-settings from 2.14.1 to 2.14.2 in /src

Bumps [pydantic-settings](https://github.com/pydantic/pydantic-settings) from 2.14.1 to 2.14.2.
- [Release notes](https://github.com/pydantic/pydantic-settings/releases)
- [Commits](pydantic/pydantic-settings@v2.14.1...v2.14.2)

---
updated-dependencies:
- dependency-name: pydantic-settings
  dependency-version: 2.14.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: regenerate requirements lockfiles

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* chore(deps): bump reportlab from 4.4.10 to 5.0.0 in /src (#566)

* chore(deps): bump reportlab from 4.4.10 to 5.0.0 in /src

Bumps [reportlab](https://www.reportlab.com/) from 4.4.10 to 5.0.0.

---
updated-dependencies:
- dependency-name: reportlab
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: regenerate requirements lockfiles

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* ci: back-merge with a merge commit so the sync stops replaying

The back-merge PR (main -> development) was squash-merged, which is the
only method the development ruleset allows. Squashing collapses all of
main's commits into one new commit that lives only on development and
carries none of main's SHAs, so the merge-base of main and development
never advances. The ancestor check (git merge-base --is-ancestor
origin/main origin/development) therefore stays false and every push to
main regenerates a PR replaying main's entire history onto development.

Make the workflow push a real merge commit (main as a second parent)
straight to development. A merge commit advances the merge-base, so the
next run's ancestor check short-circuits and nothing is reopened -- the
sync becomes idempotent.

Direct pushes to development bypass the PR flow, so the pushing token
must be in the ruleset bypass_actors; supply it as BACKMERGE_TOKEN. When
that secret is absent, or the merge conflicts, fall back to opening a PR
for manual (merge-commit) resolution instead of force-pushing.

Co-authored-by: Isaac

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Will Yuponce <will.yuponce+data@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants