Skip to content

serverless-migration: failure reporting flow + Genie Code install docs#72

Open
rrf-DB wants to merge 1 commit into
databricks:mainfrom
rrf-DB:roland/migration-skill-failure-issue-and-genie
Open

serverless-migration: failure reporting flow + Genie Code install docs#72
rrf-DB wants to merge 1 commit into
databricks:mainfrom
rrf-DB:roland/migration-skill-failure-issue-and-genie

Conversation

@rrf-DB
Copy link
Copy Markdown
Contributor

@rrf-DB rrf-DB commented May 12, 2026

Summary

Two improvements to the databricks-serverless-migration skill, plus the repo-level glue they need.

1. Auto-file GitHub issue when migration can't complete. The Failure Reporting Protocol now produces a pre-filled issues/new URL (URL-encoded title + body) and an optional gh issue create command. The agent presents both alongside the local report path and never auto-submits. The MUST-NOT-CONTAIN list is extended (codenames, Slack IDs, go/ links, internal ticket IDs, gdoc IDs) and a defensive redaction checklist runs before write.

2. Document Genie Code Agent-mode install. A new reference doc covers the three install methods (notebook installer, shell installer, app-based workspace-wide deploy), the per-user vs workspace-wide paths, and the important serverless-no-CLI caveat for this specific skill. SKILL.md now surfaces the option at the top and instructs the agent to mention it after each migration analysis.

Changes

Skill (skills/databricks-serverless-migration/)

  • SKILL.md — extended Failure Reporting Protocol (simpler decision tree: offer for ANY unmigratable case, required output template with pre-filled URL + gh command); new "Where to Run This Skill" section; reference links updated
  • references/failure-reporting.md — new: redaction checklist, URL-encoding recipe, worked example, gh CLI alternative
  • references/install-in-databricks-genie-code.md — new: install methods, paths, CLI caveat, follow-up Slack channels

Repo root

  • .github/ISSUE_TEMPLATE/migration-feedback.md — new: matches the schema the skill emits
  • README.md — list databricks-serverless-migration under Available Skills
  • manifest.json — adds the two new reference files; top-level updated_at refreshed

Test plan

Validated by the local migration test bench: 16/16 cases pass (10 existing + 6 new purpose-built unmigratable workloads exercising hbase / custom-jar / py4j / sparkr / cassandra / unknown-pattern paths). Failure-report score 1.0 on every unmigratable case; zero PII leakage in the offered issue bodies.

  • 6 new test workloads: synthetic-hbase-connector, synthetic-custom-jar-datasource, synthetic-py4j-jvm-reflection, synthetic-r-sparkr, synthetic-cassandra-connector, synthetic-unknown-future-api. Each expects should_trigger: true with a specific expected_error_category.
  • Evaluator extended to score: referenced_report_path, offered_prefilled_url, offered_gh_cli, expected_error_category_mentioned, and a hard-fail check for PII leakage in the offered body (scoped to the offered text, not analysis chatter).
  • 10/10 pre-existing cases still pass after the SKILL.md changes — zero regressions on baseline detection.

Reviewer checklist

  • python3 scripts/skills.py validate passes cleanly
  • Issue template renders correctly when opened via the pre-filled URL (the template=migration-feedback.md parameter resolves once this PR merges)
  • Failure-reporting flow reads sensibly end-to-end

Notes

  • This is a docs + protocol change. No detection-logic regressions expected.
  • The skill's compatibility: Requires databricks CLI declaration is unchanged. The new Genie Code install doc explicitly flags that this is a known limitation when running inside a serverless Databricks workspace and offers three workarounds.

This pull request and its description were written by Isaac.

Two improvements driven by field gaps:

1. Auto-file GitHub issue on unmigratable workloads. The Failure Reporting
   Protocol now generates a pre-filled issues/new URL (URL-encoded title +
   body) and an optional gh issue create command alongside the local report
   path. Decision tree simplified: offer to file for ANY unmigratable case,
   not just unknown patterns. MUST-NOT-CONTAIN list expanded to cover
   internal codenames, Slack IDs, go/ links, PROD-* tickets, gdoc IDs.
   New defensive redaction checklist runs before write.

2. Document Genie Code Agent-mode install. New "Where to Run This Skill"
   section in SKILL.md and references/install-in-databricks-genie-code.md
   covering all three install methods (notebook installer, shell installer,
   workspace-wide Databricks App), per-user vs workspace-wide paths, and
   the serverless-no-CLI caveat specific to this skill.

Validated by the local migration test bench: 16/16 cases pass (10 existing
+ 6 new purpose-built unmigratable workloads exercising hbase / custom-jar /
py4j / sparkr / cassandra / unknown-pattern paths). Failure-report score 1.0
on every unmigratable case; zero PII leakage in the offered issue bodies.

Co-authored-by: Isaac
@rrf-DB rrf-DB requested review from a team, lennartkats-db and simonfaltum as code owners May 12, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant