air: add convert-to-dabs (run YAML -> Databricks Asset Bundle) - #6111
Merged
Conversation
Collaborator
Integration test reportCommit: 16219ba
9 interesting tests: 4 RECOVERED, 4 SKIP, 1 flaky
Top 3 slowest tests (at least 2 minutes):
|
vinchenzo-db
force-pushed
the
air-convert-to-dabs-main
branch
10 times, most recently
from
August 3, 2026 22:03
e74034f to
29248d4
Compare
Adds `air convert-to-dabs`, which translates an AIR CLI run YAML into a
deployable Databricks Asset Bundle so a workload authored for `air run` can be
managed and deployed with the standard DABs workflow (validate/deploy/run).
The emitted bundle is schema-valid: the ai_runtime_task maps to the SDK
jobs.AiRuntimeTask (experiment + deployments[].{command_path,compute} +
code_source_path), with framework fields (retries, timeout, budget policy) on
the surrounding task and the runtime environment in environments[].
Snapshotting is owned by the deploy-time aicode mutator, not by convert:
code_source_path points at a local *directory* staged inside the bundle, and
`bundle deploy` (aicode.PackageAndUpload) packages it into a content-addressed
tarball and uploads it. convert only lays down the source bytes — copying the
working tree (honoring .gitignore) or materializing a pinned git commit into the
directory. requirements.yaml is likewise not emitted: aicode.SynthesizeRequirements
regenerates it from the environments[] spec, so convert folds the whole dependency
set (inline or requirements-file) into that spec instead.
env_variables / secrets / parameters have no native ai_runtime_task field, so they
ride as env_vars.json / secret_env_vars.json / hyperparameters.yaml sidecars (same
as `air run`), and a "Notes:" section tells a migrating user what was transformed
or staged out-of-band.
This is the top of a 2-PR stack: it builds on the aicode deploy-time packaging
mutator so the two compose end-to-end.
Co-authored-by: Isaac
The Windows CI job failed the convert-to-dabs acceptance test with "tar (child): Cannot connect to C: resolve failed": the system tar reads the `C:` in an absolute archive path as a remote host:path. Two fixes: - createPlainTarball now passes the archive as a bare basename with cmd.Dir set to the output directory (and an absolute parent), so no `C:\...` path reaches tar's -f argument. Mirrors how git archive is invoked; safe on GNU tar and bsdtar. (This helper is shared with the `air run` snapshot path.) - extractTarball is rewritten in pure Go (archive/tar + compress/gzip) instead of shelling out to `tar -xzf`, eliminating the same drive-letter hazard on the extract side and dropping the external-tar dependency for extraction. It rejects entries that would escape the destination (path traversal, absolute/escaping symlinks) and bounds each file copy to its header size. Adds unit tests for extractTarball (happy path incl. nested dirs + in-tree symlink; traversal + escaping-symlink rejection). Co-authored-by: Isaac
convert-to-dabs is now a purely local, syntactic translation. It no longer copies, snapshots, or git-archives the code source — the deploy-time aicode mutator (from the base PR) packages the source in place at `bundle deploy`. - code_source_path is emitted as the source directory relative to the bundle (bundle root defaults to the input YAML's directory, which contains it), rather than a copied ./code_source dir. - Removed the materialize/extract-tarball machinery and the git-pinned-commit path. code_source.snapshot.git is now rejected (deploy packages the working tree); a code_source outside the bundle directory is rejected with guidance. - writeBundle now only writes databricks.yml + command.sh + the env/secret/param sidecars. Dependencies stay folded into the environments[] spec. Tests updated: assert code_source_path points at ./src with no copy, reject git-pin and out-of-bundle sources; the acceptance test converts in place. Co-authored-by: Isaac
git and remote_volume stay unsupported, but the errors now explain why and what to do: for git, check out the revision before converting (deploy packages the working tree as-is); for remote_volume, set workspace.artifact_path (the bundle artifact location is bundle-wide, not per-code-source). Co-authored-by: Isaac
Re-running a conversion previously failed with no way to overwrite, and the suggested remedy (--output-dir) is a dead end in place: code_source must live inside the bundle dir. Add --force, and point the error at it. The next-steps message said "databricks bundle ...". A CLI without ai_runtime_task support only warns on the unknown field and then deploys a job with no AI task, so print the invoked binary instead and call out the risk. Co-authored-by: Isaac
The bundle is written next to the input YAML, so `cd .` was a no-op step; it is now printed only when the bundle lands elsewhere. Drop the notes block and the stale-CLI warning, and point at `bundle summary` for what was deployed. Co-authored-by: Isaac
vinchenzo-db
force-pushed
the
air-convert-to-dabs-main
branch
from
August 4, 2026 22:44
7a2b086 to
c7698c9
Compare
include_paths narrows the archive to a subset of root_path, which a bundle can't
express per code source — deploy packages the whole directory. Converting was
silently dropping it and uploading files the user meant to exclude, so reject it
and point at sync.exclude / .gitignore.
Emit a job resource key that YAML would type as a non-string scalar ("12345" ->
!!int) as a quoted key instead of prefixing it with "job_". yamlsaver already had
this logic for scalar *values* (isScalarValueInString); apply it to map keys too,
so the resource key keeps the experiment name.
Co-authored-by: Isaac
ben-hansen-db
approved these changes
Aug 5, 2026
ben-hansen-db
left a comment
ben-hansen-db
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the updates!
vinchenzo-db
enabled auto-merge
August 5, 2026 16:54
Keep the change inside experimental/air/ instead of teaching the shared
yamlsaver to quote map keys. quoteJobKey rewrites the emitted job resource key
when the experiment name would otherwise load as a non-string scalar ("12345" ->
!!int), which the bundle loader rejects.
NewSaverWithStyle can't do this: its style map applies to a key's whole subtree,
so quoting the job key also quotes every nested key and turns
accelerator_count: 1 into a string.
Co-authored-by: Isaac
It is a verbatim copy of the input YAML, which bundle sync already uploads, and nothing in the emitted bundle references it. Co-authored-by: Isaac
vinchenzo-db
disabled auto-merge
August 5, 2026 23:48
Write command.sh and the env/secret/param sidecars into generated_artifacts/ and emit sync.paths listing only that directory. The code directory is no longer synced as loose files: deploy still packages it into the snapshot tarball, so it was previously uploaded twice, which is costly for a large source tree. The server derives the sidecar paths from command_path's parent, so they move with command.sh. Co-authored-by: Isaac
The old message suggested --output-dir, which cannot help here: root_path is resolved relative to the input YAML, so redirecting the output moves the bundle away from the code rather than enclosing it. sync.paths is the mechanism that actually widens the sync root. Co-authored-by: Isaac
ben-hansen-db
left a comment
ben-hansen-db
left a comment
Contributor
There was a problem hiding this comment.
- Is this still true? Is experiment-bundle created?
--output-dir string Directory to write the bundle into (default: a
<experiment>-bundle folder next to the input YAML)
-
Do we do a translation for max_retries? One thing to note is that our default is three so we could set 3 retries as the default in the translation.
-
Error: code_source root_path "./src" is not inside the bundle directory
"bundleout"; ... Move the code inside the bundle, or set sync.paths in
databricks.yml to a directory that contains both (note that widens what
gets synced). Wouldn't bundle validate reject the sync paths change?
air run fills its own default of 3 when max_retries is unset, so omitting it gave the converted bundle the Jobs default instead — the same YAML retried differently depending on which path launched it. The --output-dir help still described a <experiment>-bundle folder; the default is the input YAML's directory. Co-authored-by: Isaac
Reverts dropping it. The Jobs run-output page derives its workspace path from command_path (same directory, fixed filename) to show the config a run used, so omitting the file leaves that link pointing at nothing. The input YAML cannot serve instead: `air run -f` accepts any path, and real configs are named train.yaml, workload.yaml, train_kie.yaml and so on, so the UI has no filename to derive. The duplicate copy at a fixed name is what makes the path resolvable. Co-authored-by: Isaac
Co-authored-by: Isaac
…oved) The main merge (AIR CLI Migration Pt. 2, #6166) stopped emitting a requirements.yaml sidecar — file-form deps now fold into the environments[] spec — and deleted the requirementsName constant. convert_to_dabs still filtered artifacts by it, breaking the build (undefined: requirementsName), which in turn failed lint, validate-generated, and every acceptance test. Drop the now-dead filter: buildArtifacts no longer produces requirements.yaml, so there is nothing to remove. Also update the usage_policy_id test to a UUID (the merge added UUID validation) and regenerate the engine-matrix snapshot. Co-authored-by: Isaac
ben-hansen-db
approved these changes
Aug 7, 2026
Collaborator
Integration test reportCommit: 36e6bce
18 interesting tests: 6 FAIL, 5 RECOVERED, 4 flaky, 2 SKIP, 1 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
air convert-to-dabs, which translates an AIR CLI run YAML into a deployable Databricks Asset Bundle so a workload authored forair runcan be managed and deployed with the standard DABs workflow (validate/deploy/run).The emitted bundle is schema-valid: the ai_runtime_task maps to the SDK jobs.AiRuntimeTask (experiment + deployments[].{command_path,compute} + code_source_path), with framework fields (retries, timeout, budget policy) on the surrounding task and the runtime environment in environments[].
Snapshotting is owned by the deploy-time aicode mutator, not by convert: code_source_path points at a local directory staged inside the bundle, and
bundle deploy(aicode.PackageAndUpload) packages it into a content-addressed tarball and uploads it. convert only lays down the source bytes — copying the working tree (honoring .gitignore) or materializing a pinned git commit into the directory. requirements.yaml is likewise not emitted: aicode.SynthesizeRequirements regenerates it from the environments[] spec, so convert folds the whole dependency set (inline or requirements-file) into that spec instead.env_variables / secrets / parameters have no native ai_runtime_task field, so they ride as env_vars.json / secret_env_vars.json / hyperparameters.yaml sidecars (same as
air run), and a "Notes:" section tells a migrating user what was transformed or staged out-of-band.This is the top of a 2-PR stack: it builds on the aicode deploy-time packaging mutator so the two compose end-to-end.
Co-authored-by: Isaac
Changes
Why
Tests