Cache latest-DAG lookups in clear_task_instances - #71769
Open
joaopamaral wants to merge 2 commits into
Open
Conversation
clear_task_instances calls scheduler_dagbag.get_latest_version_of_dag() and DagVersion.get_latest_version() once per task instance when clearing on the latest version. get_latest_version_of_dag deserializes the entire DAG on every call, so clearing N task instances costs N full-DAG deserializations. On a production DAG with ~8,400 tasks this measures at ~1.6s per cleared task instance: clearing one task together with its ~50 downstream tasks takes ~85s, while the identical clear with run_on_latest_version=False takes ~5s. Users read the hung request as "clear did nothing" and retry, producing overlapping clear requests that then fail with conflicts. The UI pre-selects "run on latest version" whenever the task instance's dag version differs from the latest, so routine clears hit this path. Cache both lookups per dag_id for the duration of the call. Besides removing the O(n) cost (measured ~85s -> ~4.4s for the 50-task clear), this makes the clear consistent if a new version is serialized while the request is running: previously task instances in the same clear could be pinned to different versions. Same shape as apache#14048, which batched this function's per-row TaskReschedule deletes.
joaopamaral
force-pushed
the
fix-per-ti-latest-dag-lookup-in-clear
branch
from
August 18, 2026 14:47
2634fd0 to
c938cd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
clear_task_instancescallsscheduler_dagbag.get_latest_version_of_dag()andDagVersion.get_latest_version()once per task instance when clearing on the latest version.get_latest_version_of_dagdeserializes the entire DAG on every call, so clearing N task instances costs N full-DAG deserializations.On a production DAG with ~8,400 tasks this measures at ~1.6s per cleared task instance:
run_on_latest_version(same DAG, same run, same task, cleared via
POST /api/v2/dags/{dag_id}/clearTaskInstances; the 1-TI cases show the cost is per-TI, not per-request)The UI pre-selects "run on latest version" whenever the task instance's dag version differs from the latest, so routine clears hit this path. Users read the hung request as "clear did nothing" and retry, producing overlapping clear requests that then fail with conflicts (the loser hits the
task_instance_historyunique constraint → opaque 409).Fix
Cache both lookups per
dag_idfor the duration of the call (function-local dicts, nothing outlives the request).Measured on the same deployment after the fix: 84.7s → 4.4s for the 50-TI clear;
run_on_latest_version=Trueis now indistinguishable fromFalse.Side benefit: the clear is now version-consistent — previously a new version serialized mid-request could pin task instances of the same clear to different versions.
Same shape as #14048, which batched this function's per-row TaskReschedule deletes.
Test
Added
test_clear_task_instances_caches_latest_dag_lookups: clears 4 TIs withrun_on_latest_version=Trueand asserts each lookup happens twice total (once for the TI loop, once for the dag-run update) instead of once per TI, and that all TIs land on the latest version.^ Add meaningful description above
Read the Pull Request Guidelines for more information.