fix: collect and manage generated files from job executions#50
fix: collect and manage generated files from job executions#50aron-muon merged 3 commits intoaron-muon:mainfrom
Conversation
* Add functionality to detect and download generated files from Job pods before cleanup. * Introduce method in JobExecutor for file retrieval. * Update ExecutionResult model to include attribute. * Implement method in ExecutionServiceInterface for accessing pre-downloaded file content. * Enhance orchestrator to utilize pre-downloaded content when container is unavailable. * Add unit tests for new functionality and ensure proper handling of code files.
|
@aron-muon , here's a little development to greatly ensure that the generated \ changed files are returned. |
There was a problem hiding this comment.
Pull request overview
This PR improves reliability of returning files generated during code execution (especially for Kubernetes Job-based executions where pods are cleaned up immediately), by collecting generated files earlier and adding a runner/orchestrator fallback path for retrieving those files after pod deletion.
Changes:
- Always runs generated-file detection after execution (removes keyword gating) and expands code-source file filtering across languages.
- For Job-based executions, downloads generated files before pod cleanup and caches their bytes for later retrieval.
- Adds
pop_job_file_contentplumbing plus new unit tests covering Job fallback behavior and expanded code-file skipping.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/services/kubernetes/job_executor.py |
Collects & downloads generated files during Job execution before cleanup. |
src/services/kubernetes/models.py |
Extends ExecutionResult to optionally carry Job-generated file metadata/content. |
src/services/execution/runner.py |
Centralizes code-filename skip set; caches Job file bytes and exposes pop_job_file_content. |
src/services/interfaces.py |
Adds a default pop_job_file_content hook to the execution service interface. |
src/services/orchestrator.py |
Falls back to runner’s pre-downloaded Job file bytes when the pod/container is missing. |
tests/unit/test_execution_runner.py |
Expands generated-file detection tests and adds tests for pop_job_file_content. |
tests/unit/test_orchestrator.py |
Adds coverage for orchestrator Job-file fallback when no container exists. |
Comments suppressed due to low confidence (1)
src/services/kubernetes/job_executor.py:551
- Generated-file collection for Job executions is currently gated on
result.exit_code == 0. Pool-based executions attempt generated-file detection regardless of exit code, so this can reintroduce missing outputs when code writes files and then fails. Consider collecting generated files wheneverjob.runner_urlis available (possibly with additional safeguards) rather than only on success.
)
# Execute code
result = await self.execute(
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
* Change job file content storage to use a tuple of (session_id, path) for unique identification. * Update related methods to accommodate session-based file content retrieval. * Enhance unit tests to verify session-specific behavior for job file content management.
aron-muon
left a comment
There was a problem hiding this comment.
Thank you for the contribution!
|
🎉 This PR is included in version 3.2.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
TL;DR
Fix generated files not always being returned to LibreChat after code execution.
Root causes & fixes:
savefig,open(, etc. in the code; now always runs after execution.Job-path files collected before pod cleanup — cold-path languages (Go, Rust, Java, …) returned
handle=Noneso files were never fetched before the pod was destroyed; files are now downloaded eagerly insideexecute_with_job.Code file filter expanded —
main.goandmain.rswere not excluded from the generated-files list; all 12 language source filenames are now filtered.This pull request enhances the handling of files generated during code execution, especially for Kubernetes Job-based executions where the pod is destroyed after completion. The main improvements ensure that generated files are detected, downloaded, and made available even after the execution pod is gone. The changes also expand and harden the logic for skipping language source files and improve test coverage for these scenarios.
File handling improvements for Job-based executions:
src/services/kubernetes/job_executor.py,src/services/kubernetes/models.py,src/services/execution/runner.py). [1] [2] [3] [4] [5]pop_job_file_content) to retrieve pre-downloaded file content for Job executions, allowing orchestrator services to access files after the pod is destroyed (src/services/execution/runner.py,src/services/interfaces.py,src/services/orchestrator.py). [1] [2] [3] [4]src/services/orchestrator.py). [1] [2]Source file detection and filtering:
src/services/execution/runner.py,src/services/kubernetes/job_executor.py). [1] [2] [3]Test enhancements:
pop_job_file_contentlogic and orchestrator fallback behavior (tests/unit/test_execution_runner.py,tests/unit/test_orchestrator.py). [1] [2] [3] [4] [5]These changes make file handling more robust for both interactive and batch code execution scenarios, ensuring users can reliably access generated outputs regardless of the execution backend.