fix(file): pin file task concurrency to 1 - #118
Open
Mahesh Kamble (ma-gk) wants to merge 2 commits into
Open
Conversation
The file task is not safe to run with multiple concurrent workers, so override GetTaskConcurrency to always return 1, warning if a higher task_concurrency was configured. Mirrors the existing pattern used by the http/server and aws/parameter_store tasks. Adds a pipeline YAML fixture exercising task_concurrency on file read/write tasks.
snehalahire-pattern
previously approved these changes
Sep 8, 2026
Divyanshu Tiwari (divyanshu-tiwari)
requested changes
Sep 8, 2026
Mahesh Kamble (ma-gk)
requested review from
Divyanshu Tiwari (divyanshu-tiwari) and
snehalahire-pattern
September 8, 2026 13:48
Divyanshu Tiwari (divyanshu-tiwari)
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
filetask's read path expands a glob and streams every matched file itself — it isn't driven by an input channel the way a normal competing-consumer task is. Running it withtask_concurrency > 1as a source just re-reads the same glob N times (duplicate records), with no benefit. Writes are the opposite: they're driven one record per worker off the input channel, so they're safe to fan out and should keep using the configuredtask_concurrency.This PR pins the
filetask to a single worker only when it's running as a source (no input channel), while leaving the write path's concurrency untouched.Changes
internal/pkg/pipeline/task/task.go— addedGetType() stringto theTaskinterface, backed byBase.Type(mirrors the existingGetName()/Base.Namepattern). Needed so the pipeline can identify afiletask by itstype:, not by its arbitrary user-chosenname:.internal/pkg/pipeline/pipeline.go(runTaskConcurrently) — capsconcurrencyto1whent.GetType() == "file" && input == nil(i.e. the task is a source), beforetaskWg.Add(concurrency)is called, so the wait-group count always matches the goroutines actually spawned.internal/pkg/pipeline/task/file/file.go— removed the now-redundantGetTaskConcurrency()override; the file task falls back toBase.GetTaskConcurrency()for the write path, and the read-side cap lives solely inpipeline.go.test/pipelines/file_concurrency_test.yaml(new) — fixture exercising both directions: 4 read workers against a 2-file glob, split into per-line records, then 8 write workers each writing a unique{{ macro "uuid" }}.txtfile.Why not change
file's own concurrency method instead?An earlier version of this fix overrode
file.GetTaskConcurrency()directly, forcing 1 in all cases. That breaks the write path, which legitimately wants to fan out acrosstask_concurrencyworkers. The fix needed to know which direction a givenfiletask instance is running in, and that's only known at the pipeline level (via theinputchannel), not inside the task itself at the pointGetTaskConcurrency()would be called.Test plan
go build ./...— clean.go test ./internal/pkg/pipeline/...— all pass.go vet ./...— only pre-existing, unrelatedcopylockswarnings (present before this change, incompress.go,archive.go, and an existing line infile.go).test/pipelines/file_concurrency_test.yamlend to end:task_concurrency: 4) ran with exactly 1 worker — no duplicate records, no errors.task_concurrency: 8) ran with all 8 workers as configured.names.txt(1998) +birds.txt(123), corrected for both fixture files lacking a trailing newline (wc -lundercounts by 1 each → 1999 + 124 = 2123).