Skip to content

fix(file): pin file task concurrency to 1 - #118

Open
Mahesh Kamble (ma-gk) wants to merge 2 commits into
mainfrom
fix/file-task-concurrency-1
Open

fix(file): pin file task concurrency to 1#118
Mahesh Kamble (ma-gk) wants to merge 2 commits into
mainfrom
fix/file-task-concurrency-1

Conversation

@ma-gk

@ma-gk Mahesh Kamble (ma-gk) commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

The file task's read path expands a glob and streams every matched file itself — it isn't driven by an input channel the way a normal competing-consumer task is. Running it with task_concurrency > 1 as a source just re-reads the same glob N times (duplicate records), with no benefit. Writes are the opposite: they're driven one record per worker off the input channel, so they're safe to fan out and should keep using the configured task_concurrency.

This PR pins the file task to a single worker only when it's running as a source (no input channel), while leaving the write path's concurrency untouched.

Changes

  • internal/pkg/pipeline/task/task.go — added GetType() string to the Task interface, backed by Base.Type (mirrors the existing GetName() / Base.Name pattern). Needed so the pipeline can identify a file task by its type:, not by its arbitrary user-chosen name:.
  • internal/pkg/pipeline/pipeline.go (runTaskConcurrently) — caps concurrency to 1 when t.GetType() == "file" && input == nil (i.e. the task is a source), before taskWg.Add(concurrency) is called, so the wait-group count always matches the goroutines actually spawned.
  • internal/pkg/pipeline/task/file/file.go — removed the now-redundant GetTaskConcurrency() override; the file task falls back to Base.GetTaskConcurrency() for the write path, and the read-side cap lives solely in pipeline.go.
  • test/pipelines/file_concurrency_test.yaml (new) — fixture exercising both directions: 4 read workers against a 2-file glob, split into per-line records, then 8 write workers each writing a unique {{ macro "uuid" }}.txt file.

Why not change file's own concurrency method instead?

An earlier version of this fix overrode file.GetTaskConcurrency() directly, forcing 1 in all cases. That breaks the write path, which legitimately wants to fan out across task_concurrency workers. The fix needed to know which direction a given file task instance is running in, and that's only known at the pipeline level (via the input channel), not inside the task itself at the point GetTaskConcurrency() would be called.

Test plan

  • go build ./... — clean.
  • go test ./internal/pkg/pipeline/... — all pass.
  • go vet ./... — only pre-existing, unrelated copylocks warnings (present before this change, in compress.go, archive.go, and an existing line in file.go).
  • Manually ran test/pipelines/file_concurrency_test.yaml end to end:
    • Read task (task_concurrency: 4) ran with exactly 1 worker — no duplicate records, no errors.
    • Write task (task_concurrency: 8) ran with all 8 workers as configured.
    • Output file count: 2123, exactly matching the true line count of names.txt (1998) + birds.txt (123), corrected for both fixture files lacking a trailing newline (wc -l undercounts by 1 each → 1999 + 124 = 2123).

The file task is not safe to run with multiple concurrent workers, so
override GetTaskConcurrency to always return 1, warning if a higher
task_concurrency was configured. Mirrors the existing pattern used by
the http/server and aws/parameter_store tasks.

Adds a pipeline YAML fixture exercising task_concurrency on file
read/write tasks.
@ma-gk
Mahesh Kamble (ma-gk) requested a review from a team as a code owner September 8, 2026 10:11
Comment thread internal/pkg/pipeline/task/file/file.go Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants