fix(file): eliminate duplicate reads under task_concurrency

[Mahesh Kamble (ma-gk)]

Summary

• Fixes a correctness bug in the file task's read mode: when task_concurrency > 1, every worker independently expanded the glob and read every matched file, so each file was emitted once per worker instead of once total.
• The glob is now expanded exactly once (sync.Once), and workers claim disjoint files off the shared matched-path list via an atomic index (readIdx), so N workers split the file list N ways instead of each reading every file.
• writeFile now builds the per-worker file struct explicitly (only the fields the writer needs) instead of struct-copying f, since f now carries sync.Once/atomic read-concurrency state (and the embedded task.Base mutex) that must never be struct-copied.
• Updated the task_concurrency doc row in the task README to describe the read/write semantics precisely.

Test plan

Added test/pipelines/file_concurrency_test.yaml, which exercises both directions in one pipeline:

• Read: task_concurrency: 4 against a glob matching 2 fixture files (names.txt, birds.txt) — before the fix, 4 workers × 2 files = 8 read records; after the fix, exactly 2.
• Write: task_concurrency: 8 writing one file per line after a split — output file count must exactly equal input line count (no drops, no duplicates).

Ran locally:

rm -rf /tmp/caterpillar/file_concurrency_test
go run cmd/caterpillar/caterpillar.go -conf test/pipelines/file_concurrency_test.yaml
ls /tmp/caterpillar/file_concurrency_test | wc -l   # 2123
cat test/pipelines/names.txt test/pipelines/birds.txt | wc -l   # 2121 newlines + 2 (no trailing newline in either fixture) = 2123

Output count matches expected exactly.

🤖 Generated with Claude Code