in_tail: Implement budget for max number of opening files

[cosmo0920]

Implemented a process-wide shared Tail pool. max_open_files: 1024 now caps all Tail inputs combined at 1,024 monitored files, regardless of worker count.

• Uses CFL atomic reservations.
• Warns at 75% shared usage.
• Omitted/zero values inherit the pool’s limit; conflicting positive values fail startup.
• Preserves deferred-file offsets.

Build and both Tail runtime targets passed. All 17 integration cases passed normally and under strict Valgrind, using:

tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/in_tail -q -k 'max_open_files or rotation or ignore_active_older'
VALGRIND=1 VALGRIND_STRICT=1 tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/in_tail -q -k 'max_open_files or rotation or ignore_active_older'

Database and watcher handles remain outside this budget.

Also related to #12432.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Added a configurable max_open_files limit for Tail inputs.
  • Tail inputs now share an open-file budget, defer excess files, retry when capacity becomes available, and warn at 75% usage.
  • File budget settings support unlimited operation and consistent limits across multiple Tail inputs.

• Bug Fixes

  • Improved preservation and cleanup of file state during deferred processing and restarts.

• Documentation

  • Documented open-file budget configuration, behavior, warnings, and shared limits.

• Tests

  • Added integration and runtime coverage for limits, sharing, retries, validation, and concurrency.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: b81ed389-5765-4e0d-b251-9498672b45bd

📥 Commits

Reviewing files that changed from the base of the PR and between 64d2dd8 and 60740c0.

📒 Files selected for processing (14)

• plugins/in_tail/CMakeLists.txt
• plugins/in_tail/tail.c
• plugins/in_tail/tail_config.c
• plugins/in_tail/tail_config.h
• plugins/in_tail/tail_file.c
• plugins/in_tail/tail_file_budget.c
• plugins/in_tail/tail_file_budget.h
• plugins/in_tail/tail_scan_glob.c
• plugins/in_tail/tail_scan_win32.c
• tests/integration/scenarios/in_tail/README.md
• tests/integration/scenarios/in_tail/tests/test_in_tail_001.py
• tests/integration/scenarios/in_tail/tests/test_in_tail_max_open_files.py
• tests/runtime/CMakeLists.txt
• tests/runtime/in_tail_file_budget.c

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.

---

📝 Walkthrough

Walkthrough

The Tail input plugin adds a shared max_open_files budget. It validates and shares limits across inputs, defers files at capacity, releases slots during cleanup, preserves deferred state, and adds integration and runtime tests.

Changes

Tail file-budget implementation

Layer / File(s)	Summary
Budget contract and lifecycle plugins/in_tail/tail_file_budget.*, plugins/in_tail/tail_config.*, plugins/in_tail/CMakeLists.txt	The plugin adds shared budget creation, limit resolution, atomic reservation and release, warning thresholds, reference counting, configuration validation, and build integration.
File reservation and deferred cleanup plugins/in_tail/tail.c, plugins/in_tail/tail_file.c, plugins/in_tail/tail_scan_*.c	File opens reserve slots and release them on failures or removals. Deferred files retain database state. Ignored-state cleanup occurs after successful appends.
Integration and runtime validation tests/integration/scenarios/in_tail/*, tests/runtime/*	Tests and documentation cover deferred files, warnings, invalid and conflicting limits, threaded sharing, persistence, cleanup, and concurrent reservations.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant TailInput
  participant flb_tail_file_budget
  participant tail_file_open
  participant TailScanner
  TailInput->>flb_tail_file_budget: reserve file slot
  alt slot available
    flb_tail_file_budget-->>TailInput: reservation granted
    TailInput->>tail_file_open: open file
    tail_file_open-->>TailInput: file descriptor or failure
    TailInput->>flb_tail_file_budget: release on failure or removal
  else budget full
    flb_tail_file_budget-->>TailInput: defer file
    TailScanner->>TailInput: retry on refresh scan
  end

Loading

Suggested reviewers: edsiper

Merge Risk: ⚪ Minimal · up to 60740

The shared file budget preserves deferred file offsets during startup and releases reservations through the relevant cleanup paths. No actionable merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 11.54% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 11 files. (3 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: adding an open-file budget for the in_tail plugin. It is concise and related to the pull request objectives.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 11.54% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 11 files. (3 skipped: 3 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-18T09:39:49.606711Z	60740c0	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60740c0cdd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".