Vector `file` source preserves BOM as U+FEFF char and CRLF trailing CR in EVERY line

[28515graj]

Vector version: 0.58.0

Severity: Major (silent data corruption in every line, not just header)

Description:

Vector file source uses line_delimiter (default " ") but does not strip UTF-8 BOM or trailing CR from any line. The bug affects EVERY line in the file, not just the first.

Source code (confirmed):
lib/file-source/src/file_watcher/mod.rs:247:

match read_until_with_max_size(
    reader.as_mut(),
    file_position,
    self.line_delimiter.as_ref(),
    &mut self.buf,
    self.max_line_bytes,
).await

The line_delimiter is passed verbatim. No BOM stripping. No CR stripping. The full bytes between line boundaries (or file boundary, for the first line) become the raw_line.

Reproduction (Vector 0.58.0, file with BOM + 5 CRLF lines):

File bytes:

EF BB BF line0_col1<TAB>line0_col2<CR><LF>
          line1_col1<TAB>line1_col2<CR><LF>
          line2_col1<TAB>line2_col2<CR><LF>
          line3_col1<TAB>line3_col2<CR><LF>
          line4_col1<TAB>line4_col2<CR><LF>

Vector output (5 events):

• Line 0: msg length 25, starts with U+FEFF (BOM), ends with CR
• Line 1: msg length 22, no BOM, ends with CR
• Line 2: msg length 22, no BOM, ends with CR
• Line 3: msg length 22, no BOM, ends with CR
• Line 4: msg length 22, no BOM, ends with CR

Observations:

1. The first line carries a 3-byte UTF-8 BOM (U+FEFF, displays as zero-width space) which becomes a literal character in VRL strings.
2. EVERY line (not just the first) carries a trailing \r after the data.
3. If downstream parsing splits on whitespace or uses regex like [^\t]+$, the BOM-prefixed first column fails to match.

Verification:

# Create test file with BOM + CRLF:
printf '\xEF\xBB\xBFline1\tcol2\r\ndata1\tcol2\r\n' > /tmp/bom-test.tsv

# Run Vector with simple remap:
cat > /tmp/cfg.yaml << 'EOF'
sources:
  src:
    type: file
    include: [/tmp/bom-test.tsv]
    data_dir: /tmp/vdata
    read_from: beginning
    ignore_checkpoints: true
transforms:
  parse:
    type: remap
    inputs: [src]
    source: |
      .msg = string!(.message)
      .has_bom = starts_with(.msg, "\u{FEFF}")
sinks:
  out:
    type: file
    inputs: [parse]
    path: /tmp/out.jsonl
    encoding: { codec: json }
EOF
vector --config /tmp/cfg.yaml
# Expected output: .has_bom is true for line 1, false for line 2

Impact:

1. Every line of a CRLF file carries invisible CR at the end. Pipelines that use regex like [^\t]*$ fail to match the last column.
2. The first line carries 3 invisible BOM bytes. Downstream parsers (JSON Lines, CSV readers) see corrupted first column.
3. Metrics and labels that use VRL string operations contain \r and \u{FEFF} characters.
4. Common in mixed-OS deployments where upstream Linux files are sent through Windows tools (Excel, Notepad) before reaching Vector.

Expected behavior:

Default behavior should be configurable. Two reasonable options:

• Strip BOM from the first line, strip trailing CR from all lines (if line_delimiter is \n)
• Add explicit config flags strip_bom: true and strip_cr: true

Workaround:

In every remap transform, manually strip:

raw_line = string!(.message)
if starts_with(raw_line, "\u{FEFF}") {
  raw_line = slice!(raw_line, 3)  # 3-byte BOM
}
raw_line = replace(raw_line, "\r", "")

This workaround has to be repeated in every pipeline that ingests Windows-exported files. Failure to add it causes:

• First record to fail BOM-prefixed column parsing
• Every record's last column to retain trailing CR
• Idempotency: same input read twice may produce different output if first read consumes BOM

Confirmed in source: lib/file-source/src/file_watcher/mod.rs:247 (Vector 0.58.0)

Verified empirically: 5-row file with BOM + CRLF -> 5 events received, all 5 have trailing CR, first has \u{FEFF} prefix.