fix(telemetry): make metrics best-effort so telemetry never crashes storage ops

[jayya2]

Description

OpenTelemetry version skew on some clusters made metric recording raise
AttributeError: 'AutoProxy[opentelemetry.metrics.Counter]' object has no attribute 'add',
crashing daemon upload/download jobs. The shared cross-process Counter/Gauge proxies are
registered without an explicit exposed method set, so multiprocessing's default discovery
(public_methods via dir()) drops add/set whenever the installed OpenTelemetry build
delegates those methods on the instrument.

This makes telemetry best-effort and hardens the proxies:

• Explicit proxy methods — declare exposed=["add"] / exposed=["set"] on the Counter/Gauge
  registrations so the proxy method set no longer depends on the installed OpenTelemetry's instrument
  shape (mirrors the existing explicit exposed used for Span).
• Fail-soft telemetry — disable metrics/traces on any provider/exporter construction failure (and
  cache the disabled state); latch the synchronous record path so a failing record disables metrics
  instead of crashing the storage operation; skip metrics init entirely when MSC_TELEMETRY_DISABLED
  or OTEL_SDK_DISABLED is set.
• mTLS fail-fast — clear error when the installed OTLP exporter predates the mTLS client-certificate
  options, instead of an opaque TypeError.

Adds unit tests (proxy exposed set, disable-on-failure paths, record latch, env kill-switches) and
documents disabling telemetry.

Combined-design note: with the explicit exposed, a manager-mode proxy wrapping a disabled (None)
instrument still exposes the record method, so crash-safety for that path now comes from the
record-time latch rather than up-front detection. test_manager_mode_disabled_metrics_do_not_crash_record
was updated to assert that contract.

Checklist

• Development PR
  • .release_notes/.unreleased.md
    • Notable changes to the client from this PR have been added.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: af56b673-d430-4476-90a5-00720ae806ab

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/telemetry-crash-fix-ec5bb5

---

_{Comment @coderabbitai help to get the list of available commands.}

[jayya2]

Superseded — split into two independent PRs per review:

• fix(telemetry): make metrics best-effort so telemetry never crashes storage ops #125 — best-effort telemetry (never crash a storage operation)
• fix(telemetry): declare exposed methods on shared metric instrument proxies #124 — declare explicit exposed methods on the cross-process metric instrument proxies

Closing this combined PR.