feat: Platform Hardening & Observability: Distributed Tracing, Recurring Billing, WebSocket Monitoring, Payout Security

[Just-Bamford]

Description

Summary

This PR implements four critical platform improvements addressing observability, retention, infrastructure resilience, and financial security.

Closes #1420
Closes #1421
Closes #1422
Closes #1423

Issues Addressed

1. Distributed Tracing Across Service Boundaries

• Problem: No correlation-ID propagation across Next.js → DB → Mux → Horizon call chain; debugging production incidents requires manual log correlation with no reliable way to group scattered logs
• Solution: Implemented correlation-ID-based distributed tracing using AsyncLocalStorage
• Changes:
  • lib/tracing/trace-context.ts - Core trace context management with per-request unique IDs
  • lib/tracing/logger.ts - Structured JSON logging with automatic trace context injection
  • middleware.ts - Global middleware generating/extracting trace IDs for all requests
  • lib/tracing/api-route-wrapper.ts - Higher-order route handler wrapper with automatic tracing
  • lib/tracing/fetch-tracer.ts - Outbound request wrapper for Mux/Horizon/external API calls
  • lib/tracing/db-tracer.ts - Database query tracer with SQL comment injection
  • TRACING_ARCHITECTURE.md - Complete architecture documentation
• Instrumented:
  • lib/mux/server.ts - All Mux API calls now log with trace context
  • lib/stellar/horizon.ts & lib/stellar/payments.ts - All Horizon/Stellar calls traced
  • app/api/tips/send/route.ts - End-to-end example showing full trace flow
• Result: Every request has a unique x-request-id flowing through all services; engineers can grep by trace ID to follow entire request lifecycle across all systems

2. Recurring-Billing Simulation: Expiry Alerts & One-Click Renewal

• Problem: Subscriptions are one-time purchases; no proactive expiry alerts or streamlined renewal flow; subscribers churn through forgetting to renew, not deliberate cancellation
• Solution: Implemented proactive expiry notifications (3 days before) and one-click renewal with pre-populated checkout
• Changes:
  • lib/subscriptions/expiry-notifier.ts - Expiry notification logic with configurable thresholds
  • app/api/routes-f/cron-subscription-expiry-alerts/route.ts - Cron job sending expiry alerts
  • app/api/routes-f/subscription-renew/route.ts - One-click renewal retrieval (pre-populated intent)
  • app/api/routes-f/subscription-renew-confirm/route.ts - Renewal confirmation with new payment
  • app/api/routes-f/subscription-renew/__tests__/subscription-renewal.test.ts - Comprehensive tests
  • app/api/routes-f/subscriptions/route.ts - Updated schema with expiry_alert_sent_at and renewal_count
  • AUTOPAY_ARCHITECTURE_DECISION.md - Phased autopay roadmap (Phase 1: this PR, Phase 2: custodial balance, Phase 3: Soroban smart contract)
• Result: 3-day expiry alerts + one-click renewal reducing friction; roadmap documented for eventual true autopay (Soroban or custodial)

3. WebSocket/SSE Connection Monitoring & Per-Instance Capacity Limits

• Problem: No way to monitor concurrent connections per instance; no safety valve before capacity cliff; realtime layer can degrade/crash under load with no graceful rejection
• Solution: Decoupled per-instance connection manager with configurable capacity limits, graceful shedding, and alerting (deployable once realtime infrastructure decided)
• Changes:
  • lib/realtime/connection-manager.ts - Per-instance connection tracking with capacity enforcement, three shedding strategies (reject_new, shed_oldest, shed_least_active), priority-based shedding
  • lib/realtime/metrics-exporter.ts - Prometheus + JSON metrics export, alert sink for external integrations (Datadog, PagerDuty)
  • lib/realtime/websocket-server-example.ts - Reference WebSocket server integration showing lifecycle hooks, metrics export, graceful shutdown
  • lib/realtime/__tests__/connection-manager.test.ts - 30+ tests covering capacity enforcement, shedding strategies, graceful shutdown, load scenarios
  • REALTIME_CONNECTION_MONITORING_SPEC.md - Complete specification with deployment checklist, alerting configuration, load testing strategy
• Result: Metrics infrastructure ready for any realtime choice (managed provider or self-hosted); per-instance limits prevent single instance from overwhelming; alert integration enables proactive capacity management

4. Payout-Address Verification to Prevent Silent Redirection

• Problem: Payout address can be changed via account settings with no additional verification; compromised session can redirect all future creator revenue to attacker's wallet with no detection window
• Solution: Time-locked verification (3-day cooldown) with out-of-band email notification, cancellation path, optional test-payment verification
• Changes:
  • lib/payouts/payout-address-verifier.ts - Time-locked changes with state machine (pending → verified → active), out-of-band notifications, test-payment confirmation, rate limiting, cancellation support
  • app/api/routes-f/payout-address-change/route.ts - Initiate time-locked change (authenticated)
  • app/api/routes-f/payout-address-cancel/route.ts - Cancel unverified changes (supports no-auth email links + authenticated cancellation)
  • app/api/routes-f/payout-address-verify-payment/route.ts - Confirm test payment (wallet ownership verification)
  • app/api/routes-f/cron-verify-pending-payout-changes/route.ts - Auto-verify after cooldown elapses
  • app/api/routes-f/payout-address-change/__tests__/payout-verification.test.ts - 40+ tests covering cooldown enforcement, cancellation, test-payment flow, rate limiting
  • PAYOUT_ADDRESS_SECURITY_SPEC.md - Security architecture with attack scenarios, API specs, email templates, deployment checklist
• Result: 3-day cooldown gives legitimate owner time to notice and cancel unauthorized changes; out-of-band notification reaches account owner's email (attacker unlikely to control); test-payment verifies wallet ownership; defense-in-depth against account compromise

Testing

All implementations include comprehensive test coverage:

• Tracing: End-to-end trace flow verified in app/api/tips/send/route.ts
• Subscriptions: Notification timing, cancellation, renewal flow with happy/failure paths
• WebSocket: Ramp-up, capacity ceiling, graceful shutdown, priority shedding, sustained load
• Payouts: Cooldown enforcement, cancellation, test-payment verification, rate limiting, attack scenarios

Files Changed

Core Implementation

• lib/tracing/ - Distributed tracing (5 files)
• lib/subscriptions/ - Expiry notification system (1 file)
• lib/payouts/ - Payout verification (1 file)
• lib/realtime/ - WebSocket monitoring (4 files + 1 test file)

API Endpoints

• app/api/routes-f/cron-subscription-expiry-alerts/ - Expiry alert cron
• app/api/routes-f/subscription-renew/ - One-click renewal (1 file + 1 test file)
• app/api/routes-f/subscription-renew-confirm/ - Renewal confirmation
• app/api/routes-f/payout-address-change/ - Initiate payout change (1 file + 1 test file)
• app/api/routes-f/payout-address-cancel/ - Cancel payout change
• app/api/routes-f/payout-address-verify-payment/ - Test-payment verification
• app/api/routes-f/cron-verify-pending-payout-changes/ - Payout change verification cron
• app/api/tips/send/ - End-to-end tracing example

Documentation

• TRACING_ARCHITECTURE.md - Distributed tracing design, integration guide, debugging workflow
• AUTOPAY_ARCHITECTURE_DECISION.md - Recurring billing roadmap (Phase 1/2/3) with tradeoffs
• REALTIME_CONNECTION_MONITORING_SPEC.md - WebSocket monitoring architecture, deployment checklist
• PAYOUT_ADDRESS_SECURITY_SPEC.md - Payout security design, attack scenarios, integration guide

Modified Files

• middleware.ts - Global trace ID generation and propagation
• lib/mux/server.ts - Instrumented with structured logging
• lib/stellar/horizon.ts & lib/stellar/payments.ts - Instrumented with trace context
• app/api/routes-f/subscriptions/route.ts - Updated schema for expiry tracking

[vercel]

@Just-Bamford is attempting to deploy a commit to the chibuikemmichaelilonze's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@Just-Bamford Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits