[Azure Cosmos Benchmark]Remove CodaHale metrics from azure-cosmos-benchmark, use SDK Micromet…

[xinlian12]

Remove CodaHale Metrics + Modernize Benchmark Architecture

---

1. SDK Metrics as Single Source of Truth

Removed all app-level metric tracking (MetricRegistry, Meter, Timer, HdrHistogram) from every benchmark class. Benchmark classes are now pure workload drivers. The SDK emits cosmos.client.op.calls, cosmos.client.op.latency, cosmos.client.op.RUs with full tag dimensions (Container, Operation, OperationStatusCode, ClientCorrelationId, RegionName, PartitionKeyRangeId, etc.).

2. Reactor Flux.flatMap Concurrency Control

Replaced hand-rolled Semaphore + BaseSubscriber + AtomicLong count + synchronized wait/notify with Flux.flatMap(concurrency). Subclasses return Mono<T> from performWorkload(long i). Uses Schedulers.parallel() (global shared singleton) — must NOT be disposed by the benchmark.

3. Reporter Architecture — Destination-Based

Reporting destinations configured via metrics.destination in workload JSON. Console logging (LoggingMeterRegistry) is always active alongside any destination for real-time visibility.

All reporters read from Micrometer SimpleMeterRegistry directly — no Dropwizard bridge. This eliminates the per-record() overhead of DropwizardBridgeMeterRegistry (which maintained a parallel set of Dropwizard meters backed by ExponentiallyDecayingReservoir, causing ~30% perf degradation on the hot path).

If multiple destinations are configured, a warning is logged and only the first match is used.

Destination	Config Key	Implementation
Console (always on)	none	Micrometer LoggingMeterRegistry (logs all meters via SLF4J)
CSV	destination.csv	CsvMetricsReporter — iterates SimpleMeterRegistry.getMeters() directly, writes per-metric CSV files (buffered per reporting cycle)
Cosmos DB	destination.cosmos	CosmosMetricsReporter — reads all Micrometer meters (including JVM/Netty) with explicit tag dimensions
App Insights	destination.applicationInsights	AzureMonitorMeterRegistry — SDK sends all meters+tags to Log Analytics/Kusto

4. Removed skipWarmUpOperations Support

The warmup mechanism (skipWarmUpOperations config) has been completely removed. It was tightly coupled to the Dropwizard MetricRegistry — it worked by resetting and re-registering Dropwizard meters after N initial operations. With the switch to Micrometer (where meters are managed by the SDK's ClientTelemetryMetrics, not the benchmark app), warmup metrics cannot be cleanly isolated from measurement data since counters are cumulative and the registry is shared.

5. Workload JSON Config

Console (default — no destination needed):

{
  "metrics": {
    "printingInterval": 10,
    "enableJvmStats": true
  },
  "tenantDefaults": { "serviceEndpoint": "https://...", "masterKey": "..." },
  "tenants": [{ "operation": "ReadLatency", "numberOfOperations": 100000 }]
}

CSV destination:

{
  "metrics": {
    "printingInterval": 10,
    "destination": {
      "csv": { "reportingDirectory": "/tmp/benchmark-output" }
    }
  },
  "tenantDefaults": { "serviceEndpoint": "https://...", "masterKey": "..." },
  "tenants": [{ "operation": "ReadLatency", "numberOfOperations": 100000 }]
}

Cosmos DB destination:

{
  "metrics": {
    "printingInterval": 10,
    "destination": {
      "cosmos": {
        "serviceEndpoint": "https://<results-account>.documents.azure.com:443/",
        "masterKey": "<results-account-key>",
        "database": "benchmarkResults",
        "container": "metrics",
        "testVariationName": "baseline-read-1k",
        "branchName": "main",
        "commitId": "abc1234"
      }
    }
  },
  "tenantDefaults": { "serviceEndpoint": "https://...", "masterKey": "..." },
  "tenants": [{ "operation": "ReadLatency", "numberOfOperations": 100000 }]
}

Application Insights destination:

{
  "metrics": {
    "printingInterval": 10,
    "destination": {
      "applicationInsights": {
        "connectionString": "InstrumentationKey=...;IngestionEndpoint=...",
        "stepSeconds": 10,
        "testCategory": "nightly-perf"
      }
    }
  },
  "tenantDefaults": { "serviceEndpoint": "https://...", "masterKey": "..." },
  "tenants": [{ "operation": "ReadLatency", "numberOfOperations": 100000 }]
}

Typed config classes with null validation: CsvReporterConfig, CosmosReporterConfig, AppInsightsReporterConfig.

6. Resource Lifecycle

• Scheduler: Schedulers.parallel() (global shared, not disposed)
• Meter registries: compositeRegistry.close(), appInsightsRegistry.close() in finally
• Console: loggingRegistry.close() in finally
• JVM binders: JvmGcMetrics etc. stored and .close()d in finally
• ThreadPrefixGaugeSet: close() method shuts down internal executor
• Netty metrics: compositeRegistry added to Metrics.globalRegistry when enabled, removed in finally

7. LinkedIn CTL Workload Metrics

Injected shared MeterRegistry into AsyncClientFactory.buildAsyncClient() so LICtlWorkload's Cosmos client participates in SDK telemetry reporting.

8. Files

New:

• ReportingDestination.java
• CsvMetricsReporter.java (reads Micrometer SimpleMeterRegistry directly, writes CSV)
• CosmosMetricsReporter.java (Cosmos DB upload with full Micrometer tag dimensions)
• CsvReporterConfig.java, CosmosReporterConfig.java, AppInsightsReporterConfig.java
• ReporterConfigValidationTest.java (9 unit tests for config validation)
• workload-config-sample.json

Deleted:

• DropwizardBridgeMeterRegistry.java (no longer needed)
• ScheduledReporterFactory.java, BenchmarkRequestSubscriber.java, NettyHttpMetricsReporter.java
• linkedin/impl/metrics/MetricsFactory.java, MetricsImpl.java, Metrics.java

Dependencies:

• Removed: metrics-core, metrics-jvm, metrics-graphite, hdrhistogram-metrics-reservoir
• Kept: micrometer-core, micrometer-registry-azure-monitor, micrometer-registry-graphite

---

9. Benchmark Results — PR vs Main (Multi-Destination)

VM: Standard_D16s_v3 (16 vCPU, 64GB) | Workload: ReadThroughput, 30 tenants, GATEWAY mode, 10-min duration | Commit: PR 00fe591 vs Main 00a35f4

Console Destination (PR vs Main)

Metric	PR	Main	Δ
Throughput	44.4M ops (70.3K ops/s)	43.7M ops (68.6K ops/s)	≡ equivalent
Latency (mean)	6.8 ms	6.9 ms	≡
CPU	94.0% avg	94.3% avg	≡
GC	565 collections, 18ms max	565 collections, 20ms max	≡

CSV Destination (PR vs Main)

PR uses Micrometer CsvMetricsReporter → per-metric CSV files. Main uses CodaHale CsvReporter.

Metric	PR (Micrometer)	Main (CodaHale)	Δ
Throughput	44.1M ops (71.7K ops/s)	41.4M ops (69.0K ops/s)	≡ equivalent
Latency (mean)	6.9 ms	—	—
GC	560 collections	554 collections	≡

Cosmos DB Destination (PR only)

Main branch has a known NPE bug in CosmosTotalResultReporter that prevents uploads.

Metric	PR
Throughput	50.7M ops (81.0K ops/s)
CPU	46.6% avg, 99.8% max
GC	142 collections, 17ms max

Application Insights Destination (PR vs Main)

Metric	PR	Main
Throughput	44.6M ops (73.9K ops/s)	41.1M ops (66.3K ops/s)
Data Points Ingested	68,934	—
Unique Metric Names	32	—

Cross-Destination Comparison (PR branch, same commit)

Metric	Console	CSV	Cosmos DB	App Insights
Total Ops	44.4M	44.1M	50.7M	44.6M
Ops/s	70.3K	71.7K	81.0K	73.9K
Peak Threads	262	277	291	288
Fixed pools identical?	✅	✅	✅	✅
Netty conns/endpoint	20	20	20	20

Conclusion: ✅ No regression detected. All fixed-size thread pools, netty connections, latency, and throughput are equivalent between PR and main across all 4 destinations.

[Copilot]

Pull request overview

This PR refactors the azure-cosmos-benchmark tool to remove app-level CodaHale/Dropwizard metric tracking from benchmark workloads and instead rely on Micrometer + Cosmos SDK-emitted meters as the primary metrics source, with a new destination-based reporting model and simplified concurrency control.

Changes:

• Replaced per-benchmark metric tracking and custom subscribers/semaphore concurrency with Flux.flatMap(concurrency) patterns and scheduler injection.
• Introduced destination-based metrics reporting (console/CSV via Dropwizard bridge, Cosmos DB uploader, Azure Monitor/App Insights registry).
• Updated benchmark orchestration/config parsing and removed legacy metrics/reporting infrastructure and dependencies.

Reviewed changes

Copilot reviewed 48 out of 48 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
sdk/cosmos/azure-cosmos-benchmark/workload-config-sample.json	Adds sample JSON illustrating new metrics.destination configuration.
sdk/cosmos/azure-cosmos-benchmark/src/test/java/com/azure/cosmos/benchmark/WorkflowTest.java	Updates tests to construct workloads with a Reactor Scheduler instead of MetricRegistry.
sdk/cosmos/azure-cosmos-benchmark/src/test/java/com/azure/cosmos/benchmark/ReadMyWritesConsistencyTest.java	Updates consistency test to use Scheduler injection instead of MetricRegistry.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/impl/metrics/MetricsImpl.java	Deletes legacy LinkedIn CTL metrics implementation (CodaHale-based).
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/impl/metrics/MetricsFactory.java	Deletes legacy metrics factory used by LinkedIn CTL code.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/impl/QueryExecutor.java	Removes per-operation metrics tracking hooks from query execution path.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/impl/Metrics.java	Deletes legacy metrics interface.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/impl/GetExecutor.java	Removes per-operation metrics tracking hooks from get execution path.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/impl/CosmosDBDataAccessor.java	Updates constructors/signatures after removing MetricsFactory dependency.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/TestRunner.java	Removes shared Dropwizard MetricRegistry dependency from CTL runners.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/QueryTestRunner.java	Removes MetricRegistry plumbing after metrics removal.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/LICtlWorkload.java	Removes MetricRegistry from workload construction and runner creation.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/GetTestRunner.java	Removes MetricRegistry from runner construction.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/linkedin/CompositeReadTestRunner.java	Removes MetricRegistry from runner construction.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/encryption/AsyncEncryptionWriteBenchmark.java	Converts workload execution to return Mono and removes subscriber/semaphore timing logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/encryption/AsyncEncryptionReadBenchmark.java	Converts workload execution to return Mono and removes subscriber/semaphore timing logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/encryption/AsyncEncryptionQuerySinglePartitionMultiple.java	Converts workload execution to return Mono and removes subscriber/semaphore logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/encryption/AsyncEncryptionQueryBenchmark.java	Converts workload execution to return Mono and removes custom latency subscriber logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/encryption/AsyncEncryptionBenchmark.java	Reworks core async runner to use Flux.generate + flatMap(concurrency) with injected scheduler and removes CodaHale meters.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/ctl/AsyncCtlWorkload.java	Reworks CTL workload runner to return Mono operations and use flatMap(concurrency) with injected scheduler.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/ThreadPrefixGaugeSet.java	Replaces Dropwizard MetricSet with Micrometer MeterBinder for thread-prefix gauges.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/TenantWorkloadConfig.java	Removes legacy meter name constants tied to old Dropwizard metrics.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/SyncWriteBenchmark.java	Removes Dropwizard MetricRegistry dependency from sync benchmarks.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/SyncReadBenchmark.java	Removes Dropwizard MetricRegistry dependency from sync benchmarks.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/SyncBenchmark.java	Removes Dropwizard meters/timers from sync benchmark base.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/ScheduledReporterFactory.java	Deletes legacy factory that chose ConsoleReporter vs CsvReporter from a MetricRegistry.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/ReportingDestination.java	Adds enum for metrics destination selection.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/ReadMyWriteWorkflow.java	Converts workflow execution to return Mono and removes semaphore/subscriber-based dispatch.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/NettyHttpMetricsReporter.java	Deletes custom CSV netty pool metrics sampler reporter.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/DropwizardBridgeMeterRegistry.java	Adds Micrometer->Dropwizard bridge registry for Dropwizard reporters.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/CsvReporterConfig.java	Adds typed config for CSV destination.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/CsvMetricsReporter.java	Adds Dropwizard Console/CSV reporter wrapper reading from the bridge registry.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/CosmosTotalResultReporter.java	Deletes legacy Dropwizard ScheduledReporter that uploaded aggregate results to Cosmos.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/CosmosReporterConfig.java	Adds typed config for Cosmos DB destination.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/CosmosMetricsReporter.java	Adds Micrometer meter uploader to Cosmos DB preserving tag dimensions.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/BenchmarkRequestSubscriber.java	Deletes legacy subscriber used for semaphore-based concurrency + latency timers.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/BenchmarkOrchestrator.java	Reworks orchestrator to build composite registries, handle destinations, inject scheduler, and manage lifecycle cleanup.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/BenchmarkConfig.java	Reworks config parsing to support destination-specific reporter configs under metrics.destination.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/AsyncWriteBenchmark.java	Converts async write workload to return Mono and removes custom subscriber-based timing logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/AsyncReadManyBenchmark.java	Converts readMany workload to return Mono and removes subscriber-based timing logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/AsyncReadBenchmark.java	Converts async read workload to return Mono and removes subscriber-based timing logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/AsyncQuerySinglePartitionMultiple.java	Converts query workload to return Mono and removes subscriber-based timing logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/AsyncQueryBenchmark.java	Converts async query workload to return Mono and removes custom latency subscriber logic.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/AsyncMixedBenchmark.java	Converts mixed workload to return Mono and removes subscriber/semaphore dispatch.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/AsyncBenchmark.java	Reworks core async runner to Flux.generate + flatMap(concurrency) and injected scheduler; removes Dropwizard tracking.
sdk/cosmos/azure-cosmos-benchmark/src/main/java/com/azure/cosmos/benchmark/AppInsightsReporterConfig.java	Adds typed config for App Insights/Azure Monitor destination.
sdk/cosmos/azure-cosmos-benchmark/pom.xml	Removes legacy Dropwizard metrics modules/reservoir dependency; keeps metrics-core and Micrometer registries.
eng/versioning/external_dependencies.txt	Removes dropped third-party dependencies from tracked list.

---

You can also share your feedback on Copilot code review. Take the survey.

[xinlian12]

PR Deep Review — Remove CodaHale Metrics + Modernize Benchmark Architecture

Overall: Excellent modernization — net -1015 lines, dramatically simplified benchmark subclasses, and idiomatic Reactor concurrency control. The direction is right. Below are 13 findings (2 blocking, 6 recommendations, 4 suggestions, 1 observation) focused on correctness gaps, resource leaks, and cross-SDK insights from the .NET benchmark tool.

Cross-SDK: The .NET Cosmos benchmark (Azure/azure-cosmos-dotnet-v3) retains warmup support, supports coexistent reporting destinations (console + CSV/Cosmos simultaneously), and maintains app-level metrics alongside SDK metrics — all areas where this PR diverges.

---

_{⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.}

[FabianMeiswinkel]

Love it! LGTM - Thx!