PHOENIX-7992 Add replay-side performance metrics for HA replication - #2607
Draft
Himanshu-g81 wants to merge 3 commits into
Draft
PHOENIX-7992 Add replay-side performance metrics for HA replication#2607Himanshu-g81 wants to merge 3 commits into
Himanshu-g81 wants to merge 3 commits into
Conversation
Adds six standby replay-side performance metrics, each inheriting the
per-haGroup dimension from its metric source.
ReplicationLogDiscoveryReplay:
- endToEndReplayLagMs (histogram): round-eligible -> replay finished
- pickupLagMs (histogram): round-eligible -> file claimed (rename into
the in-progress directory)
ReplicationLogDiscovery (shared base):
- roundsExceedingRoundTime (counter): rounds whose new-file processing
duration exceeded roundTimeMills
ReplicationLogProcessor:
- mutationsReplayedCount (counter): replay throughput across all files
- mutationsPerFile (histogram): per-file mutation count, recorded only
for files that replayed at least one mutation so rotation-only files
do not skew the distribution toward zero
ReplicationLogTracker:
- markFileInProgressCollisionCount (counter): claim renames lost to a
competing process, a strict subset of the request count
The lag metrics use the round-eligible instant (round end + waiting
buffer) as the zero reference, so they exclude the fixed built-in wait
rather than measuring raw file age.
Tests: metric-source plumbing for all six metrics, plus recording-site
tests for the collision metric (rename-loses vs rename-wins) and the
slow-round metric (fast round vs slow round).
Record pickupLag only on a file's first claim, skipping in-progress reclaims that re-stamp the rename timestamp; anchor round-boundary files to the earlier round in getRoundEligibleTime; and rename two counters to match what they actually measure: mutationsReplayedCount -> successfulFileMutationsReplayedCount and markFileInProgressCollisionCount -> markFileInProgressRenameFailedCount. Adds ReplicationLogDiscoveryReplayProcessFileTest covering the eligibility math, first-claim-only pickup lag, and mid-replay failure handling.
…fy metric docs Follow-up addressing review feedback on apache#2607. Tests (C1): drive ReplicationLogProcessor.processLogFile end to end to guard the per-file mutation-metric recording sites, and add a mid-replay-failure case proving mutations applied before a failure are not counted (the writes sit before the catch). Assert the reset-immune successfulFileMutationsReplayedCount counter rather than the mutationsPerFile histogram max: MutableHistogram.getMax() is reset to 0 by the metrics system's periodic snapshot(), so a positive max is not stable to read after a multi-second real replay. Zero-mutation and failed files still assert a 0 histogram max, which is stable. Reclaim path (S1): convert the unchecked NumberFormatException from getFileTimestamp on a malformed in-progress file name into an IOException, so a single bad name is isolated to processOneRandomFile's per-file handler (marked failed and retry-counted) instead of escaping as a RuntimeException that aborts the whole in-progress sweep for the cycle. Adds a unit test. Docs: - roundsExceedingRoundTime lives on the shared discovery base, so clarify it is emitted on both the forwarder and replay sides, not replay-only (A5). - Note the endToEndReplayLag / pickupLag histograms are survivorship-biased (sampled only on success / first claim) and should be read alongside the failure and staleness signals (D1). - Note markFileInProgressRenameFailedCount counts only rename() == false, not an IOException thrown during the rename (D2). - Note endToEndReplayLag samples rotation-only / zero-mutation files that mutationsPerFile excludes, so the two cover different file populations (C2). - Fix stale processFile(Path) Javadoc to processFile(Path, boolean) (C4). - Log a breadcrumb when a first-claim file carries no rename timestamp (D4).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds six per-HA-group standby replay-side performance metrics (replay lag, pickup lag, mutations replayed + per-file, rename-failed count, slow-round count) for HA replication — new observability surface, no behavioral change.