HDFS-17955. Fix Checkpointer startup checkpoint on fresh VMs - #8649
Merged
Conversation
The Checkpointer relies on lastCheckpointTime=0 combined with now >= lastCheckpointTime + checkpointPeriodMSec to trigger the first checkpoint when shouldCheckpointAtStartup() returns true. This breaks when monotonicNow() (System.nanoTime based) returns a value smaller than checkpointPeriodMSec (3600000ms = 1 hour), which happens on freshly booted VMs such as GitHub-hosted runners. The checkpoint never fires, causing TestBackupNode to hang indefinitely. Add an explicit lastCheckpointTime == 0 check so the first checkpoint fires immediately regardless of the monotonic clock value. Assisted-by: GLM 5.2
Member
Author
|
cc @Hexiaoqiao @ajfabbri @slfan1989 |
|
💔 -1 overall
This message was automatically generated. |
Hexiaoqiao
approved these changes
Aug 3, 2026
Hexiaoqiao
left a comment
Contributor
There was a problem hiding this comment.
LGTM. +1. Thanks @pan3793 .
pan3793
added a commit
that referenced
this pull request
Aug 3, 2026
Authored-by: Cheng Pan <chengpan@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
pan3793
added a commit
that referenced
this pull request
Aug 3, 2026
Authored-by: Cheng Pan <chengpan@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
Member
Author
|
thanks, merged to trunk/3.5/3.4 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
The Checkpointer triggers the first checkpoint when
shouldCheckpointAtStartup()returns true by leavinglastCheckpointTime = 0and relying on
now >= lastCheckpointTime + checkpointPeriodMSectoevaluate true. This condition breaks when
monotonicNow()(
System.nanoTimebased) returns a value smaller thancheckpointPeriodMSec(3600000ms = 1 hour), which happens on freshlybooted VMs such as GitHub-hosted runners. The checkpoint never fires,
and any caller waiting for it hangs indefinitely.
Add an explicit
lastCheckpointTime == 0check so the first checkpointfires immediately regardless of the monotonic clock value. After the
first checkpoint,
lastCheckpointTimeis set to the current time andnormal period-based scheduling resumes.
This is a latent bug present since the Checkpointer logic was introduced
in 2011 (a196766). It was exposed by the GitHub-hosted runner image
ubuntu24/20260720.247, which provisions VMs fast enough that hostuptime is below 1 hour at test execution. A BackupNode that restarts
within 1 hour of boot on any deployment has the same bug.
How was this patch tested?
Verified via GitHub Actions CI on a fork branch. Before the fix,
TestBackupNodehangs indefinitely (fork timeout after 900s). After thefix, the Checkpointer fires the startup checkpoint immediately and
TestBackupNodepasses.For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
<tool>"where
<tool>is the name of the AI tool used.https://www.apache.org/legal/generative-tooling.html
Contains content generated by GLM 5.2