Skip to content

HDFS-17955. Fix Checkpointer startup checkpoint on fresh VMs - #8649

Merged
pan3793 merged 1 commit into
apache:trunkfrom
pan3793:HDFS-17955
Aug 3, 2026
Merged

HDFS-17955. Fix Checkpointer startup checkpoint on fresh VMs#8649
pan3793 merged 1 commit into
apache:trunkfrom
pan3793:HDFS-17955

Conversation

@pan3793

@pan3793 pan3793 commented Aug 2, 2026

Copy link
Copy Markdown
Member

Description of PR

The Checkpointer triggers the first checkpoint when
shouldCheckpointAtStartup() returns true by leaving lastCheckpointTime = 0
and relying on now >= lastCheckpointTime + checkpointPeriodMSec to
evaluate true. This condition breaks when monotonicNow()
(System.nanoTime based) returns a value smaller than
checkpointPeriodMSec (3600000ms = 1 hour), which happens on freshly
booted VMs such as GitHub-hosted runners. The checkpoint never fires,
and any caller waiting for it hangs indefinitely.

Add an explicit lastCheckpointTime == 0 check so the first checkpoint
fires immediately regardless of the monotonic clock value. After the
first checkpoint, lastCheckpointTime is set to the current time and
normal period-based scheduling resumes.

This is a latent bug present since the Checkpointer logic was introduced
in 2011 (a196766). It was exposed by the GitHub-hosted runner image
ubuntu24/20260720.247, which provisions VMs fast enough that host
uptime is below 1 hour at test execution. A BackupNode that restarts
within 1 hour of boot on any deployment has the same bug.

How was this patch tested?

Verified via GitHub Actions CI on a fork branch. Before the fix,
TestBackupNode hangs indefinitely (fork timeout after 900s). After the
fix, the Checkpointer fires the startup checkpoint immediately and
TestBackupNode passes.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (HDFS-17955)?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

Contains content generated by GLM 5.2

The Checkpointer relies on lastCheckpointTime=0 combined with
now >= lastCheckpointTime + checkpointPeriodMSec to trigger the first
checkpoint when shouldCheckpointAtStartup() returns true. This breaks
when monotonicNow() (System.nanoTime based) returns a value smaller
than checkpointPeriodMSec (3600000ms = 1 hour), which happens on
freshly booted VMs such as GitHub-hosted runners. The checkpoint
never fires, causing TestBackupNode to hang indefinitely.

Add an explicit lastCheckpointTime == 0 check so the first checkpoint
fires immediately regardless of the monotonic clock value.

Assisted-by: GLM 5.2
@pan3793

pan3793 commented Aug 2, 2026

Copy link
Copy Markdown
Member Author

cc @Hexiaoqiao @ajfabbri @slfan1989
this should fix the root cause of the recent consistently failed "Test hdfs - other", and HADOOP-19950 (#8651) and HDFS-17956 (#8650) are used to diagnose this issue

@hadoop-yetus

Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 19m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 46m 17s trunk passed
+1 💚 compile 1m 44s trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚 compile 1m 52s trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚 checkstyle 1m 48s trunk passed
+1 💚 mvnsite 1m 56s trunk passed
+1 💚 javadoc 1m 28s trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚 javadoc 1m 28s trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚 spotbugs 4m 18s trunk passed
+1 💚 shadedclient 37m 43s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 25s the patch passed
+1 💚 compile 1m 17s the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚 javac 1m 17s the patch passed
+1 💚 compile 1m 21s the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚 javac 1m 21s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 19s the patch passed
+1 💚 mvnsite 1m 28s the patch passed
+1 💚 javadoc 0m 57s the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚 javadoc 1m 4s the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚 spotbugs 4m 7s the patch passed
+1 💚 shadedclient 36m 46s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 260m 34s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 53s The patch does not generate ASF License warnings.
427m 23s
Subsystem Report/Notes
Docker ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8649/1/artifact/out/Dockerfile
GITHUB PR #8649
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 19fb92f5b7d5 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ea1f862
Default Java Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8649/1/testReport/
Max. process+thread count 2275 (vs. ulimit of 10000)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8649/1/console
versions git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

@Hexiaoqiao Hexiaoqiao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. +1. Thanks @pan3793 .

@slfan1989 slfan1989 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pan3793
pan3793 merged commit a645446 into apache:trunk Aug 3, 2026
3 of 5 checks passed
pan3793 added a commit that referenced this pull request Aug 3, 2026
Authored-by: Cheng Pan <chengpan@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
pan3793 added a commit that referenced this pull request Aug 3, 2026
Authored-by: Cheng Pan <chengpan@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
@pan3793

pan3793 commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

thanks, merged to trunk/3.5/3.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants