Skip to content

HDFS-17944. Validate short-circuit streams cache size - #8644

Open
johntomcat7408-cmyk wants to merge 1 commit into
apache:trunkfrom
johntomcat7408-cmyk:HDFS-17944-short-circuit-cache-size
Open

HDFS-17944. Validate short-circuit streams cache size#8644
johntomcat7408-cmyk wants to merge 1 commit into
apache:trunkfrom
johntomcat7408-cmyk:HDFS-17944-short-circuit-cache-size

Conversation

@johntomcat7408-cmyk

Copy link
Copy Markdown

Description of PR

JIRA: HDFS-17944

dfs.client.read.shortcircuit.streams.cache.size=-1 currently passes
DfsClientConf parsing and is rejected only when a new
ShortCircuitCache is built. This exposes the internal maxTotalSize
parameter in the error, while reuse of an existing ClientContext can
avoid that late validation entirely.

This PR:

  • validates that the configured streams cache size is non-negative while
    building DfsClientConf;
  • reports the public configuration key in the exception;
  • adds client-module tests for the invalid -1 boundary and the valid 0
    boundary.

Zero remains valid and keeps its existing cache-retention semantics. The
change does not modify public APIs, protocols, dependencies,
hdfs-default.xml, or the defensive validation in ShortCircuitCache.

Source and attribution: the issue comes from HDFS-17944. The implementation
was independently written against the current trunk. Its error style is
consistent with merged HDFS-17943 / PR #8603,
and the existing zero boundary was confirmed from merged
HDFS-16653 / PR #5568.
Open PR #7215 also touches
DfsClientConf.java, but only for lease renewal configuration; no code or
text from that unmerged PR was reused.

How was this patch tested?

On Windows x86_64 with JDK 21.0.11 and the repository Maven Wrapper:

.\mvnw.cmd --batch-mode --no-transfer-progress `
  -pl :hadoop-hdfs-client -am "-P=-native-win" `
  "-Dtest=TestDfsClientConf" `
  "-Dsurefire.failIfNoSpecifiedTests=false" `
  "-DskipShade" "-DskipDocs" test

Result: BUILD SUCCESS; 10 reactor modules succeeded; 2 tests ran with
0 failures, 0 errors, and 0 skips.

.\mvnw.cmd --batch-mode --no-transfer-progress `
  -pl :hadoop-hdfs -am "-P=-native-win" `
  "-Dtest=TestShortCircuitCache#testInvalidConfiguration" `
  "-Dsurefire.failIfNoSpecifiedTests=false" `
  "-DskipShade" "-DskipDocs" test

Result: BUILD SUCCESS; 12 reactor modules succeeded; 1 test ran with
0 failures, 0 errors, and 0 skips. Bash tests were skipped because bats
was not installed; the targeted Java test completed successfully.

.\mvnw.cmd --batch-mode --no-transfer-progress `
  -pl :hadoop-hdfs-client -am "-P=-native-win" `
  "-DskipTests" "-DskipShade" "-DskipDocs" checkstyle:check

Result: exit code 0. The new test has no Checkstyle violations.
DfsClientConf.java retains a pre-existing MethodLength warning at line
171; the production change is at line 844 and introduces no new violation.

A full repository build was not run.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: N/A; this HDFS client change does not affect an object storage connector.
  • New dependencies: N/A; this PR adds no dependencies.
  • License and notice updates: N/A; this PR does not change distributed dependencies or licensing content.

AI Tooling

  • Contains content generated by Codex
  • My use of AI contributions follows the ASF legal policy.

Reject negative streams cache sizes while building DfsClientConf so
invalid values do not leak the internal maxTotalSize diagnostic or
depend on ClientContext reuse. Preserve zero as a valid value and cover
both boundaries in the client module.
@hadoop-yetus

Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 20m 11s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 46m 25s trunk passed
+1 💚 compile 1m 19s trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚 compile 1m 18s trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚 checkstyle 0m 58s trunk passed
+1 💚 mvnsite 1m 23s trunk passed
+1 💚 javadoc 1m 5s trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚 javadoc 1m 0s trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚 spotbugs 3m 51s trunk passed
+1 💚 shadedclient 37m 26s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 53s the patch passed
+1 💚 compile 0m 48s the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚 javac 0m 48s the patch passed
+1 💚 compile 0m 49s the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚 javac 0m 49s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 25s the patch passed
+1 💚 mvnsite 0m 53s the patch passed
+1 💚 javadoc 0m 36s the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚 javadoc 0m 35s the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚 spotbugs 3m 35s the patch passed
+1 💚 shadedclient 37m 0s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 39s hadoop-hdfs-client in the patch passed.
+1 💚 asflicense 0m 35s The patch does not generate ASF License warnings.
162m 42s
Subsystem Report/Notes
Docker ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8644/1/artifact/out/Dockerfile
GITHUB PR #8644
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 814b523a61fe 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 354fc94
Default Java Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8644/1/testReport/
Max. process+thread count 585 (vs. ulimit of 10000)
modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8644/1/console
versions git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants