fix: stop endless recursion in resolveValues on assignment cycles - #530
Merged
Conversation
Fixes #525. resolveValues follows a variable's assignments and its initializer to find its value. It had no cycle check. Code like String algorithm = "AES"; String copy = algorithm; ... algorithm = copy; made it recurse forever and crash the scan with a StackOverflowError. The existing selections.size() > 15 guard never fires here, because identifier-to-identifier hops do not grow the selections list. The bug is old, but 1.6.1 exposed it: since e1fdab3 the engine resolves the arguments of every method call (for detached call records), not only the arguments of matched crypto calls. So a cycle anywhere in the scanned code now reaches resolveValues. The fix moves the variable branch into resolveVariableValues and tracks which variables are being resolved on the current path. A variable that is already on the path is not followed again. The variable is released again when its resolution is done, so two sibling branches may still resolve through the same variable. A new red-green test reproduces the exact alternating stack trace from the issue and checks that the value still resolves through the cycle to the constant initializer.
Contributor
san-zrl
approved these changes
Aug 31, 2026
san-zrl
left a comment
Contributor
There was a problem hiding this comment.
LGTM. This also fixes the StackOverflow in the CI pipeline when scanning
BlockCipherReorganizer.java with a simpler call stack:
[main] ERROR org.sonar.java.ast.JavaAstScanner - A stack overflow error occurred while analyzing file: 'src/main/java/com/ibm/mapper/reorganizer/rules/BlockCipherReorganizer.java'
java.lang.StackOverflowError
at com.ibm.engine.language.java.JavaDetectionEngine.resolveValues(JavaDetectionEngine.java:318)
at com.ibm.engine.language.java.JavaDetectionEngine.resolveValues(JavaDetectionEngine.java:337)
at com.ibm.engine.language.java.JavaDetectionEngine.resolveValues(JavaDetectionEngine.java:337)
at com.ibm.engine.language.java.JavaDetectionEngine.resolveValues(JavaDetectionEngine.java:337)
...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #525.
What was wrong
JavaDetectionEngine.resolveValuesfollows a variable's assignments and its initializer to find its value. It had no cycle check. Code like this made it recurse forever:Resolving
copyjumps between the assignment branch (line 337) and the initializer branch (line 353) forever. This is exactly the alternating stack trace in #525. The existingselections.size() > 15guard never fires, because identifier-to-identifier hops do not growselections.The bug is old and exists in 1.6.0 too. But 1.6.1 exposed it: since e1fdab3 the engine resolves the arguments of every method call (to build detached call records), not only the arguments of matched crypto calls. So a cycle anywhere in the scanned code now crashes the scan. The same change caused the "Detected constructor definition has more then one argument" log flood from the issue.
The fix
The variable branch of
resolveValuesmoves into a new methodresolveVariableValues. It tracks which variable symbols are being resolved on the current path:resolvingVariablesset.finallyblock removes it again.The release step matters: it cuts only real cycles. Two sibling branches (for example two array elements) may still resolve through the same shared variable. A global "visited" set without release would drop the second value.
Red-green test
JcaCipherGetInstanceCyclicAssignmentTestreproduces the issue:mainthe test dies with the exact alternatingStackOverflowErrorfrom Regression in 1.6.1 (compared to 1.6.0): StackOverflowError and excessive logging #525."AES/ECB/PKCS5Padding", giving the normalAES-128-ECB-PKCS5finding. So cutting the cycle does not lose the detection.Not in this PR
LOGGER.infofor constructors with more than one argument. Fixes StackOverflow in JavaDetectionEngine:resolveValues #529 already demotes it totrace; that part fits better there.buildDetachedCalldoes per call. That is a performance topic, not a crash, and deserves its own issue.Verified with
mvn test -pl engine,java: 164 tests, 0 failures. Spotless and checkstyle clean.