[MINOR] fix(test): cover SparkContext shutdown path in HoodieMetadataTableValidator - #19349
[MINOR] fix(test): cover SparkContext shutdown path in HoodieMetadataTableValidator#19349lokeshj1703 wants to merge 1 commit into
Conversation
9c10e6c to
6e38832
Compare
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the contribution! This PR adds two tests covering the catch (SparkException) branches in HoodieMetadataTableValidator.doMetadataTableValidation — one asserting a SparkContext-shutdown message is wrapped in HoodieException, and one asserting other Spark failures still yield HoodieValidationException("Unexpected spark failure"). I traced both tests against the production code: the exception propagation, the assertThrows/assertFalse distinction (relying on HoodieValidationException extends HoodieException), the no-op validateFilesInPartition path (all sub-flags default false), and the static-nested-class serialization reasoning all check out. No correctness issues found. A few style/readability suggestions in the inline comments. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. One small readability nit on sneakyThrow; otherwise the tests are well-structured and the explanatory comments are genuinely useful.
cc @yihua
| } | ||
|
|
||
| @SuppressWarnings("unchecked") | ||
| private static <T extends Throwable> void sneakyThrow(Throwable e) throws T { |
There was a problem hiding this comment.
🤖 nit: could you add a brief Javadoc on sneakyThrow explaining the constraint that drives it? Something like /** SparkException is checked; overrides cannot widen the throws clause, so this unchecked-cast rethrow is needed. */ would save the next reader a "why not just throw it?" moment.
6e38832 to
b0e4be9
Compare
|
@lokeshj1703 Can you modify the description such that it passes the Overall, LGTM, address the comments above and it's good to go. |
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the contribution! This PR adds two tests covering the catch (SparkException) branches in doMetadataTableValidation — distinguishing the SparkContext-shutdown case (wrapped as HoodieException) from the generic case (HoodieValidationException). I traced the injected SparkException through the actual control flow and confirmed both tests reach the intended branch: the overridden validateRecordIndex propagates past the inner HoodieValidationException-only catch to the outer SparkException handler, and all validateFilesInPartition flags default to false so nothing throws earlier. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.
cc @yihua
What is the purpose of this PR?
Add two tests that verify the
catch (SparkException)behaviour introduced in #18371(
ExceptionUtil.validateErrorMsgguard indoMetadataTableValidation):testDoMetadataTableValidationThrowsHoodieExceptionOnSparkContextShutdown— aSparkExceptioncontaining"cancelled because SparkContext was shut down"must be wrapped inHoodieException, notHoodieValidationException.testDoMetadataTableValidationThrowsHoodieValidationExceptionOnUnexpectedSparkFailure— any otherSparkExceptionmust still produceHoodieValidationException("Unexpected spark failure"), covering the false-branch of theExceptionUtil.validateErrorMsgcondition for Sonar branch coverage.Key implementation details
HoodieMetadataTableValidatorwould implicitly captureTestHoodieMetadataTableValidator.this, which is notSerializable. That causes Spark'sparallelize().map()to throw"Task not serializable", which the outercatchconverts toHoodieValidationExceptionrather than the intendedHoodieException.validateLatestFileSlices/validateAllFileGroupsleftfalse: enabling them invokesvalidateFilesInPartitioninside the Spark map job, which can throwHoodieValidationExceptionbeforevalidateRecordIndexis ever reached, masking the exception the test intends to assert.Does this PR introduce any user-facing changes?
No — test-only change.