Skip to content

[MINOR] fix(test): cover SparkContext shutdown path in HoodieMetadataTableValidator - #19349

Open
lokeshj1703 wants to merge 1 commit into
apache:masterfrom
lokeshj1703:eng-38734-mdt-validator-test-fixes
Open

[MINOR] fix(test): cover SparkContext shutdown path in HoodieMetadataTableValidator#19349
lokeshj1703 wants to merge 1 commit into
apache:masterfrom
lokeshj1703:eng-38734-mdt-validator-test-fixes

Conversation

@lokeshj1703

Copy link
Copy Markdown
Collaborator

What is the purpose of this PR?

Add two tests that verify the catch (SparkException) behaviour introduced in #18371
(ExceptionUtil.validateErrorMsg guard in doMetadataTableValidation):

  1. testDoMetadataTableValidationThrowsHoodieExceptionOnSparkContextShutdown — a SparkException containing "cancelled because SparkContext was shut down" must be wrapped in HoodieException, not HoodieValidationException.

  2. testDoMetadataTableValidationThrowsHoodieValidationExceptionOnUnexpectedSparkFailure — any other SparkException must still produce HoodieValidationException("Unexpected spark failure"), covering the false-branch of the ExceptionUtil.validateErrorMsg condition for Sonar branch coverage.

Key implementation details

  • Static nested classes instead of anonymous classes: an anonymous subclass of HoodieMetadataTableValidator would implicitly capture TestHoodieMetadataTableValidator.this, which is not Serializable. That causes Spark's parallelize().map() to throw "Task not serializable", which the outer catch converts to HoodieValidationException rather than the intended HoodieException.
  • validateLatestFileSlices / validateAllFileGroups left false: enabling them invokes validateFilesInPartition inside the Spark map job, which can throw HoodieValidationException before validateRecordIndex is ever reached, masking the exception the test intends to assert.

Does this PR introduce any user-facing changes?

No — test-only change.

@github-actions github-actions Bot added the size:M PR with lines of changes in (100, 300] label Jul 22, 2026
@lokeshj1703
lokeshj1703 force-pushed the eng-38734-mdt-validator-test-fixes branch from 9c10e6c to 6e38832 Compare July 22, 2026 13:13

@hudi-agent hudi-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the contribution! This PR adds two tests covering the catch (SparkException) branches in HoodieMetadataTableValidator.doMetadataTableValidation — one asserting a SparkContext-shutdown message is wrapped in HoodieException, and one asserting other Spark failures still yield HoodieValidationException("Unexpected spark failure"). I traced both tests against the production code: the exception propagation, the assertThrows/assertFalse distinction (relying on HoodieValidationException extends HoodieException), the no-op validateFilesInPartition path (all sub-flags default false), and the static-nested-class serialization reasoning all check out. No correctness issues found. A few style/readability suggestions in the inline comments. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. One small readability nit on sneakyThrow; otherwise the tests are well-structured and the explanatory comments are genuinely useful.

cc @yihua

}

@SuppressWarnings("unchecked")
private static <T extends Throwable> void sneakyThrow(Throwable e) throws T {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 nit: could you add a brief Javadoc on sneakyThrow explaining the constraint that drives it? Something like /** SparkException is checked; overrides cannot widen the throws clause, so this unchecked-cast rethrow is needed. */ would save the next reader a "why not just throw it?" moment.

⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lokeshj1703
lokeshj1703 force-pushed the eng-38734-mdt-validator-test-fixes branch from 6e38832 to b0e4be9 Compare July 24, 2026 07:31
@hudi-bot

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@voonhous

voonhous commented Jul 24, 2026

Copy link
Copy Markdown
Member

@lokeshj1703 Can you modify the description such that it passes the PR Compliance / validate-pr validation? Thanks.

Overall, LGTM, address the comments above and it's good to go.

@hudi-agent hudi-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the contribution! This PR adds two tests covering the catch (SparkException) branches in doMetadataTableValidation — distinguishing the SparkContext-shutdown case (wrapped as HoodieException) from the generic case (HoodieValidationException). I traced the injected SparkException through the actual control flow and confirmed both tests reach the intended branch: the overridden validateRecordIndex propagates past the inner HoodieValidationException-only catch to the outer SparkException handler, and all validateFilesInPartition flags default to false so nothing throws earlier. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.

cc @yihua

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M PR with lines of changes in (100, 300]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants