Skip to content

[SPARK-49442][SS][FOLLOWUP] Use monotonic clock for Kafka partition cache TTL - #57943

Open
cloud-fan wants to merge 1 commit into
apache:masterfrom
cloud-fan:SPARK-49442-followup
Open

[SPARK-49442][SS][FOLLOWUP] Use monotonic clock for Kafka partition cache TTL#57943
cloud-fan wants to merge 1 commit into
apache:masterfrom
cloud-fan:SPARK-49442-followup

Conversation

@cloud-fan

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Followup to #57351.

This PR measures the Kafka partition metadata cache TTL with System.nanoTime instead of
System.currentTimeMillis. It also represents the cached timestamp with Option[Long] rather than
using zero as a sentinel. Debug logging continues to report cache age in milliseconds.

Why are the changes needed?

TTL measurement should use a monotonic time source. Wall-clock adjustments can otherwise make a
cache entry appear older or younger than it is, causing premature refreshes or extending its
lifetime. System.nanoTime measures elapsed time without that sensitivity.

Does this PR introduce any user-facing change?

No. The cache option and its behavior under a stable clock are unchanged.

How was this patch tested?

Existing cache reuse and expiry coverage was run with Java 17:

build/sbt 'sql-kafka-0-10/testOnly org.apache.spark.sql.kafka010.KafkaOffsetReaderSuite'

All 22 tests passed. No new test was added because the change only substitutes the JVM's monotonic
elapsed-time source; reliably simulating host wall-clock changes is not practical in this suite.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)

…ache TTL

Measure the partition metadata cache TTL with System.nanoTime so wall-clock adjustments cannot incorrectly extend or expire cached metadata. This follows up apache#57351.
@cloud-fan

Copy link
Copy Markdown
Contributor Author

cc @vinodkc @HeartSaVioR

@uros-b

uros-b commented Aug 12, 2026

Copy link
Copy Markdown
Member

Thank you @cloud-fan!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants