Discuss `spark.comet.enabled` parameter behavior

[comphead]

What is the problem the feature request solves?

The Rust log line

INFO core/src/lib.rs: Comet native library version <version> initialized

is emitted even when Comet is disabled at the session level (e.g. spark.comet.enabled=false) or when the plugin bails out on the off-heap gate. The message is misleading because it implies Comet is running
when in fact no plan is being accelerated.

scala> spark.conf.get("spark.comet.enabled")
res0: String = false

scala> spark.read.parquet(s3://dataset/date=2026-07-21/hour=10/").limit(1).explain()
== Physical Plan ==                                                             
CollectLimit 1
+- *(1) ColumnarToRow
   +- FileScan parquet

scala> :quit
26/08/05 15:01:17 INFO core/src/lib.rs: Comet native library version 1.0.0-apple initialized <<<<< should not be here

Steps to reproduce

1. Register the Comet plugin: --conf spark.plugins=org.apache.spark.CometPlugin.
2. Disable Comet at the session level: --conf spark.comet.enabled=false (or leave spark.memory.offHeap.enabled=false and spark.comet.exec.onHeap.enabled=false so the plugin's off-heap check fails).
3. Start a Spark shell / submit a job.
4. Observe the driver / executor logs still contain INFO core/src/lib.rs: Comet native library version ... initialized.

[comphead]

This initially seemed like a straightforward issue, but it quickly turned into a rabbit hole. Before fixing it, we first need to answer a more fundamental question:

Should spark.comet.enabled be treated as a static configuration that is evaluated once at application startup and is not expected to change afterward?

Or should it be treated as a runtime configuration, similar to spark.sql.adaptive.enabled, allowing Comet to be enabled or disabled on a per-query basis? This is the current behavior.

If it is a runtime configuration, we need to ensure that the Comet code is preloaded even when the application starts with Comet disabled. However, this means users may still encounter Comet-related log messages or exceptions despite opting out. To completely disable Comet, users would also need to remove org.apache.spark.CometPlugin from spark.plugins, which is not necessarily obvious.

If it is a static configuration, Comet can be loaded only when enabled during startup, avoiding unnecessary initialization. This approach is simpler and avoids confusing messages, but it removes the flexibility of enabling Comet on a per-query basis.

For example, when an application starts with:

spark.comet.enabled=false
spark.plugins=org.apache.spark.CometPlugin

Spark still loads the plugin, triggering static initialization such as:

static {
  try {
    load();
  } catch (Throwable th) {
    LOG.warn("Failed to load comet library", th);
    // logging may not be initialized yet, so also write to stderr
    System.err.println("Failed to load comet library: " + th.getMessage());
    loadErr = th;
  }
}

As a result, Comet initialization runs regardless of the value of spark.comet.enabled.

[comphead]

@andygrove @mbutrovich @parthchandra WDYT?

[comphead]

Maybe we just need to document current behavior, saying user can occasionally see Comet related messages/exceptions even if they opted out, and to remove Comet completely they expected to remove Comet from plugins

[andygrove]

Pasting from slack conversation:

Seems like you are asking for two separate configs. A static load comet config and a runtime enable comet config.