Context
Downstream libraries (e.g. transformers, via integrations/hub_kernels.py) would like to identify which calls to the Hub originate from them, so we can understand what is driving kernel usage.
get_kernel() / install_kernel() already accept a user_agent (str | dict) that is forwarded to snapshot_download() (see kernels/src/kernels/load.py, install.py, hf_hub.py). However, the layer-repo path (LayerRepository + kernelize(), kernels/src/kernels/layer/) has no way to pass one — and that's the path transformers mostly uses (layer repo definitions), so downloads triggered through kernelize are currently unattributable.
Ask
- Allow wiring a
user_agent through LayerRepository (and/or kernelize()) down to the Hub download calls, consistent with get_kernel().
- Keep it strictly optional / opt-in — telemetry has caused pushback before.
HF_HUB_DISABLE_TELEMETRY already covers the huggingface_hub side, so respecting that is enough on our end.
Timing
Preferably land this after the kernel-to-kernel dependency work (kernels-as-dependencies, e.g. einops, CuTeDSL) merges, since how these arguments are passed around will change with it.
cc @danieldk
From a Slack discussion between Sayak Paul, Anton and Daniël de Kok.
Requested by Sayak Paul - Slack thread - Agent trace
Context
Downstream libraries (e.g.
transformers, viaintegrations/hub_kernels.py) would like to identify which calls to the Hub originate from them, so we can understand what is driving kernel usage.get_kernel()/install_kernel()already accept auser_agent(str | dict) that is forwarded tosnapshot_download()(seekernels/src/kernels/load.py,install.py,hf_hub.py). However, the layer-repo path (LayerRepository+kernelize(),kernels/src/kernels/layer/) has no way to pass one — and that's the pathtransformersmostly uses (layer repo definitions), so downloads triggered throughkernelizeare currently unattributable.Ask
user_agentthroughLayerRepository(and/orkernelize()) down to the Hub download calls, consistent withget_kernel().HF_HUB_DISABLE_TELEMETRYalready covers thehuggingface_hubside, so respecting that is enough on our end.Timing
Preferably land this after the kernel-to-kernel dependency work (kernels-as-dependencies, e.g.
einops, CuTeDSL) merges, since how these arguments are passed around will change with it.cc @danieldk
From a Slack discussion between Sayak Paul, Anton and Daniël de Kok.
Requested by Sayak Paul - Slack thread - Agent trace