Dynamically load nvcuvid library?

[traversaro]

🚀 The feature

It would be great if torchcodec dynamically loaded the nvcuvid library instead of directly linking it, or at least have an option to dynamically load the nvcuvid library instead of directly link it.

Motivation, pitch

First of all, thanks a lot for the great work on the new NVCUVID backend (#910 and following PRs). I looked into that as part of the 0.8.0, and I noticed that currently we directly link the nvcuvid library in

torchcodec/src/torchcodec/_core/CMakeLists.txt

Line 131 in 3827dfe

${NVCUVID_LIBRARY}

.

This specific choice works great in most cases, but in some corner cases can be problematic (that problably only emerge in conda-forge packaging and hence why I stumbled on them), in particular:

• On some Linux NVIDIA system without NVDEC silicon (such as Tegra/Jetson Orin), the libnvcuvid.so library is not installed as part of the system driver. So, that always meant that torchcodec's GPU capabilities were not working, but at least before 0.8.0 it was possible to use a torchcodec CUDA build to actually do CPU decoding, as libnvcuvid.so library was not required unless ffmpeg was actually requested to use _cuvid decoder. After 0.8.0 instead, for a torchcodec CUDA build, it is required for libnvcuvid.so to be present in the system, otherwise the libtorchcodec_core7.so can't be loaded at all, preventing the use of torchcodec even just for CPU usage.
• On Windows to actually link the nvcuvid.dll library, you also need the nvcuvid.lib import library, that is not installed by the NVIDIA driver as nvcuvid.dll, but it installed separately by the "Video Codec for application developers", that needs to be manually downloaded from https://developer.nvidia.com/nvidia-video-codec-sdk/download by accepting the " NVIDIA Software Developer License Agreement". This is different from Linux, there the libnvcuvid.so is the one installed in the system, but that can also be used when linking the library.

How does other projects like ffmpeg or dali deal with this? Instead of directly linking to nvcuvid library, they dynamically load nvcuvid, gracefully raising an error if the library can't be loaded, see:

• ffmpeg
  • https://github.com/FFmpeg/FFmpeg/blob/20051ed3af4c8c355b8749d7b59593fc9a56bf80/libavcodec/nvdec.c#L211
  • https://github.com/FFmpeg/nv-codec-headers/blob/876af32a202d0de83bd1d36fe74ee0f7fcf86b0d/include/ffnvcodec/dynlink_loader.h#L434 (interestingly, all the code under https://github.com/FFmpeg/nv-codec-headers is MIT licensed and so more easily reusable w.r.t. to ffmpeg itself that is LGPL)
• dali:
  • https://github.com/NVIDIA/DALI/blob/e2db4d795524dd2274dec9fbe479d2e8e50c6f23/dali/operators/video/dynlink_nvcuvid/dynlink_nvcuvid.cc

If there is interested for something like that I probably can also contribute in that direction, even if I can't provide a definite timeframe for this.

[NicolasHug]

Thanks for the request @traversaro . I think this is reasonable, I'd just like to make sure I understand what you're suggesting. I'll summarize my understanding below.

We're already dynamically linking to nvcuvid (in contrast to statically linking to it), since we're using the SHARED CMake rule:

torchcodec/src/torchcodec/_core/CMakeLists.txt

Lines 128 to 140 in 2117716

	list(APPEND core_library_dependencies
	${CUDA_nppi_LIBRARY}
	${CUDA_nppicc_LIBRARY}
	${NVCUVID_LIBRARY}
	)
	endif()
	make_torchcodec_sublibrary(
	"${core_library_name}"
	SHARED
	"${core_sources}"
	"${core_library_dependencies}"
	)

and it's clear from ldd that our wheels are dynamically linked too:

~/dev/torchcodec (main*) » ldd /home/nicolashug/.opt/miniconda3/envs/zzz/lib/python3.13/site-packages/torchcodec/libtorchcodec_core4.so | grep cuvid
	libnvcuvid.so.1 => /lib64/libnvcuvid.so.1 (0x00007faee0800000)

But now it means that nvcuvid.so must be on the users machine when they import torchodec, because that's when we dlopen libtorchcodec_coreN.so, which in turn will try to dlopen nvcuvid.so.

So, IIUC, what you are suggesting @traversaro is that we remove the dependency of libtorchcodec_core4Nso on nvcuvid.so, and instead we do the dlopen(nvcuvid_lib) call ourselves.... somewhere. Am I getting this right?
And one benefit of doing it this way is that this "somewhere" can be "When the users explicitly requests the NVCUVID backend", but if they just need CPU decoding then nothing will fail.

[traversaro]

Exactly, your recap is correct.

[NicolasHug]

Thanks for confirming @traversaro . This makes sense, I think that's something we could do.

This reminded me that there are actually some NVCUVID docs about exactly this, with example of both linux and Windows: https://docs.nvidia.com/video-technologies/video-codec-sdk/13.0/nvdec-video-decoder-api-prog-guide/index.html#dynamic-loading

If there is interested for something like that I probably can also contribute in that direction, even if I can't provide a definite timeframe for this.

I'd love to review a PR, thanks for volunteering!