Is your feature request related to a problem? Please describe.
Would it be possible to enable multi-GPU streamed UMAP calculations?
Some of the existing algorithms for CSR-based cuML don't work with float 64 (only float32) and as single cell datasets get bigger, the nearest neighbor graphs surpass float32 limitations and don't always fit in smaller scale GPUs.
Describe the solution you'd like
Nearest neighbor graphs would be streaming from CSR host into multiple GPUs, with regional estimations.
This would enable the nearest neighbor graphs to stay within the float32 limits, and avoid the recurrent float64 error in CuPy CSR matrices.
It would also allow GPU-accelerated UMAP calculations to be done on massive datasets via smaller GPU clusters, like L40S or A10Gs.
Is there a CPU based implementation
I don't believe there is a CPU-based implementation?
but there is an option available within Nvidia's existing tech stack
https://developer.nvidia.com/blog/even-faster-and-more-scalable-umap-on-the-gpu-with-rapids-cuml/
Is your feature request related to a problem? Please describe.
Would it be possible to enable multi-GPU streamed UMAP calculations?
Some of the existing algorithms for CSR-based cuML don't work with float 64 (only float32) and as single cell datasets get bigger, the nearest neighbor graphs surpass float32 limitations and don't always fit in smaller scale GPUs.
Describe the solution you'd like
Nearest neighbor graphs would be streaming from CSR host into multiple GPUs, with regional estimations.
This would enable the nearest neighbor graphs to stay within the float32 limits, and avoid the recurrent float64 error in CuPy CSR matrices.
It would also allow GPU-accelerated UMAP calculations to be done on massive datasets via smaller GPU clusters, like L40S or A10Gs.
Is there a CPU based implementation
I don't believe there is a CPU-based implementation?
but there is an option available within Nvidia's existing tech stack
https://developer.nvidia.com/blog/even-faster-and-more-scalable-umap-on-the-gpu-with-rapids-cuml/