[Bug] run nccl_api_test failed

[yizhang2077]

I run nccl_api_test in apps and have errors, but when I link nccl_api_test with standard nccl lib then I can pass the test.
link with mscclpp_nccl and run nccl_api_test

link with standard nccl and run nccl_api_test

Do you know what the problom is? Thanks

[chhwang]

mscclpp_nccl currently supports only 8-GPU collectives. We will improve in a future version.

[dromniscience]

I ran into the same issue with v0.6.0's nccl_api_test on 4 GPUs. I confirmed that in v0.6.0, the code called ncclAllreduceFallback(), regardless of the GPU number, because all the provided execution plans are in place; however, nccl_api_test is not in place. Ultimately, nccl_api_test calls the kernel function allreduce8(), and the error is thrown there. I believe allreduce8() should work correctly for single-node communication with any number of GPUs. Its code must be buggy somewhere.

[Binyang2014]

It's not related to kernel, we just hard coding to let nranksPerNode = 8. If you want to try other scale, pls use branch binyli/nccl, and change the NRANKS_PER_NODE = 8; NPEERS = 7 manually at

mscclpp/apps/nccl/src/common.hpp

Lines 20 to 21 in 37f33d2

	constexpr int NRANKS_PER_NODE = 8;
	constexpr int NPEERS = 7;

.

At least allreduce should work