Our system initializes a global batch that handles multiple sequences concurrently. Each sequence/task is assigned its own n_seq_id.
When two tasks are being decoded, if at some point we only need to run decoding for one of them (i.e., only one task's batch is active), the presence of the other task's KV cache data still affects the performance of the current decode step. In other words, the decoding speed is slowed down because the GPU/memory still has to process or account for the inactive task's KV data.
Is there any way to avoid or mitigate this overhead? Are there known solutions or workarounds in llama.cpp to handle such cases more efficiently?
Thanks.
Our system initializes a global batch that handles multiple sequences concurrently. Each sequence/task is assigned its own n_seq_id.
When two tasks are being decoded, if at some point we only need to run decoding for one of them (i.e., only one task's batch is active), the presence of the other task's KV cache data still affects the performance of the current decode step. In other words, the decoding speed is slowed down because the GPU/memory still has to process or account for the inactive task's KV data.
Is there any way to avoid or mitigate this overhead? Are there known solutions or workarounds in llama.cpp to handle such cases more efficiently?
Thanks.