Is the server can be run with multiple states?

[alexeyche]

From the code I see that both self.eval_tokens and self.eval_logits are shared and basically clash, when two separate requests will come. Am I reading it correctly, it can't be used with anything bigger than 1 global chat state, meaning having basically just one user?

[abetlen]

@alexeyche that's correct that the object can only process a single request at a time. llama.cpp doesn't yet support batching requests so there's no real way to make this possible until that happens. The alternative "solution" that I'm working on is to allow users to load multiple models at the same time, but this will take twice the amount of RAM and likely be quite slow.

[alexeyche]

Got it! Thank you!

[dillfrescott]

I'd love to see the ability for simultaneous requests!