What could be the reason for ValueError: could not broadcast input array from shape (32000,) into shape (0,) during token generation.

[qnixsynapse]

Hello. I am occasionally getting this error:

lib/python3.11/site-packages/llama_cpp/llama.py", line 765, in generate
    self.eval(tokens)
lib/python3.11/site-packages/llama_cpp/llama.py", line 501, in eval
    self.scores[self.n_tokens + offset : self.n_tokens + n_tokens, :].reshape(
ValueError: could not broadcast input array from shape (32000,) into shape (0,)

with a 13B llama2-chat model during token generation.

What could be the reason for it? This started happening recently.

[QtRoS]

I'm facing this issue too. In my case this exception happens when the same model is used to generate text multiple times, without reload. Looks like it's context grows out of control. Are there any methods to clear context?

[tk-master]

Yes, this error has come up when the model tries to generate but the context limit got full, you are supposed to handle context limits yourself.. you could for example tokenize the prompt/history to check the context size and then remove older messages if needed (not a great solution but it works, i think ooba does this).
Alternatively you can try increasing the context size.. n_ctx=4096

[qnixsynapse]

I think my context size was 4096 only. And it was probably the second message.

[tk-master]

hmm if you have a way to replicate it then let us know

[qnixsynapse]

Actually, I am unable to reproduce currently, which is weird. Maybe it got fixed.

Since 32000 is the vocab size of llama2. So I am guessing it might be due to a weird memory corruption.