Name and Version
- llama.cpp version: b9935
- Backend: OpenVINO (NPU)
- OS: Windows 11
- Model: Qwen2.5-1.5B-Instruct-Q4_K_M
- Env vars:
- GGML_OPENVINO_DEVICE=NPU
- GGML_OPENVINO_STATEFUL_EXECUTION=0
Operating systems
No response
Which llama.cpp modules do you know to be affected?
No response
Command line
Problem description & steps to reproduce
Describe the bug
When running llama-server with OpenVINO backend on NPU (GGML_OPENVINO_DEVICE=NPU, GGML_OPENVINO_STATEFUL_EXECUTION=0), the
first model load always fails with an ov::Exception about inconsistent argument shapes in the RoPE Multiply node. The
process crashes (exit code 0xC0000005). Running the same command a second time succeeds — the model loads and works
correctly.
Additionally, on the first run, Flash Attention is disabled due to a device mismatch warning.
Environment
First run (fails)
W resolve_fused_ops: layer 0 is assigned to device CPU but Flash Attention is assigned to device OPENVINO0 (usually due to
missing support)
W resolve_fused_ops: Flash Attention not supported, set to disabled
E GGML OpenVINO backend ov::Exception: Check 'TRShape::broadcast_merge_into(output_shape, input_shapes[1], autob)' failed:
While validating node 'opset1::Multiply Multiply_1900 (opset1::Split Split_1899[1]:f32[1,2,12,64], opset1::Multiply
Multiply_1897[0]:f32[1,256,1,64]) -> (dynamic[...])':
Argument shapes are inconsistent.
E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
E process_ubatch: failed to compute graph, compute status: -1
E llama_decode: failed to decode, ret = -3
Process exits with status -1073741819 (0xC0000005).
Second run (succeeds)
Same command, same env vars. Model loads normally with progress updates, Flash Attention warning does not appear, and
inference works:
I srv load_model: model loaded
prompt eval time = 1657.21 ms / 30 tokens (18.10 tokens per second)
eval time = 1011.12 ms / 10 tokens (9.89 tokens per second)
Root cause analysis
Problem 1: RoPE shape mismatch (crash)
In NPU static mode, ov_graph_compute_static compiles two models:
- Prefill model: inp_pos shape [1,1,1,256] (prefill_chunk_size=256)
- Decode model: inp_pos shape [1,1,1,1]
In the RoPE TYPE_NORMAL (Flux) path (ggml-openvino/openvino/op/rope.cpp:96-157):
- data_node is reshaped to [1, -1, n_heads, head_size] with -1 as dynamic seq dim
- expand_cos_sin uses special_zero=true Reshape, which passes the seq dim through dynamically from the cos/sin tables →
results in [1, 256, 1, 64] for prefill
- NPUW FOLD optimization (NPUW_FOLD=YES) parametrizes repeated per-layer subgraphs. During folding, the Split output's seq
dim is resolved to 2 (likely from the initial graph's token count), while the cos/sin retains the full 256-dim → shapes
[1,2,12,64] vs [1,256,1,64] become incompatible at the Multiply node
The second run succeeds because the NPUW driver cache (NPUW_CACHE_DIR) has the compiled model from the first attempt, and
the cache-hit path bypasses the problematic folding step.
Problem 2: Flash Attention device mismatch (feature degradation)
In resolve_fused_ops (src/llama-context.cpp:473-540):
- model.dev_layer(0) returns CPU (layer 0's weight buffer type)
- ggml_backend_sched_get_tensor_backend assigns the Flash Attention op to OPENVINO0
- Device mismatch → Flash Attention is disabled
On the second run the scheduler may have corrected the assignment, or the warning simply doesn't trigger because the graph
is cached.
Possible fixes
- RoPE shape: In expand_cos_sin, use ShapeOf(data_node) to dynamically derive the seq dimension instead of relying on
special_zero passthrough, ensuring the data and cos/sin paths always agree. Alternatively, pin the data seq dim to
prefill_chunk_size in the prefill model instead of using -1.
- Flash Attention device: Ensure select_buft(0) returns the OpenVINO backend's buffer type when running in NPU mode, so
model.dev_layer(0) returns OPENVINO0 instead of CPU.
Reproduction
$env:GGML_OPENVINO_STATEFUL_EXECUTION="0"
$env:GGML_OPENVINO_DEVICE="NPU"
llama-server --model <path-to-qwen2.5-1.5b-q4_k_m.gguf> --n-gpu-layers -1 --ctx-size 512 --host 127.0.0.1 --port 8090
First run: crash
Second run: works
First Bad Commit
No response
Relevant log output
Logs
PS> $env:GGML_OPENVINO_STATEFUL_EXECUTION="0"
PS> $env:GGML_OPENVINO_DEVICE="NPU"
PS> & "<llama-server-path>" `
>> --models-preset "<preset-path>" `
>> --models-max 1 `
>> --host 127.0.0.1 `
>> --port 8090 `
>> --api-key <api-key> `
>> --log-file "<log-path>" `
>> --no-webui
0.00.210.589 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.210.680 I srv init: The UI is disabled
0.00.210.681 I srv init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
0.00.213.227 I srv load_models: Loaded 0 cached model presets
0.00.214.058 I srv load_models: Loaded 11 custom model presets from <preset-path>
0.00.214.657 I srv operator (): Available models (11) (*: custom preset)
0.00.214.662 I srv operator (): * Qwen/qwen2_5-0_5b-instruct-q4_k_m
0.00.214.663 I srv operator (): * Qwen/qwen2_5-1_5b-instruct-q4_k_m
0.00.214.663 I srv operator (): * bartowski/Llama-3_2-1B-Instruct-Q4_K_M
0.00.214.663 I srv operator (): * bartowski/Qwen_Qwen3-0_6B-Q4_K_M
0.00.214.663 I srv operator (): * bartowski/Qwen_Qwen3-1_7B-Q4_K_M
0.00.214.664 I srv operator (): * boclaw/Tima-3.0-4B-GGUF-v2/Tima-3.0-4B-Q4_K_M.gguf
0.00.214.664 I srv operator (): * unsloth/DeepSeek-R1-Distill-Qwen-1_5B-Q4_K_M
0.00.214.664 I srv operator (): * unsloth/Llama-3_1-8B-Instruct-IQ4_XS
0.00.214.664 I srv operator (): * unsloth/Llama-3_2-1B-Instruct-IQ4_XS
0.00.214.665 I srv operator (): * unsloth/Qwen3-0_6B-IQ4_XS
0.00.214.665 I srv operator (): * unsloth/gemma-4-12b-it-IQ4_XS
0.00.214.783 I srv llama_server: starting server in router mode. models will be automatically loaded on-demand
0.00.223.818 I srv llama_server: listening on http://127.0.0.1:8090
0.00.223.821 W srv llama_server: NOTE: router mode is experimental
0.00.223.821 W srv llama_server: it is not recommended to use this mode in untrusted environments
0.09.610.124 I srv load: spawning server instance with name=Qwen/qwen2_5-1_5b-instruct-q4_k_m on port 59687
0.09.610.191 I srv load: spawning server instance with args:
0.09.610.191 I srv load: <llama-server-path>
0.09.610.192 I srv load: --no-context-shift
0.09.610.192 I srv load: --host
0.09.610.192 I srv load: 127.0.0.1
0.09.610.192 I srv load: --log-file
0.09.610.192 I srv load: <log-path>
0.09.610.193 I srv load: --port
0.09.610.193 I srv load: 59687
0.09.610.193 I srv load: --no-webui
0.09.610.193 I srv load: --alias
0.09.610.193 I srv load: Qwen/qwen2_5-1_5b-instruct-q4_k_m
0.09.610.193 I srv load: --ctx-size
0.09.610.193 I srv load: 512
0.09.610.194 I srv load: --no-cont-batching
0.09.610.194 I srv load: --cache-ram
0.09.610.194 I srv load: -1
0.09.610.194 I srv load: --model
0.09.610.194 I srv load: <model-path>
0.09.610.195 I srv load: --n-gpu-layers
0.09.610.195 I srv load: -1
0.09.610.195 I srv load: --parallel
0.09.610.195 I srv load: 1
[59687] 0.00.199.922 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
[59687] 0.00.199.994 I srv init: The UI is disabled
[59687] 0.00.199.994 I srv init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
[59687]
cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.0}}
[59687] 0.00.204.211 I srv load_model: loading model '<model-path>'
[59687] 0.07.707.333 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the
model. its type will be overridden
[59687]
cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.0}}
[59687]
cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":1.0}}
[59687] 0.08.074.192 W resolve_fused_ops: layer 0 is assigned to device CPU but Flash Attention is assigned to device
OPENVINO0 (usually due to missing support)
[59687] 0.08.074.197 W resolve_fused_ops: Flash Attention not supported, set to disabled
[59687] 0.12.314.642 E GGML OpenVINO backend ov::Exception: Check 'TRShape::broadcast_merge_into(output_shape,
input_shapes[1], autob)' failed at src\core\shape_inference\include\eltwise_shape_inference.hpp:28:
[59687] While validating node 'opset1::Multiply Multiply_1900 (opset1::Split Split_1899[1]:f32[1,2,12,64],
opset1::Multiply Multiply_1897[0]:f32[1,256,1,64]) -> (dynamic[...])' with friendly_name 'Multiply_1900':
[59687] Argument shapes are inconsistent.
[59687]
[59687] 0.12.315.351 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
[59687] 0.12.315.354 E process_ubatch: failed to compute graph, compute status: -1
[59687] 0.12.315.587 W decode: removing memory module entries for seq_id = 0, pos = [0, +inf)
[59687] 0.12.316.802 E llama_decode: failed to decode, ret = -3
0.32.248.326 I srv operator (): instance name=Qwen/qwen2_5-1_5b-instruct-q4_k_m exited with status -1073741819
2.27.326.951 I srv operator (): operator (): cleaning up before exit...
Second run (succeeds)
PS> & "<llama-server-path>" `
>> --models-preset "<preset-path>" `
>> --models-max 1 `
>> --host 127.0.0.1 `
>> --port 8090 `
>> --api-key <api-key> `
>> --log-file "<log-path>" `
>> --no-webui
0.00.194.209 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.194.283 I srv init: The UI is disabled
0.00.194.283 I srv init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
0.00.196.514 I srv load_models: Loaded 0 cached model presets
0.00.197.289 I srv load_models: Loaded 11 custom model presets from <preset-path>
0.00.197.632 I srv operator (): Available models (11) (*: custom preset)
0.00.197.638 I srv operator (): * Qwen/qwen2_5-0_5b-instruct-q4_k_m
0.00.197.638 I srv operator (): * Qwen/qwen2_5-1_5b-instruct-q4_k_m
0.00.197.639 I srv operator (): * bartowski/Llama-3_2-1B-Instruct-Q4_K_M
0.00.197.639 I srv operator (): * bartowski/Qwen_Qwen3-0_6B-Q4_K_M
0.00.197.639 I srv operator (): * bartowski/Qwen_Qwen3-1_7B-Q4_K_M
0.00.197.639 I srv operator (): * boclaw/Tima-3.0-4B-GGUF-v2/Tima-3.0-4B-Q4_K_M.gguf
0.00.197.640 I srv operator (): * unsloth/DeepSeek-R1-Distill-Qwen-1_5B-Q4_K_M
0.00.197.640 I srv operator (): * unsloth/Llama-3_1-8B-Instruct-IQ4_XS
0.00.197.640 I srv operator (): * unsloth/Llama-3_2-1B-Instruct-IQ4_XS
0.00.197.640 I srv operator (): * unsloth/Qwen3-0_6B-IQ4_XS
0.00.197.641 I srv operator (): * unsloth/gemma-4-12b-it-IQ4_XS
0.00.197.753 I srv llama_server: starting server in router mode. models will be automatically loaded on-demand
0.00.214.387 I srv llama_server: listening on http://127.0.0.1:8090
0.00.214.394 W srv llama_server: NOTE: router mode is experimental
0.00.214.395 W srv llama_server: it is not recommended to use this mode in untrusted environments
0.02.837.704 I srv load: spawning server instance with name=Qwen/qwen2_5-1_5b-instruct-q4_k_m on port 50098
0.02.837.748 I srv load: spawning server instance with args:
0.02.837.748 I srv load: <llama-server-path>
0.02.837.749 I srv load: --no-context-shift
0.02.837.749 I srv load: --host
0.02.837.749 I srv load: 127.0.0.1
0.02.837.749 I srv load: --log-file
0.02.837.749 I srv load: <log-path>
0.02.837.750 I srv load: --port
0.02.837.750 I srv load: 50098
0.02.837.750 I srv load: --no-webui
0.02.837.750 I srv load: --alias
0.02.837.750 I srv load: Qwen/qwen2_5-1_5b-instruct-q4_k_m
0.02.837.750 I srv load: --ctx-size
0.02.837.750 I srv load: 512
0.02.837.751 I srv load: --no-cont-batching
0.02.837.751 I srv load: --cache-ram
0.02.837.751 I srv load: -1
0.02.837.751 I srv load: --model
0.02.837.751 I srv load: <model-path>
0.02.837.751 I srv load: --n-gpu-layers
0.02.837.751 I srv load: -1
0.02.837.752 I srv load: --parallel
0.02.837.752 I srv load: 1
[50098] 0.00.191.840 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
[50098] 0.00.191.950 I srv init: The UI is disabled
[50098] 0.00.191.951 I srv init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
[50098]
cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.0}}
[50098] 0.00.195.044 I srv load_model: loading model '<model-path>'
[50098] 0.00.611.620 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the
model. its type will be overridden
[50098]
cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.0}}
[50098] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.2
903786599636078}}
[50098] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.3
1039851903915405}}
[50098] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.3
2722577452659607}}
... (loading progress continues) ...
[50098]
cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":1.0}}
[50098] 0.18.038.164 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 512, kv_unified = 'false'
[50098] 0.18.067.467 I srv llama_server: model loaded
[50098] 0.18.067.482 I srv llama_server: listening on http://127.0.0.1:50098
[50098] cmd_child_to_router:state:{"state":"ready","payload":{"id":"Qwen/qwen2_5-1_5b-instruct-q4_k_m","aliases":["Qwen/qw
en2_5-1_5b-instruct-q4_k_m"],"tags":[],"object":"model","created":1784254902,"owned_by":"llamacpp","meta":{"vocab_type":2,
"n_vocab":151936,"n_ctx":512,"n_ctx_train":32768,"n_embd":1536,"n_params":1777088000,"size":1111370240,"ftype":"Q4_K -
Medium"}}}
0.48.308.989 I srv proxy_reques: proxying request to model Qwen/qwen2_5-1_5b-instruct-q4_k_m on port 50098
[50098] 0.45.449.267 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
[50098] 0.45.449.426 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
[50098] 0.48.118.186 I slot print_timing: id 0 | task 0 | prompt eval time = 1657.21 ms / 30 tokens ( 55.24 ms
per token, 18.10 tokens per second)
[50098] 0.48.118.192 I slot print_timing: id 0 | task 0 | eval time = 1011.12 ms / 10 tokens ( 101.11 ms
per token, 9.89 tokens per second)
[50098] 0.48.118.193 I slot print_timing: id 0 | task 0 | total time = 2668.34 ms / 40 tokens
[50098] 0.48.118.203 I slot print_timing: id 0 | task 0 | graphs reused = 9
[50098] 0.48.119.700 I slot release: id 0 | task 0 | stop processing: n_tokens = 39, truncated = 0
11.51.741.991 I srv unload: stopping model instance name=Qwen/qwen2_5-1_5b-instruct-q4_k_m
11.51.742.800 I srv operator (): stopping model instance name=Qwen/qwen2_5-1_5b-instruct-q4_k_m
[50098] 11.48.870.883 I srv operator (): exit command received, exiting...
[50098] 11.48.871.442 I srv operator (): operator (): cleaning up before exit...
11.52.965.023 I srv operator (): instance name=Qwen/qwen2_5-1_5b-instruct-q4_k_m exited with status 0
11.55.273.988 I srv operator (): operator (): cleaning up before exit..
Name and Version
Operating systems
No response
Which llama.cpp modules do you know to be affected?
No response
Command line
Problem description & steps to reproduce
Describe the bug
When running llama-server with OpenVINO backend on NPU (GGML_OPENVINO_DEVICE=NPU, GGML_OPENVINO_STATEFUL_EXECUTION=0), the
first model load always fails with an ov::Exception about inconsistent argument shapes in the RoPE Multiply node. The
process crashes (exit code 0xC0000005). Running the same command a second time succeeds — the model loads and works
correctly.
Additionally, on the first run, Flash Attention is disabled due to a device mismatch warning.
Environment
First run (fails)
W resolve_fused_ops: layer 0 is assigned to device CPU but Flash Attention is assigned to device OPENVINO0 (usually due to
missing support)
W resolve_fused_ops: Flash Attention not supported, set to disabled
E GGML OpenVINO backend ov::Exception: Check 'TRShape::broadcast_merge_into(output_shape, input_shapes[1], autob)' failed:
While validating node 'opset1::Multiply Multiply_1900 (opset1::Split Split_1899[1]:f32[1,2,12,64], opset1::Multiply
Multiply_1897[0]:f32[1,256,1,64]) -> (dynamic[...])':
Argument shapes are inconsistent.
E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
E process_ubatch: failed to compute graph, compute status: -1
E llama_decode: failed to decode, ret = -3
Process exits with status -1073741819 (0xC0000005).
Second run (succeeds)
Same command, same env vars. Model loads normally with progress updates, Flash Attention warning does not appear, and
inference works:
I srv load_model: model loaded
prompt eval time = 1657.21 ms / 30 tokens (18.10 tokens per second)
eval time = 1011.12 ms / 10 tokens (9.89 tokens per second)
Root cause analysis
Problem 1: RoPE shape mismatch (crash)
In NPU static mode, ov_graph_compute_static compiles two models:
In the RoPE TYPE_NORMAL (Flux) path (ggml-openvino/openvino/op/rope.cpp:96-157):
results in [1, 256, 1, 64] for prefill
dim is resolved to 2 (likely from the initial graph's token count), while the cos/sin retains the full 256-dim → shapes
[1,2,12,64] vs [1,256,1,64] become incompatible at the Multiply node
The second run succeeds because the NPUW driver cache (NPUW_CACHE_DIR) has the compiled model from the first attempt, and
the cache-hit path bypasses the problematic folding step.
Problem 2: Flash Attention device mismatch (feature degradation)
In resolve_fused_ops (src/llama-context.cpp:473-540):
On the second run the scheduler may have corrected the assignment, or the warning simply doesn't trigger because the graph
is cached.
Possible fixes
special_zero passthrough, ensuring the data and cos/sin paths always agree. Alternatively, pin the data seq dim to
prefill_chunk_size in the prefill model instead of using -1.
model.dev_layer(0) returns OPENVINO0 instead of CPU.
Reproduction
$env:GGML_OPENVINO_STATEFUL_EXECUTION="0"
$env:GGML_OPENVINO_DEVICE="NPU"
llama-server --model <path-to-qwen2.5-1.5b-q4_k_m.gguf> --n-gpu-layers -1 --ctx-size 512 --host 127.0.0.1 --port 8090
First run: crash
Second run: works
First Bad Commit
No response
Relevant log output
Logs