Eval bug: ggml_vulkan: Device memory allocation of size N failed with ub > 4096 and c > 4096 and b > 4096

[deiteris]

Name and Version

version: 5061 (916c83b)
built with MSVC 19.38.33134.0 for x64

Operating systems

Windows

GGML backends

Vulkan

Hardware

Ryzen 7 5800H + AMD Radeon RX 6600M

Models

Any model

Problem description & steps to reproduce

When trying to run llama-server with -ub 8192 -b 8192 -c 8192, it crashes with ggml_vulkan: Device memory allocation of size 3959422976 failed. with any model I try (the allocation size differs between models), even though I have enough GPU memory after model is loaded.

I tried smaller models to exclude possible OOM (the log includes nomic-embed-text-v1.5) and I see that ~100mb of VRAM gets allocated for a model (0.9GB used), then it crashes when trying to allocate 3959422976 bytes.

When setting any of these parameters to 4096, the model loads successfully.

The same occurs with any model. Tried with Qwen2.5 3B Q8_0 and nomic-embed-text-v1.5 Q8_0.

First Bad Commit

No response

Relevant log output

.\llama-server.exe --embedding -ub 8192 -b 8192 -c 8192 --host 127.0.0.1 --port 8080 -m nomic-embed-text-v1.5.Q8_0.gguf -ngl 99
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6600M (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
build: 5061 (916c83bf) with MSVC 19.38.33134.0 for x64
system info: n_threads = 8, n_threads_batch = 8, total_threads = 16

system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 15
main: loading model
srv load_model: loading model 'nomic-embed-text-v1.5.Q8_0.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon RX 6600M) - 8176 MiB free
llama_model_loader: loaded meta data with 23 key-value pairs and 112 tensors from nomic-embed-text-v1.5.Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = nomic-bert
llama_model_loader: - kv 1: general.name str = nomic-embed-text-v1.5
llama_model_loader: - kv 2: nomic-bert.block_count u32 = 12
llama_model_loader: - kv 3: nomic-bert.context_length u32 = 2048
llama_model_loader: - kv 4: nomic-bert.embedding_length u32 = 768
llama_model_loader: - kv 5: nomic-bert.feed_forward_length u32 = 3072
llama_model_loader: - kv 6: nomic-bert.attention.head_count u32 = 12
llama_model_loader: - kv 7: nomic-bert.attention.layer_norm_epsilon f32 = 0.000000
llama_model_loader: - kv 8: general.file_type u32 = 7
llama_model_loader: - kv 9: nomic-bert.attention.causal bool = false
llama_model_loader: - kv 10: nomic-bert.pooling_type u32 = 1
llama_model_loader: - kv 11: nomic-bert.rope.freq_base f32 = 1000.000000
llama_model_loader: - kv 12: tokenizer.ggml.token_type_count u32 = 2
llama_model_loader: - kv 13: tokenizer.ggml.bos_token_id u32 = 101
llama_model_loader: - kv 14: tokenizer.ggml.eos_token_id u32 = 102
llama_model_loader: - kv 15: tokenizer.ggml.model str = bert
llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,30522] = ["[PAD]", "[unused0]", "[unused1]", "...
llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,30522] = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,30522] = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 100
llama_model_loader: - kv 20: tokenizer.ggml.seperator_token_id u32 = 102
llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 0
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 51 tensors
llama_model_loader: - type q8_0: 61 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 138.65 MiB (8.51 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 5
load: token to piece cache size = 0.2032 MB
print_info: arch = nomic-bert
print_info: vocab_only = 0
print_info: n_ctx_train = 2048
print_info: n_embd = 768
print_info: n_layer = 12
print_info: n_head = 12
print_info: n_head_kv = 12
print_info: n_rot = 64
print_info: n_swa = 0
print_info: n_swa_pattern = 1
print_info: n_embd_head_k = 64
print_info: n_embd_head_v = 64
print_info: n_gqa = 1
print_info: n_embd_k_gqa = 768
print_info: n_embd_v_gqa = 768
print_info: f_norm_eps = 1.0e-12
print_info: f_norm_rms_eps = 0.0e+00
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 3072
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 0
print_info: pooling type = 1
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 2048
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 137M
print_info: model params = 136.73 M
print_info: general.name = nomic-embed-text-v1.5
print_info: vocab type = WPM
print_info: n_vocab = 30522
print_info: n_merges = 0
print_info: BOS token = 101 '[CLS]'
print_info: EOS token = 102 '[SEP]'
print_info: UNK token = 100 '[UNK]'
print_info: SEP token = 102 '[SEP]'
print_info: PAD token = 0 '[PAD]'
print_info: MASK token = 103 '[MASK]'
print_info: LF token = 0 '[PAD]'
print_info: EOG token = 102 '[SEP]'
print_info: max token length = 21
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 12 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 13/13 layers to GPU
load_tensors: Vulkan0 model buffer size = 114.89 MiB
load_tensors: CPU_Mapped model buffer size = 23.76 MiB
......................................................
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 8192
llama_context: n_ctx_per_seq = 8192
llama_context: n_batch = 8192
llama_context: n_ubatch = 8192
llama_context: causal_attn = 0
llama_context: flash_attn = 0
llama_context: freq_base = 1000.0
llama_context: freq_scale = 1
llama_context: n_ctx_pre_seq (8192) > n_ctx_train (2048) -- possible training context overflow
llama_context: Vulkan_Host output buffer size = 0.00 MiB
init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 12, can_shift = 1
init: Vulkan0 KV buffer size = 288.00 MiB
llama_context: KV self size = 288.00 MiB, K (f16): 144.00 MiB, V (f16): 144.00 MiB
ggml_vulkan: Device memory allocation of size 3959422976 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate Vulkan0 buffer of size 3959422976
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
common_init_from_params: failed to create context with model 'nomic-embed-text-v1.5.Q8_0.gguf'
srv load_model: failed to load model, 'nomic-embed-text-v1.5.Q8_0.gguf'
srv operator (): operator (): cleaning up before exit...

[0cc4m]

There might actually be a bug here, I don't see a reason for it to attempt a 4GB memory allocation. I'll look into it, thank you for the report.

[0cc4m]

In the meantime you could try without -ub 8192. This is an unusually high value for ubatch, normally this is 512 (default). In some cases 1024 or 2048 gives more performance, but this is not guaranteed. Maybe trying to calculate a ubatch of size 8192 at the same time leads to this large allocation.

[deiteris]

Yes, lower -ub works, so -ub 2048 -b 8192 -c 8192 loads properly. For the reference, reducing context size also works, so -ub 8192 -b 8192 -c 4096 combination works as well for example.

The reason why I was using larger ubatch is because some requests for embedding were failing with input is too large to process. increase the physical batch size error. Maybe there's a problem with Continue.dev extension, or something is wrong with batch calculation, but I'm not sure how to debug this correctly.

[0cc4m]

Interesting. @ggerganov why does the server require larger ubatch here?

[ggerganov]

Embedding models require that the entire context of the request to fit inside a single micro-batch. The reason for this requirement is that embedding models use non-causal attention. I.e. past tokens have to attend to future tokens too (not only to past tokens as in regular causal attention). So splitting the context into small batches no longer works - all tokens have to "see" each other at once.

For the reference, reducing context size also works, so -ub 8192 -b 8192 -c 4096 combination works as well for example.

This will silently reduce the batch size to 4096 to fit in the maximum context.

[deiteris]

Then another question here. Does this also mean that multiple concurrent tasks will compete for the context here, since I basically allocate context that is enough for a single batch? For example, given -ub 1024 -b 1024 -c 1024, if there will be 2 parallel embedding requests with 1024 tokens, should one of them also fail with input is too large error? Because I think I observe this behavior, but not confident if that's the reason yet.

[0cc4m]

Embedding models require that the entire context of the request to fit inside a single micro-batch. The reason for this requirement is that embedding models use non-causal attention. I.e. past tokens have to attend to future tokens too (not only to past tokens as in regular causal attention). So splitting the context into small batches no longer works - all tokens have to "see" each other at once.

Okay, thank you, then the issue is on the Vulkan side. I'll look into it.

[ggerganov]

Does this also mean that multiple concurrent tasks will compete for the context here, since I basically allocate context that is enough for a single batch?

Yes, they will compete for the context. And since these are embeddings, they will also compete for the micro-batch size.

Let's say that the max prompt of a single task is 1024 tokens and you want to be able to process a maximum of 4 tasks in parallel. You can then run:

llama-server ... -ub 4096 -b 4096 -c 4096 -np 4

Note that if you run the following, the tasks will still be processed in parallel, but only those that fit together in a batch of 1024 tokens - the rest will be processed in sequential batches:

llama-server ... -ub 1024 -b 4096 -c 4096 -np 4

(at least this is the expectation - let us know if you observe something else)

In general, when computing embeddings, you can expect the following performance curve:

• For small prompts (tens of tokens per prompt), stacking multiple tasks together will lead to significant performance improvement, potentially linear with the number of tasks.
• Increasing the prompt size, this improvement will be less and less pronounced
• At some point, depending on the compute capacity of the hardware, a task with a prompt of let's say ~2048 tokens will be enough to saturate the compute and you will stop seeing performance gains from stacking tasks in parallel
• And beyond that point (i.e. very large prompt) the performance of multiple tasks will start to degrade compared to sequential, because of the underlying KV cache implementation in llama.cpp

So my guess is that you don't want to use -np more than 2, 3 or 4 (unless you have very tiny prompts). And if the prompt of a single task alone is expected to be ~8192 tokens, then for sure use -np 1. This should be the optimal almost guaranteed.

[deiteris]

@ggerganov Ok, I did few tests and apparently there's an issue that is subject to a separate issue.

Using the following command:

llama-server ... -ub 4096 -b 4096 -c 4096 -np 4

Everything works pretty much as expected. Amount of tokens that a task slot can handle appears to be ub / np. So in this example, each slot gets a 1024 tokens window. This does seem to give a nice boost depending on the embeddings chunking strategy (my current embeddings are up to 1024 tokens), but I haven't measured precisely yet.

However, using the following command:

llama-server ... -ub 1024 -b 4096 -c 4096 -np 4

The server crashes with GGML_ASSERT((cparams.causal_attn || cparams.n_ubatch >= n_tokens_all) && "non-causal attention requires n_ubatch >= n_tokens") failed as soon as it receives the next batch of tasks:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6600M (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
build: 5080 (997b1b42) with MSVC 19.38.33134.0 for x64
system info: n_threads = 8, n_threads_batch = 8, total_threads = 16

system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 192.168.0.2, port: 8080, http threads: 15
main: loading model
srv load_model: loading model 'C:\Temp\snowflake-arctic-embed-l-v2.0-q8_0.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon RX 6600M) - 8176 MiB free
llama_model_loader: loaded meta data with 36 key-value pairs and 389 tensors from C:\Temp\snowflake-arctic-embed-l-v2.0-q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = bert
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Snowflake Arctic Embed L v2.0
llama_model_loader: - kv 3: general.version str = v2.0
llama_model_loader: - kv 4: general.basename str = snowflake-arctic-embed-l
llama_model_loader: - kv 5: general.size_label str = 567M
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.tags arr[str,8] = ["sentence-transformers", "feature-ex...
llama_model_loader: - kv 8: general.languages arr[str,74] = ["af", "ar", "az", "be", "bg", "bn", ...
llama_model_loader: - kv 9: bert.block_count u32 = 24
llama_model_loader: - kv 10: bert.context_length u32 = 8192
llama_model_loader: - kv 11: bert.embedding_length u32 = 1024
llama_model_loader: - kv 12: bert.feed_forward_length u32 = 4096
llama_model_loader: - kv 13: bert.attention.head_count u32 = 16
llama_model_loader: - kv 14: bert.attention.layer_norm_epsilon f32 = 0.000010
llama_model_loader: - kv 15: general.file_type u32 = 7
llama_model_loader: - kv 16: bert.attention.causal bool = false
llama_model_loader: - kv 17: bert.pooling_type u32 = 2
llama_model_loader: - kv 18: tokenizer.ggml.model str = t5
llama_model_loader: - kv 19: tokenizer.ggml.pre str = default
llama_model_loader: - kv 20: tokenizer.ggml.tokens arr[str,250002] = ["<s>", "<pad>", "</s>", "<unk>", ","...
llama_model_loader: - kv 21: tokenizer.ggml.scores arr[f32,250002] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 22: tokenizer.ggml.token_type arr[i32,250002] = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 23: tokenizer.ggml.add_space_prefix bool = true
llama_model_loader: - kv 24: tokenizer.ggml.token_type_count u32 = 1
llama_model_loader: - kv 25: tokenizer.ggml.remove_extra_whitespaces bool = true
llama_model_loader: - kv 26: tokenizer.ggml.precompiled_charsmap arr[u8,237539] = [0, 180, 2, 0, 0, 132, 0, 0, 0, 0, 0,...
llama_model_loader: - kv 27: tokenizer.ggml.bos_token_id u32 = 0
llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 29: tokenizer.ggml.unknown_token_id u32 = 3
llama_model_loader: - kv 30: tokenizer.ggml.seperator_token_id u32 = 2
llama_model_loader: - kv 31: tokenizer.ggml.padding_token_id u32 = 1
llama_model_loader: - kv 32: tokenizer.ggml.mask_token_id u32 = 250001
llama_model_loader: - kv 33: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 34: tokenizer.ggml.add_eos_token bool = true
llama_model_loader: - kv 35: general.quantization_version u32 = 2
llama_model_loader: - type f32: 244 tensors
llama_model_loader: - type q8_0: 145 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 598.63 MiB (8.86 BPW)
load: model vocab missing newline token, using special_pad_id instead
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 4
load: token to piece cache size = 2.1668 MB
print_info: arch = bert
print_info: vocab_only = 0
print_info: n_ctx_train = 8192
print_info: n_embd = 1024
print_info: n_layer = 24
print_info: n_head = 16
print_info: n_head_kv = 16
print_info: n_rot = 64
print_info: n_swa = 0
print_info: n_swa_pattern = 1
print_info: n_embd_head_k = 64
print_info: n_embd_head_v = 64
print_info: n_gqa = 1
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 1.0e-05
print_info: f_norm_rms_eps = 0.0e+00
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 4096
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 0
print_info: pooling type = 2
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 8192
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 335M
print_info: model params = 566.70 M
print_info: general.name = Snowflake Arctic Embed L v2.0
print_info: vocab type = UGM
print_info: n_vocab = 250002
print_info: n_merges = 0
print_info: BOS token = 0 '<s>'
print_info: EOS token = 2 '</s>'
print_info: UNK token = 3 '<unk>'
print_info: SEP token = 2 '</s>'
print_info: PAD token = 1 '<pad>'
print_info: MASK token = 250001 '[PAD250000]'
print_info: LF token = 0 '<s>'
print_info: EOG token = 2 '</s>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors: Vulkan0 model buffer size = 307.22 MiB
load_tensors: CPU_Mapped model buffer size = 291.41 MiB
......................................................
llama_context: constructing llama_context
llama_context: n_seq_max = 3
llama_context: n_ctx = 4096
llama_context: n_ctx_per_seq = 1365
llama_context: n_batch = 4096
llama_context: n_ubatch = 1024
llama_context: causal_attn = 0
llama_context: flash_attn = 0
llama_context: freq_base = 10000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (1365) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host output buffer size = 0.00 MiB
init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 24, can_shift = 1
init: Vulkan0 KV buffer size = 384.00 MiB
llama_context: KV self size = 384.00 MiB, K (f16): 192.00 MiB, V (f16): 192.00 MiB
llama_context: Vulkan0 compute buffer size = 88.01 MiB
llama_context: Vulkan_Host compute buffer size = 12.01 MiB
llama_context: graph nodes = 825
llama_context: graph splits = 4 (with bs=1024), 2 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv init: initializing slots, n_slots = 3
slot init: id 0 | task -1 | new slot n_ctx_slot = 1365
slot init: id 1 | task -1 | new slot n_ctx_slot = 1365
slot init: id 2 | task -1 | new slot n_ctx_slot = 1365
main: model loaded
main: chat template, chat_template:{%- for message in messages -%}
{{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{- '<|im_start|>assistant
' -}}
{%- endif -%}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://192.168.0.2:8080 - starting the main loop
srv update_slots: all slots are idle
slot launch_slot_: id 0 | task 0 | processing task
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 1365, n_keep = 0, n_prompt_tokens = 830
slot update_slots: id 0 | task 0 | kv cache rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 830, n_tokens = 830, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_past = 830, n_tokens = 830
slot release: id 0 | task 0 | stop processing: n_past = 830, truncated = 0
slot launch_slot_: id 1 | task 2 | processing task
slot launch_slot_: id 2 | task 3 | processing task
slot launch_slot_: id 0 | task 4 | processing task
slot update_slots: id 0 | task 4 | new prompt, n_ctx_slot = 1365, n_keep = 0, n_prompt_tokens = 255
srv log_server_r: request: POST /v1/embeddings 192.168.0.7 200
slot update_slots: id 0 | task 4 | kv cache rm [0, end)
slot update_slots: id 0 | task 4 | prompt processing progress, n_past = 255, n_tokens = 255, progress = 1.000000
slot update_slots: id 0 | task 4 | prompt done, n_past = 255, n_tokens = 255
slot update_slots: id 1 | task 2 | new prompt, n_ctx_slot = 1365, n_keep = 0, n_prompt_tokens = 852
slot update_slots: id 1 | task 2 | kv cache rm [0, end)
slot update_slots: id 1 | task 2 | prompt processing progress, n_past = 852, n_tokens = 1107, progress = 1.000000
slot update_slots: id 1 | task 2 | prompt done, n_past = 852, n_tokens = 1107
slot update_slots: id 2 | task 3 | new prompt, n_ctx_slot = 1365, n_keep = 0, n_prompt_tokens = 246
slot update_slots: id 2 | task 3 | kv cache rm [0, end)
slot update_slots: id 2 | task 3 | prompt processing progress, n_past = 246, n_tokens = 1353, progress = 1.000000
slot update_slots: id 2 | task 3 | prompt done, n_past = 246, n_tokens = 1353
C:\Sources\llama.cpp\src\llama-context.cpp:1220: GGML_ASSERT((cparams.causal_attn || cparams.n_ubatch >= n_tokens_all) && "non-causal attention requires n_ubatch >= n_tokens") failed

[ggerganov]

Yup, it shouldn't crash - something to fix. For now, you can workaround using:

# note that the batch size is also 1024 now
llama-server ... -ub 1024 -b 1024 -c 4096 -np 4

It should do what the other parameters are supposed to do. Can you confirm?

[deiteris]

Yeah, by setting -b 1024 it seems to work as expected.

[0cc4m]

I don't know the specific reason for this, but this embedding model has a very large kv buffer, which also grows exponentially with context size. So 8192 just barely fits into the maximum allocation that Vulkan supports (4GB), but since you are using AMD on Windows, that maximum is just 2GB (not sure why AMD does that). On Linux it should work on your hardware.

There is no bug on the Vulkan side. The feature that would alleviate this is Flash Attention, but that is (in the Vulkan backend) only available on beta Nvidia drivers for now.