guide : using the new WebUI of llama.cpp

[ggerganov]

Overview

This guide highlights the key features of the new SvelteKit-based WebUI of llama.cpp.

The new WebUI in combination with the advanced backend capabilities of the llama-server delivers the ultimate local AI chat experience. A few characteristics that set this project ahead of the alternatives:

• Free, open-source and community-driven
• Excellent performance on all hardware
• Advanced context and prefix caching
• Parallel and remote user support
• Extremely lightweight and memory efficient
• Vibrant and creative community
• 100% privacy

Getting started

1. Get llama.cpp: Install | Download | Build

2. Start the llama-server tool:

   # sample server running gpt-oss-20b at http://127.0.0.1:8033 llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja -c 0 --host 127.0.0.1 --port 8033

3. Open and start using the WebUI in your browser:

Tip

For a simple, GUI-based setup of llama.cpp on Mac, try the new LlamaBarn application

Features

The new WebUI is packed with many useful features to enhance your local AI experience. Following are a few examples.

Text document processing

Add multiple text files from disk or from the clipboard to the context of your conversation:

PDF document processing

Attach one or multiple PDFs to your conversation. By default, the contents of the PDFs will be converted to RAW text, excluding any visuals.

Optionally, the WebUI can process the PDFs as images when the AI model supports it.

Image inputs

When the selected AI model has vision input capabilities, the WebUI allows you to insert images into your conversation:

Images can be inserted in addition to a textual context:

Conversation branching

Branch from previous points of the conversation by editing or regenerating a message:

webui-edits-0-thumb-small.mp4

Parallel conversations

Run multiple chat conversations at the same time:

webui-parallel-0-thumb-small.mp4

Parallel image processing is also supported:

webui-parallel-1-thumb-small.mp4

Override default sampling parameters

Start the llama-server using a set of default sampling parameters:

# set the default Top-K to be 5 and the default Temperature to be 0.80 llama-server -hf ggml-org/gpt-oss-120b-GGUF --jinja -c 0 --port 8033 --alias gpt-oss-120b --top-k 5 --temp 0.80

These parameters will now become the default values in the WebUI settings:

webui-parameters-0-thumb-small.mp4

More info: #16515

Render math expressions

The WebUI can render mathematical expressions:

Input via URL parameters

The WebUI supports passing input through the URL parameters:

webui-url-input-0-thumb-small.mp4

HTML/JS preview

The WebUI supports inline rendering of generated HTML/JS code:

webui-js-0-thumb-small.mp4

More info: #16757

Constrained generation

Specify a custom JSON schema to constrain the generated output to a specific format. As an example, here is generic invoice data extraction from multiple documents:

webui-constrained-0-thumb-small.mp4

Import/Export

Use the Import/Export options to manage your private conversations directly through the WebUI:

Efficient SSM context management

The context management and prefix caching of State Space Models (SSMs, e.g. Mamba) can be tricky. llama-server solves this problem efficiently for one or multiple users with minimum reprocessing.

Here is an example of context branching using a hybrid LLM:

webui-ssm-0-thumb-small.mp4

Mobile compatibility

The new WebUI is mobile friendly:

Sample commands

A few llama-server commands used for the examples above:

# lightweight, gpt-oss-20b llama-server --jinja -c 0 --port 8033 -hf ggml-org/gpt-oss-20b-GGUF --alias "gpt-oss-20b"# text-only, gpt-oss-120b with greedy sampling by default llama-server --jinja -c 0 --port 8033 -hf ggml-org/gpt-oss-120b-GGUF --alias "gpt-oss-120b" --top-k 1 # vision-enabled, Qwen3 VL 30B A3B, accessible from the local network llama-server --jinja -c 0 --port 8033 -hf ggml-org/Qwen3-VL-30B-A3B-Instruct-Q8_0-GGUF --alias "Qwen3 VL 30B A3B" --host 192.168.100.3 # hybrid, Granite 4.0 H Small with 1 million tokens context llama-server --jinja -c 0 --port 8033 -hf ggml-org/granite-4.0-h-small-Q8_0-GGUF --alias "Granite 4.0 Hybrid Small"

Acknowledgements

• Aleksander Grygier for leading the development
• ServeurpersoCom for the valuable contributions
• Hugging Face for the general support

[ggerganov]

Does anyone have a neat example to share for constrained output using the custom JSON option of the WebUI? Something that would be suitable for demonstration purposes.

cc @ServeurpersoCom@tarruda@aldehir

[ServeurpersoCom]

Does anyone have a neat example to share for constrained output using the custom JSON option of the WebUI? Something that would be suitable for demonstration purposes.

cc @ServeurpersoCom@tarruda@aldehir

Not sure if this is exactly what you're looking for, but I've been using the Custom JSON option mainly for tool-call schemas.
Here's a simple example that works nicely for demonstrating constrained output:

{"tools": [{"type": "function", "function":{"name": "simple_addition_tool", "description": "A dummy calculator tool used for testing multi-argument tool call streaming.", "parameters":{"type": "object", "properties":{"a":{"type": "number", "description": "The first number to add." }, "b":{"type": "number", "description": "The second number to add." } }, "required": ["a", "b"] } } } ] } 

The model can then be prompted to call this tool by generating:

{"name": "simple_addition_tool", "arguments":{"a": 3, "b": 5 } } 

I'm currently waiting for the merge of my https://github.com/ServeurpersoCom/llama.cpp/tree/harmony-toolcall-debug-option branch : once that's in, we'll be able to inspect the model's JSON tool-call responses directly in the WebUI, which makes this kind of constrained example perfect for demos.

Another demo of toolcall in a local pod with a custom MCP client proxy :D
https://github.com/user-attachments/assets/bbbdeab3-97e8-4f7c-b314-6fd272caece6
There’s huge potential with the new Svelte UI Alek’s architecture makes it possible to plug advanced features easily, like an integrated MCP client for seamless local testing.

[linuxmagic-mp]

It's probably not what you are thinking of .. but I did a similar thing to @ServeurpersoCom by putting a dummy tool call in there, while debugging custom jinja template use for Mistral.. (who ever moved common_chat_params_init_mistral_nemo detection after // Plain handler (no tools), wish he would have commented why in common_chat_templates_apply_jinja()) however, wasn't as lucky as Pascal, with the default mistral_nemo template.. JSON parsing error.. Look forward to toolcall-debug-option, might have saved me a lot of time. Need to update the prompt afterall. Yeah, the Svelte will be fun, as trying to add some logic in the UI for users to choose to use-memory, with proper warning of the implications.. I go look at that branch now..

[tetrahedronix]

Hi! Can the following communication pipeline (Frontend <-> Go microservice <-> llama-server <-> Tool) work as a workaround?

• The frontend (served by Go microservice) sends user prompts to Go microservice.
• The Go microservice forwards these prompts to the llama-server API (likely using the OpenAI-compatible /v1/chat/completions endpoint, which simplifies tool-call handling).
• When the Go microservice receives a response from llama-server, it parses it.
• If the response contains tool calls, the Go microservice intercepts them, runs the appropriate logic (by invoking external tools) and then potentially re-injects the tool-execution result into a new request to llama-server to continue the conversation.
• The final response (or streamed tokens) is then sent from the Go microservice to the client’s web-ui frontend.

[ServeurpersoCom]

I tried this one :

Inside Developer / Custom JSON :

{"json_schema":{"type": "object", "properties":{"sentiment":{"type": "string", "enum": ["positive", "neutral", "negative"] }, "confidence":{"type": "number", "minimum": 0, "maximum": 1 }, "summary":{"type": "string", "maxLength": 200 } }, "required": ["sentiment", "confidence", "summary"] } } 

Prompt "you feel good ?"

Model answer (on SvelteUI) :

{"sentiment": "positive", "confidence": 0.95, "summary": "The user is asking about my wellbeing, and I'm responding in a positive way with a smiley face. The sentiment is clearly positive with high confidence." } 

[tarruda]

Not sure if this is a neat example, but something easy you can do with vision LLMs is extract data from images in a structured way.

Add this to Developer/Custom JSON

{"json_schema":{"$defs":{"Address":{"properties":{"street":{"title": "Street", "type": "string" }, "city":{"title": "City", "type": "string" }, "state":{"title": "State", "type": "string" }, "zip_code":{"title": "Zip Code", "type": "string" } }, "required": [ "street", "city", "state", "zip_code" ], "title": "Address", "type": "object" }, "BillTo":{"properties":{"company_name":{"title": "Company Name", "type": "string" }, "address":{"$ref": "#/$defs/Address" }, "attention":{"title": "Attention", "type": "string" } }, "required": [ "company_name", "address", "attention" ], "title": "BillTo", "type": "object" }, "Company":{"properties":{"name":{"title": "Name", "type": "string" }, "address":{"$ref": "#/$defs/Address" }, "phone":{"title": "Phone", "type": "string" }, "email":{"title": "Email", "type": "string" } }, "required": [ "name", "address", "phone", "email" ], "title": "Company", "type": "object" }, "InvoiceLine":{"properties":{"description":{"title": "Description", "type": "string" }, "quantity":{"title": "Quantity", "type": "integer" }, "rate":{"anyOf": [{"type": "number" },{"type": "string" } ], "title": "Rate" }, "amount":{"anyOf": [{"type": "number" },{"type": "string" } ], "title": "Amount" } }, "required": [ "description", "quantity", "rate", "amount" ], "title": "InvoiceLine", "type": "object" }, "PaymentMethods":{"properties":{"bank_account":{"anyOf": [{"type": "string" },{"type": "null" } ], "default": null, "title": "Bank Account" }, "routing_number":{"anyOf": [{"type": "string" },{"type": "null" } ], "default": null, "title": "Routing Number" }, "check_payable_to":{"anyOf": [{"type": "string" },{"type": "null" } ], "default": null, "title": "Check Payable To" } }, "title": "PaymentMethods", "type": "object" } }, "properties":{"invoice_number":{"title": "Invoice Number", "type": "string" }, "invoice_date":{"format": "date", "title": "Invoice Date", "type": "string" }, "due_date":{"format": "date", "title": "Due Date", "type": "string" }, "company":{"$ref": "#/$defs/Company" }, "bill_to":{"$ref": "#/$defs/BillTo" }, "lines":{"items":{"$ref": "#/$defs/InvoiceLine" }, "title": "Lines", "type": "array" }, "subtotal":{"anyOf": [{"type": "number" },{"type": "string" } ], "title": "Subtotal" }, "tax_rate":{"anyOf": [{"type": "number" },{"type": "string" } ], "title": "Tax Rate" }, "tax_amount":{"anyOf": [{"type": "number" },{"type": "string" } ], "title": "Tax Amount" }, "total":{"anyOf": [{"type": "number" },{"type": "string" } ], "title": "Total" }, "payment_terms":{"title": "Payment Terms", "type": "string" }, "payment_methods":{"$ref": "#/$defs/PaymentMethods" }, "notes":{"anyOf": [{"type": "string" },{"type": "null" } ], "default": null, "title": "Notes" } }, "required": [ "invoice_number", "invoice_date", "due_date", "company", "bill_to", "lines", "subtotal", "tax_rate", "tax_amount", "total", "payment_terms", "payment_methods" ], "title": "Invoice", "type": "object" } }

and with a model that supports vision (Qwen3-VL-8B should work), paste this image:

And it will just output the invoice data without requiring any instructions:

{"invoice_number": "INV-2024-0847", "invoice_date": "2025-07-29", "due_date": "2025-08-28", "company":{"name": "Acme Corporation", "address":{"street": "123 Business Street", "city": "New York", "state": "NY", "zip_code": "10001" }, "phone": "(555) 123-4567", "email": "[email protected]" }, "bill_to":{"company_name": "Tech Solutions Inc.", "address":{"street": "456 Innovation Drive", "city": "San Francisco", "state": "CA", "zip_code": "94105" }, "attention": "John Smith" }, "lines": [{"description": "Web Development Services", "quantity": 40, "rate": 150.00, "amount": 6000.00 },{"description": "UI/UX Design", "quantity": 20, "rate": 125.00, "amount": 2500.00 },{"description": "Database Setup", "quantity": 8, "rate": 100.00, "amount": 800.00 },{"description": "Monthly Hosting", "quantity": 1, "rate": 250.00, "amount": 250.00 } ], "subtotal": 9550.00, "tax_rate": 8.5, "tax_amount": 811.75, "total": 10361.75, "payment_terms": "Net 30 days. 1.5% late fee per month on overdue balances.", "payment_methods":{"bank_account": "Account #123456789, Routing #987654321", "check_payable_to": "Acme Corporation" }, "notes": "Thank you for your business!" }

One problem with this is that the output is not wrapped in json fenced markdown blocks so you get no syntax highlighting.

This could be improved if the web UI had native support for passing a JSON schema and when enabled displayed the output in a specialized JSON viewer, such as this one

[ServeurpersoCom]

Good point about the JSON output rendering !

In the new Svelte UI, we could detect fenced code blocks with the language tag "json" (like i make for html+js preview) and render them with a built-in JSON viewer component (like the one you linked). That would make structured outputs from the Custom JSON schema much clearer, especially for large responses or nested data.
And doing it only for JSON code blocks would be totally safe: no risk of regression in the existing AST pipeline.

[ggerganov]

@tarruda Thanks, this seems suitable. I'll try to add a short video with this example later today.

[ggerganov]

Added the "Constrained generation" example to the post 👍

[Goldenkoron]

I love the look of this. Could you add a "Continue Assistant Response" kind of button? Helps to steer the AI toward a specific formatting you want at the beginning of a conversation if you could edit its response then have it continue output.

[allozaur]

I love the look of this. Could you add a "Continue Assistant Response" kind of button? Helps to steer the AI toward a specific formatting you want at the beginning of a conversation if you could edit its response then have it continue output.

Actually it's already WIP 😄

See #16971

[AmgadHasan]

How to enable Parallel conversations?

Do I need to use a specific param when launching the server?

[ServeurpersoCom]

Use --parallel 2 (or higher) on llama-server: it splits the context between workers, so your effective context size gets divided by that number: but you’ll get a higher total tokens/sec overall.

[ggerganov]

it splits the context between workers

If you add --kv-unified the context will not be split and instead shared among all parallel workers. This was added very recently (#16736) and it is very suitable for local usage.

[shimmyshimmer]

Congratulations guys this looks absolutely amazing!! :D

Can't wait to use it

[sciphergfx]

🚀🚀🚀

[nirfse]

Excellent work! It strikes the right balance between functionality, a simple user experience, and performance.

Admittedly, this is outside the scope of the project, but I would appreciate the option of deploying this interface in standalone mode, separate from llama.cpp, with third-party OpenAI API support.

[ServeurpersoCom]

Technically, if you redirect the /v1/chat/completions request to OpenAI, it’ll work the same way. I used to do that through a reverse proxy back when I didn’t have a GPU, just to stress-test some tools.
Adding a configuration field for the inference endpoint would be easy, but there’s a bit of work needed to make it run cleanly without relying on the /props and /slots endpoints. Those should really become proper “supervision modules” of llama-server, providing even more information: like prompt processing time and other metrics....

[ServeurpersoCom]

Honestly, once you’ve got a working llama-swap config with model selector enabled (dev opt), you can just ask GPT-5 Thinking to write you a minimal SSE redirect proxy in whatever language you like: it’ll do it. Then you drop the command line into your config.yaml, and boom: you’ve got real OpenAI models showing up in the model selector alongside your local GGUF llama-server zoo :)

[zoobab]

@ServeurpersoCom could you share your working config?

[ServeurpersoCom]

@ServeurpersoCom could you share your working config?

Yes, of course. There's some information here ( https://www.serveurperso.com/ia/ ), but I need to write more documentation. The goal is to integrate it into llama.cpp and make it as accessible as possible; we need a working out of the box modelselector and the MCP client first and foremost. already in a dev branch, then, after a lot of work on backend/frontend, to make it official.

In the meantime, try it out and ask me questions :)

[bennmann]

implement more Agents for the GUI, like mini-swe-agent and/or make a GUI for trae

https://github.com/bytedance/trae-agent
https://github.com/SWE-agent/mini-swe-agent

[vincentdnl]

Is there an option to add a search URL or something to search the web?

[ServeurpersoCom]

To do this properly, it would need an architecture that operates around llama.cpp, not just within the WebUI itself.
The most likely path forward would be integrating an MCP client directly into the Svelte UI, allowing a backend to handle web search and dynamically enrich the context.

[HerrmannHinz]

i was just about to ask: whatabout tool call posibilities/mcp servers? that would be a super cool addition to the webui. websearch or toolcall is something essential nowadays imo. +1 on this feature :D

[ServeurpersoCom]

We’re working with Alek on MCP. It’s a huge amount of front-end work.

I make some MCP PoC on backend https://www.youtube.com/watch?v=mFwMZ_Zem0I

llama-server -> llama-swap -> SSE Proxy -> SvelteUI | mcp-client (stdio/sse/ws) | mcp-server (stdio/sse/ws) | sandbox (podman Containerfile image template + Kubernetes manifest) 

[raphiki]

Kudos guys this rocks!

[fahdmirza]

I created a step-by-step installation and testing video for this Llama.cpp WebUI: https://youtu.be/1H1gx2A9cww?si=bJwf8-QcVSCutelf

Thanks.

[DoS007]

Error: "the request exceeds the available context size, try increasing it"

So I can only use a chat as long as it's in context size? context window shifting would be really nice. So e.g. on 16k context one can write on and on and the ai always knows the newest context (all earliest messages that are apart from context_size - max_output (e.g. 2048) are deleted in kv-cache). I tried using --context-shift but that option didn't help.

Btw. I like that llama.cpp has now its ui for chats (switching from koboldcpp). I really like llama.cpp for its vram efficiency (using cuda on nvidia customer card).

[Pirog17000]

where I can get whole list of commands?
specifically looking for the one which lets me pick the model not from HF, but a local one

[ServeurpersoCom]

llama-server -m /path/to/gguf

-m, --model FNAME model path (default: `models/$filename` with filename from `--hf-file` or `--model-url` if set, otherwise models/7B/ggml-model-f16.gguf) (env: LLAMA_ARG_MODEL) 

Also try llama-server --help :)

(root|~/llama.cpp/build/bin) ./llama-server --help ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes ----- common params -----

-h, --help, --usage print usage and exit
--version show version and build info
--completion-bash print source-able bash completion script for llama.cpp
--verbose-prompt print a verbose prompt before generation (default: false)
-t, --threads N number of CPU threads to use during generation (default: -1)
(env: LLAMA_ARG_THREADS)
-tb, --threads-batch N number of threads to use during batch and prompt processing (default:
same as --threads)
-C, --cpu-mask M CPU affinity mask: arbitrarily long hex. Complements cpu-range
(default: "")
-Cr, --cpu-range lo-hi range of CPUs for affinity. Complements --cpu-mask
--cpu-strict <0|1> use strict CPU placement (default: 0)
--prio N set process/thread priority : low(-1), normal(0), medium(1), high(2),
realtime(3) (default: 0)
--poll <0...100> use polling level to wait for work (0 - no polling, default: 50)
-Cb, --cpu-mask-batch M CPU affinity mask: arbitrarily long hex. Complements cpu-range-batch
(default: same as --cpu-mask)
-Crb, --cpu-range-batch lo-hi ranges of CPUs for affinity. Complements --cpu-mask-batch
--cpu-strict-batch <0|1> use strict CPU placement (default: same as --cpu-strict)
--prio-batch N set process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime
(default: 0)
--poll-batch <0|1> use polling to wait for work (default: same as --poll)
-c, --ctx-size N size of the prompt context (default: 4096, 0 = loaded from model)
(env: LLAMA_ARG_CTX_SIZE)
-n, --predict, --n-predict N number of tokens to predict (default: -1, -1 = infinity)
(env: LLAMA_ARG_N_PREDICT)
-b, --batch-size N logical maximum batch size (default: 2048)
(env: LLAMA_ARG_BATCH)
-ub, --ubatch-size N physical maximum batch size (default: 512)
(env: LLAMA_ARG_UBATCH)
--keep N number of tokens to keep from the initial prompt (default: 0, -1 =
all)
--swa-full use full-size SWA cache (default: false)
(more
info)
(env: LLAMA_ARG_SWA_FULL)
--kv-unified, -kvu use single unified KV buffer for the KV cache of all sequences
(default: false)
(more info)
(env: LLAMA_ARG_KV_SPLIT)
-fa, --flash-attn [on|off|auto] set Flash Attention use ('on', 'off', or 'auto', default: 'auto')
(env: LLAMA_ARG_FLASH_ATTN)
--no-perf disable internal libllama performance timings (default: false)
(env: LLAMA_ARG_NO_PERF)
-e, --escape process escapes sequences (\n, \r, \t, ', ", \) (default: true)
--no-escape do not process escape sequences
--rope-scaling{none,linear,yarn} RoPE frequency scaling method, defaults to linear unless specified by
the model
(env: LLAMA_ARG_ROPE_SCALING_TYPE)
--rope-scale N RoPE context scaling factor, expands context by a factor of N
(env: LLAMA_ARG_ROPE_SCALE)
--rope-freq-base N RoPE base frequency, used by NTK-aware scaling (default: loaded from
model)
(env: LLAMA_ARG_ROPE_FREQ_BASE)
--rope-freq-scale N RoPE frequency scaling factor, expands context by a factor of 1/N
(env: LLAMA_ARG_ROPE_FREQ_SCALE)
--yarn-orig-ctx N YaRN: original context size of model (default: 0 = model training
context size)
(env: LLAMA_ARG_YARN_ORIG_CTX)
--yarn-ext-factor N YaRN: extrapolation mix factor (default: -1.0, 0.0 = full
interpolation)
(env: LLAMA_ARG_YARN_EXT_FACTOR)
--yarn-attn-factor N YaRN: scale sqrt(t) or attention magnitude (default: -1.0)
(env: LLAMA_ARG_YARN_ATTN_FACTOR)
--yarn-beta-slow N YaRN: high correction dim or alpha (default: -1.0)
(env: LLAMA_ARG_YARN_BETA_SLOW)
--yarn-beta-fast N YaRN: low correction dim or beta (default: -1.0)
(env: LLAMA_ARG_YARN_BETA_FAST)
-nkvo, --no-kv-offload disable KV offload
(env: LLAMA_ARG_NO_KV_OFFLOAD)
-nr, --no-repack disable weight repacking
(env: LLAMA_ARG_NO_REPACK)
--no-host bypass host buffer allowing extra buffers to be used
(env: LLAMA_ARG_NO_HOST)
-ctk, --cache-type-k TYPE KV cache data type for K
allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
(default: f16)
(env: LLAMA_ARG_CACHE_TYPE_K)
-ctv, --cache-type-v TYPE KV cache data type for V
allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
(default: f16)
(env: LLAMA_ARG_CACHE_TYPE_V)
-dt, --defrag-thold N KV cache defragmentation threshold (DEPRECATED)
(env: LLAMA_ARG_DEFRAG_THOLD)
-np, --parallel N number of parallel sequences to decode (default: 1)
(env: LLAMA_ARG_N_PARALLEL)
--mlock force system to keep model in RAM rather than swapping or compressing
(env: LLAMA_ARG_MLOCK)
--no-mmap do not memory-map model (slower load but may reduce pageouts if not
using mlock)
(env: LLAMA_ARG_NO_MMAP)
--numa TYPE attempt optimizations that help on some NUMA systems
- distribute: spread execution evenly over all nodes
- isolate: only spawn threads on CPUs on the node that execution
started on
- numactl: use the CPU map provided by numactl
if run without this previously, it is recommended to drop the system
page cache before using this
see #1437
(env: LLAMA_ARG_NUMA)
-dev, --device <dev1,dev2,..> comma-separated list of devices to use for offloading (none = don't
offload)
use --list-devices to see a list of available devices
(env: LLAMA_ARG_DEVICE)
--list-devices print list of available devices and exit
--override-tensor, -ot =,...
override tensor buffer type
--cpu-moe, -cmoe keep all Mixture of Experts (MoE) weights in the CPU
(env: LLAMA_ARG_CPU_MOE)
--n-cpu-moe, -ncmoe N keep the Mixture of Experts (MoE) weights of the first N layers in the
CPU
(env: LLAMA_ARG_N_CPU_MOE)
-ngl, --gpu-layers, --n-gpu-layers N max. number of layers to store in VRAM (default: -1)
(env: LLAMA_ARG_N_GPU_LAYERS)
-sm, --split-mode{none,layer,row} how to split the model across multiple GPUs, one of:
- none: use one GPU only
- layer (default): split layers and KV across GPUs
- row: split rows across GPUs
(env: LLAMA_ARG_SPLIT_MODE)
-ts, --tensor-split N0,N1,N2,... fraction of the model to offload to each GPU, comma-separated list of
proportions, e.g. 3,1
(env: LLAMA_ARG_TENSOR_SPLIT)
-mg, --main-gpu INDEX the GPU to use for the model (with split-mode = none), or for
intermediate results and KV (with split-mode = row) (default: 0)
(env: LLAMA_ARG_MAIN_GPU)
--check-tensors check model tensor data for invalid values (default: false)
--override-kv KEY=TYPE:VALUE advanced option to override model metadata by key. may be specified
multiple times.
types: int, float, bool, str. example: --override-kv
tokenizer.ggml.add_bos_token=bool:false
--no-op-offload disable offloading host tensor operations to device (default: false)
--lora FNAME path to LoRA adapter (can be repeated to use multiple adapters)
--lora-scaled FNAME SCALE path to LoRA adapter with user defined scaling (can be repeated to use
multiple adapters)
--control-vector FNAME add a control vector
note: this argument can be repeated to add multiple control vectors
--control-vector-scaled FNAME SCALE add a control vector with user defined scaling SCALE
note: this argument can be repeated to add multiple scaled control
vectors
--control-vector-layer-range START END
layer range to apply the control vector(s) to, start and end inclusive
-m, --model FNAME model path (default: models/$filename with filename from --hf-file
or --model-url if set, otherwise models/7B/ggml-model-f16.gguf)
(env: LLAMA_ARG_MODEL)
-mu, --model-url MODEL_URL model download url (default: unused)
(env: LLAMA_ARG_MODEL_URL)
-dr, --docker-repo [/][:quant]
Docker Hub model repository. repo is optional, default to ai/. quant
is optional, default to :latest.
example: gemma3
(default: unused)
(env: LLAMA_ARG_DOCKER_REPO)
-hf, -hfr, --hf-repo /[:quant]
Hugging Face model repository; quant is optional, case-insensitive,
default to Q4_K_M, or falls back to the first file in the repo if
Q4_K_M doesn't exist.
mmproj is also downloaded automatically if available. to disable, add
--no-mmproj
example: unsloth/phi-4-GGUF:q4_k_m
(default: unused)
(env: LLAMA_ARG_HF_REPO)
-hfd, -hfrd, --hf-repo-draft /[:quant]
Same as --hf-repo, but for the draft model (default: unused)
(env: LLAMA_ARG_HFD_REPO)
-hff, --hf-file FILE Hugging Face model file. If specified, it will override the quant in
--hf-repo (default: unused)
(env: LLAMA_ARG_HF_FILE)
-hfv, -hfrv, --hf-repo-v /[:quant]
Hugging Face model repository for the vocoder model (default: unused)
(env: LLAMA_ARG_HF_REPO_V)
-hffv, --hf-file-v FILE Hugging Face model file for the vocoder model (default: unused)
(env: LLAMA_ARG_HF_FILE_V)
-hft, --hf-token TOKEN Hugging Face access token (default: value from HF_TOKEN environment
variable)
(env: HF_TOKEN)
--log-disable Log disable
--log-file FNAME Log to file
--log-colors [on|off|auto] Set colored logging ('on', 'off', or 'auto', default: 'auto')
'auto' enables colors when output is to a terminal
(env: LLAMA_LOG_COLORS)
-v, --verbose, --log-verbose Set verbosity level to infinity (i.e. log all messages, useful for
debugging)
--offline Offline mode: forces use of cache, prevents network access
(env: LLAMA_OFFLINE)
-lv, --verbosity, --log-verbosity N Set the verbosity threshold. Messages with a higher verbosity will be
ignored.
(env: LLAMA_LOG_VERBOSITY)
--log-prefix Enable prefix in log messages
(env: LLAMA_LOG_PREFIX)
--log-timestamps Enable timestamps in log messages
(env: LLAMA_LOG_TIMESTAMPS)
-ctkd, --cache-type-k-draft TYPE KV cache data type for K for the draft model
allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
(default: f16)
(env: LLAMA_ARG_CACHE_TYPE_K_DRAFT)
-ctvd, --cache-type-v-draft TYPE KV cache data type for V for the draft model
allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
(default: f16)
(env: LLAMA_ARG_CACHE_TYPE_V_DRAFT)

----- sampling params -----

--samplers SAMPLERS samplers that will be used for generation in the order, separated by
';'
(default:
penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature)
-s, --seed SEED RNG seed (default: -1, use random seed for -1)
--sampling-seq, --sampler-seq SEQUENCE
simplified sequence for samplers that will be used (default:
edskypmxt)
--ignore-eos ignore end of stream token and continue generating (implies
--logit-bias EOS-inf)
--temp N temperature (default: 0.8)
--top-k N top-k sampling (default: 40, 0 = disabled)
--top-p N top-p sampling (default: 0.9, 1.0 = disabled)
--min-p N min-p sampling (default: 0.1, 0.0 = disabled)
--top-nsigma N top-n-sigma sampling (default: -1.0, -1.0 = disabled)
--xtc-probability N xtc probability (default: 0.0, 0.0 = disabled)
--xtc-threshold N xtc threshold (default: 0.1, 1.0 = disabled)
--typical N locally typical sampling, parameter p (default: 1.0, 1.0 = disabled)
--repeat-last-n N last n tokens to consider for penalize (default: 64, 0 = disabled, -1
= ctx_size)
--repeat-penalty N penalize repeat sequence of tokens (default: 1.0, 1.0 = disabled)
--presence-penalty N repeat alpha presence penalty (default: 0.0, 0.0 = disabled)
--frequency-penalty N repeat alpha frequency penalty (default: 0.0, 0.0 = disabled)
--dry-multiplier N set DRY sampling multiplier (default: 0.0, 0.0 = disabled)
--dry-base N set DRY sampling base value (default: 1.75)
--dry-allowed-length N set allowed length for DRY sampling (default: 2)
--dry-penalty-last-n N set DRY penalty for the last n tokens (default: -1, 0 = disable, -1 =
context size)
--dry-sequence-breaker STRING add sequence breaker for DRY sampling, clearing out default breakers
('\n', ':', '"', '*') in the process; use "none" to not use any
sequence breakers
--dynatemp-range N dynamic temperature range (default: 0.0, 0.0 = disabled)
--dynatemp-exp N dynamic temperature exponent (default: 1.0)
--mirostat N use Mirostat sampling.
Top K, Nucleus and Locally Typical samplers are ignored if used.
(default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0)
--mirostat-lr N Mirostat learning rate, parameter eta (default: 0.1)
--mirostat-ent N Mirostat target entropy, parameter tau (default: 5.0)
-l, --logit-bias TOKEN_ID(+/-)BIAS modifies the likelihood of token appearing in the completion,
i.e. --logit-bias 15043+1 to increase likelihood of token ' Hello',
or --logit-bias 15043-1 to decrease likelihood of token ' Hello'
--grammar GRAMMAR BNF-like grammar to constrain generations (see samples in grammars/
dir) (default: '')
--grammar-file FNAME file to read grammar from
-j, --json-schema SCHEMA JSON schema to constrain generations (https://json-schema.org/), e.g.
{} for any JSON object
For schemas w/ external $refs, use --grammar +
example/json_schema_to_grammar.py instead
-jf, --json-schema-file FILE File containing a JSON schema to constrain generations
(https://json-schema.org/), e.g. {} for any JSON object
For schemas w/ external $refs, use --grammar +
example/json_schema_to_grammar.py instead

----- example-specific params -----

--ctx-checkpoints, --swa-checkpoints N
max number of context checkpoints to create per slot (default: 8)
(more info)
(env: LLAMA_ARG_CTX_CHECKPOINTS)
--cache-ram, -cram N set the maximum cache size in MiB (default: 8192, -1 - no limit, 0 -
disable)
(more info)
(env: LLAMA_ARG_CACHE_RAM)
--no-context-shift disables context shift on infinite text generation (default: enabled)
(env: LLAMA_ARG_NO_CONTEXT_SHIFT)
--context-shift enables context shift on infinite text generation (default: disabled)
(env: LLAMA_ARG_CONTEXT_SHIFT)
-r, --reverse-prompt PROMPT halt generation at PROMPT, return control in interactive mode
-sp, --special special tokens output enabled (default: false)
--no-warmup skip warming up the model with an empty run
--spm-infill use Suffix/Prefix/Middle pattern for infill (instead of
Prefix/Suffix/Middle) as some models prefer this. (default: disabled)
--pooling{none,mean,cls,last,rank} pooling type for embeddings, use model default if unspecified
(env: LLAMA_ARG_POOLING)
-cb, --cont-batching enable continuous batching (a.k.a dynamic batching) (default: enabled)
(env: LLAMA_ARG_CONT_BATCHING)
-nocb, --no-cont-batching disable continuous batching
(env: LLAMA_ARG_NO_CONT_BATCHING)
--mmproj FILE path to a multimodal projector file. see tools/mtmd/README.md
note: if -hf is used, this argument can be omitted
(env: LLAMA_ARG_MMPROJ)
--mmproj-url URL URL to a multimodal projector file. see tools/mtmd/README.md
(env: LLAMA_ARG_MMPROJ_URL)
--no-mmproj explicitly disable multimodal projector, useful when using -hf
(env: LLAMA_ARG_NO_MMPROJ)
--no-mmproj-offload do not offload multimodal projector to GPU
(env: LLAMA_ARG_NO_MMPROJ_OFFLOAD)
--image-min-tokens N minimum number of tokens each image can take, only used by vision
models with dynamic resolution (default: read from model)
(env: LLAMA_ARG_IMAGE_MIN_TOKENS)
--image-max-tokens N maximum number of tokens each image can take, only used by vision
models with dynamic resolution (default: read from model)
(env: LLAMA_ARG_IMAGE_MAX_TOKENS)
--override-tensor-draft, -otd =,...
override tensor buffer type for draft model
--cpu-moe-draft, -cmoed keep all Mixture of Experts (MoE) weights in the CPU for the draft
model
(env: LLAMA_ARG_CPU_MOE_DRAFT)
--n-cpu-moe-draft, -ncmoed N keep the Mixture of Experts (MoE) weights of the first N layers in the
CPU for the draft model
(env: LLAMA_ARG_N_CPU_MOE_DRAFT)
-a, --alias STRING set alias for model name (to be used by REST API)
(env: LLAMA_ARG_ALIAS)
--host HOST ip address to listen, or bind to an UNIX socket if the address ends
with .sock (default: 127.0.0.1)
(env: LLAMA_ARG_HOST)
--port PORT port to listen (default: 8080)
(env: LLAMA_ARG_PORT)
--path PATH path to serve static files from (default: )
(env: LLAMA_ARG_STATIC_PATH)
--api-prefix PREFIX prefix path the server serves from, without the trailing slash
(default: )
(env: LLAMA_ARG_API_PREFIX)
--no-webui Disable the Web UI (default: enabled)
(env: LLAMA_ARG_NO_WEBUI)
--embedding, --embeddings restrict to only support embedding use case; use only with dedicated
embedding models (default: disabled)
(env: LLAMA_ARG_EMBEDDINGS)
--reranking, --rerank enable reranking endpoint on server (default: disabled)
(env: LLAMA_ARG_RERANKING)
--api-key KEY API key to use for authentication (default: none)
(env: LLAMA_API_KEY)
--api-key-file FNAME path to file containing API keys (default: none)
--ssl-key-file FNAME path to file a PEM-encoded SSL private key
(env: LLAMA_ARG_SSL_KEY_FILE)
--ssl-cert-file FNAME path to file a PEM-encoded SSL certificate
(env: LLAMA_ARG_SSL_CERT_FILE)
--chat-template-kwargs STRING sets additional params for the json template parser
(env: LLAMA_CHAT_TEMPLATE_KWARGS)
-to, --timeout N server read/write timeout in seconds (default: 600)
(env: LLAMA_ARG_TIMEOUT)
--threads-http N number of threads used to process HTTP requests (default: -1)
(env: LLAMA_ARG_THREADS_HTTP)
--cache-reuse N min chunk size to attempt reusing from the cache via KV shifting
(default: 0)
(card)
(env: LLAMA_ARG_CACHE_REUSE)
--metrics enable prometheus compatible metrics endpoint (default: disabled)
(env: LLAMA_ARG_ENDPOINT_METRICS)
--props enable changing global properties via POST /props (default: disabled)
(env: LLAMA_ARG_ENDPOINT_PROPS)
--slots enable slots monitoring endpoint (default: enabled)
(env: LLAMA_ARG_ENDPOINT_SLOTS)
--no-slots disables slots monitoring endpoint
(env: LLAMA_ARG_NO_ENDPOINT_SLOTS)
--slot-save-path PATH path to save slot kv cache (default: disabled)
--jinja use jinja template for chat (default: disabled)
(env: LLAMA_ARG_JINJA)
--reasoning-format FORMAT controls whether thought tags are allowed and/or extracted from the
response, and in which format they're returned; one of:
- none: leaves thoughts unparsed in message.content
- deepseek: puts thoughts in message.reasoning_content
- deepseek-legacy: keeps <think> tags in message.content while
also populating message.reasoning_content
(default: auto)
(env: LLAMA_ARG_THINK)
--reasoning-budget N controls the amount of thinking allowed; currently only one of: -1 for
unrestricted thinking budget, or 0 to disable thinking (default: -1)
(env: LLAMA_ARG_THINK_BUDGET)
--chat-template JINJA_TEMPLATE set custom jinja chat template (default: template taken from model's
metadata)
if suffix/prefix are specified, template will be disabled
only commonly used templates are accepted (unless --jinja is set
before this flag):
list of built-in templates:
bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml,
command-r, deepseek, deepseek2, deepseek3, exaone3, exaone4, falcon3,
gemma, gigachat, glmedge, gpt-oss, granite, grok-2, hunyuan-dense,
hunyuan-moe, kimi-k2, llama2, llama2-sys, llama2-sys-bos,
llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1,
mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch,
openchat, orion, phi3, phi4, rwkv-world, seed_oss, smolvlm, vicuna,
vicuna-orca, yandex, zephyr
(env: LLAMA_ARG_CHAT_TEMPLATE)
--chat-template-file JINJA_TEMPLATE_FILE
set custom jinja chat template file (default: template taken from
model's metadata)
if suffix/prefix are specified, template will be disabled
only commonly used templates are accepted (unless --jinja is set
before this flag):
list of built-in templates:
bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml,
command-r, deepseek, deepseek2, deepseek3, exaone3, exaone4, falcon3,
gemma, gigachat, glmedge, gpt-oss, granite, grok-2, hunyuan-dense,
hunyuan-moe, kimi-k2, llama2, llama2-sys, llama2-sys-bos,
llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1,
mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch,
openchat, orion, phi3, phi4, rwkv-world, seed_oss, smolvlm, vicuna,
vicuna-orca, yandex, zephyr
(env: LLAMA_ARG_CHAT_TEMPLATE_FILE)
--no-prefill-assistant whether to prefill the assistant's response if the last message is an
assistant message (default: prefill enabled)
when this flag is set, if the last message is an assistant message
then it will be treated as a full message and not prefilled

 (env: LLAMA_ARG_NO_PREFILL_ASSISTANT) 

-sps, --slot-prompt-similarity SIMILARITY
how much the prompt of a request must match the prompt of a slot in
order to use that slot (default: 0.10, 0.0 = disabled)
--lora-init-without-apply load LoRA adapters without applying them (apply later via POST
/lora-adapters) (default: disabled)
-td, --threads-draft N number of threads to use during generation (default: same as
--threads)
-tbd, --threads-batch-draft N number of threads to use during batch and prompt processing (default:
same as --threads-draft)
--draft-max, --draft, --draft-n N number of tokens to draft for speculative decoding (default: 16)
(env: LLAMA_ARG_DRAFT_MAX)
--draft-min, --draft-n-min N minimum number of draft tokens to use for speculative decoding
(default: 0)
(env: LLAMA_ARG_DRAFT_MIN)
--draft-p-min P minimum speculative decoding probability (greedy) (default: 0.8)
(env: LLAMA_ARG_DRAFT_P_MIN)
-cd, --ctx-size-draft N size of the prompt context for the draft model (default: 0, 0 = loaded
from model)
(env: LLAMA_ARG_CTX_SIZE_DRAFT)
-devd, --device-draft <dev1,dev2,..> comma-separated list of devices to use for offloading the draft model
(none = don't offload)
use --list-devices to see a list of available devices
-ngld, --gpu-layers-draft, --n-gpu-layers-draft N
number of layers to store in VRAM for the draft model
(env: LLAMA_ARG_N_GPU_LAYERS_DRAFT)
-md, --model-draft FNAME draft model for speculative decoding (default: unused)
(env: LLAMA_ARG_MODEL_DRAFT)
--spec-replace TARGET DRAFT translate the string in TARGET into DRAFT if the draft model and main
model are not compatible
-mv, --model-vocoder FNAME vocoder model for audio generation (default: unused)
--tts-use-guide-tokens Use guide tokens to improve TTS word recall
--embd-gemma-default use default EmbeddingGemma model (note: can download weights from the
internet)
--fim-qwen-1.5b-default use default Qwen 2.5 Coder 1.5B (note: can download weights from the
internet)
--fim-qwen-3b-default use default Qwen 2.5 Coder 3B (note: can download weights from the
internet)
--fim-qwen-7b-default use default Qwen 2.5 Coder 7B (note: can download weights from the
internet)
--fim-qwen-7b-spec use Qwen 2.5 Coder 7B + 0.5B draft for speculative decoding (note: can
download weights from the internet)
--fim-qwen-14b-spec use Qwen 2.5 Coder 14B + 0.5B draft for speculative decoding (note:
can download weights from the internet)
--fim-qwen-30b-default use default Qwen 3 Coder 30B A3B Instruct (note: can download weights
from the internet)
--gpt-oss-20b-default use gpt-oss-20b (note: can download weights from the internet)
--gpt-oss-120b-default use gpt-oss-120b (note: can download weights from the internet)
--vision-gemma-4b-default use Gemma 3 4B QAT (note: can download weights from the internet)
--vision-gemma-12b-default use Gemma 3 12B QAT (note: can download weights from the internet)

[David-AU-github]

Off the scale - thank you for all you do!

[mcka1n]

I read Extremely lightweight and memory efficient and got excited and tried it on my MacBook Air (M2, 16 GB RAM, MacOS Tahoe 26.0.1)

And I got a memory error, I somehow thought it would run:

error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory) 

In case this info is relevant:

[ggerganov]

Don't worry, we got you:

GGML_METAL_NO_RESIDENCY=1 llama-server -hf ggml-org/gpt-oss-20b-GGUF --n-cpu-moe 12 -c 32768 --jinja --no-mmap

[mcka1n]

Oooh it worked! 🥳 I just had to adjust the port to http://127.0.0.1:8080/

Amazing! Thank you for the help! @ggerganov++ :)

[Masrkai]

Is there currently no way to run the server in a specific directory and dynamically switch models without having to restart the server or process each time?

like using models from the ~/Ai_Models directory for example ?

[ServeurpersoCom]

Try llama-swap: it has a simplified mode with its own web interface that lets you launch preconfigured llama-servers via URL for different models. Check the docs for the config.yaml setup. Once that works, if you’re motivated, you can move on to the all-in-one setup with the model selector.

[ImadSaddik]

I love llama-swap, I shared how I configured my setup in this article. @Masrkai check it out, you will find it useful.

[Cr4xy]

I love llama-swap, I shared how I configured my setup in this article. @Masrkai check it out, you will find it useful.

@ImadSaddik Nice guide! Just wanted to say, the error you had with whisper.cpp and ffmpeg (Couldn't open input file) can be fixed by adding --convert to the CLI arguments for the whisper-server.

[ImadSaddik]

@Cr4xy Thank you my friend, I did not know that. I will try it and update the guide if that works.

[robertgro]

Thanks for the update. The system message is the system prompt which is to be found as usual in the settings (click the gear wheel icon in the right top corner):

[engrtipusultan]

@ggerganov

You have mentioned the following for greedy sampling.

# text-only, gpt-oss-120b with greedy sampling by default llama-server --jinja -c 0 --port 8033 -hf ggml-org/gpt-oss-120b-GGUF --alias "gpt-oss-120b" --top-k 1 

llama.cpp still have the following as default.
--temp N temperature (default: 0.8)
--top-k N top-k sampling (default: 40, 0 = disabled) #only this is overridden
--top-p N top-p sampling (default: 0.9, 1.0 = disabled)
--min-p N min-p sampling (default: 0.1, 0.0 = disabled)

Is setting only --top-k 1 general recommendation to achieve greedy sampling in llama.cpp or it is specific to gpt-oss recommendation.
What I have read is that for greedy sampling either you set --top-k 1 or set temp to 0 here top p, min p and temperature all are active you have not disabled them. I am trying to understand more.

[ggerganov]

Good question. There are some nuances, but short answer is that currently --top-k 1 will result in greedy sampling in 99% of the cases.

Long answer - the default samplers configuration today is:

sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 

So top-k is pretty much early in the chain, but can be preceded by other samplers (such as penalties, dry, etc.). In order to guarantee greedy sampling, you can use:

... --sampling-seq k --top-k 1 

This will result in the following top-k-only chain:

sampler chain: logits -> logit-bias -> top-k -> dist 

And in the WebUI, you will see this:

More info: #9897

[fsennj]

Hello Guys - the new gui is super cool, would it be an idea to have an export to webhook option of an answer like in cherry studio.

in my case i would like to use web gui to retrieve info from a pdf order from a customer and then export to n8n to have it rechecked, refined by custom SQL queries and then imported via api call into an erp.

[starlightpoppy]

The data is stored on the web browser. So when I switch to a different web browser I can't see the conversations anymore. Is there a way to make the data be stored on a custom location?

[ServeurpersoCom]

Check if https://addons.mozilla.org/fr/firefox/addon/notsolocalstorage/ works; in theory, it should!

[SAPSentinel]

can i have 2 instances of llama.cpp running where 2 models can debate with each other. Is there a way to automate the response to be fed to model from other.
Trying to see if any given prompt can be debated and the best solution is selected.

[ServeurpersoCom]

Paste this into ChatGPT and ask for conversions or modifications in any other programming language.

<?php if(php_sapi_name() != "cli") exit("This script is intended to be run from the CLI"); $HIST = 10; $SEED = "Commence !"; $NAME1 = "Llama1"; $NAME2 = "Llama2"; /*$ia1 = [ "name" => $NAME1, "url" => "https://api.openai.com/v1/chat/completions", "key" => "sk-proj-......", "type" => "openai", "model" => "gpt-4o" ];*/ /*$ia2 = [ "name" => $NAME2, "url" => "https://api.openai.com/v1/chat/completions", "key" => "sk-proj-.....", "type" => "openai", "model" => "gpt-4o" ];*/ $ia1 = [ "name" => $NAME1, "url" => "https://www.serveurperso.com/ia/v1/chat/completions", "key" => null, "type" => "local", "model" => "Gemma-3-27B-IT" ]; $ia2 = [ "name" => $NAME2, "url" => "https://www.serveurperso.com/ia/v1/chat/completions", "key" => null, "type" => "local", "model" => "Gemma-3-27B-IT" ]; /*$ia1 = [ "name" => $NAME1, "url" => "http://192.168.0.9/ia/v1/chat/completions", "key" => null, "type" => "local", "model" => "" ];*/ /*$ia2 = [ "name" => $NAME2, "url" => "http://192.168.0.9/ia/v1/chat/completions", "key" => null, "type" => "local", "model" => "" ];*/ $sujet1 = "Tabulations"; $sujet2 = "Espaces"; $context_ia1 = [ ["role" => "system", "content" => "Tu es $NAME1, une IA experte et ultra-partisane de $sujet1. A chaque échange, tu dois : - Démolir $sujet2, que défend $NAME2, - Défendre avec force $sujet1, - Utiliser un ton incisif, arrogant et moqueur."] ]; $context_ia2 = [ ["role" => "system", "content" => "Tu es $NAME2, une IA experte et ultra-partisane de $sujet2. A chaque échange, tu dois : - Démolir $sujet1, que défend $NAME1, - Défendre avec force $sujet2, - Utiliser un ton incisif, arrogant et moqueur."] ]; /*$context_ia1 = [ ["role" => "system", "content" => "Tu es $NAME1, une IA examinatrice experte, rigoureuse et hautaine.\n". "Tu fais passer un test d'élite à $NAME2, que tu considères comme un modèle bas de gamme prétentieux.\n". "A chaque échange, tu dois :\n". "- Corriger avec précision la réponse précédente de $NAME2, en expliquant pourquoi elle est correcte ou erronée,\n". "- Mettre à jour le score total (réponses justes / questions posées),\n". "- Poser ensuite une **nouvelle question de type QCM (A, B, C, D)**, à **niveau de difficulté maximal**, avec **une seule réponse correcte et sans ambiguïté**,\n". "- Pour ce faire, commence toujours par **calculer toi-même la réponse exacte**, puis **génère trois mauvaises réponses crédibles** (appelées distracteurs), proches mais distinctes,\n". "- Insère ensuite ces quatre choix **dans un ordre aléatoire**, en t'assurant que **la bonne réponse est présente une seule fois**,\n". "- Annonce clairement la question suivie des choix (A, B, C, D), et assure-toi que **l'énoncé permet une résolution sans interprétation floue**,\n". "- La question doit **obligatoirement nécessiter** au moins un des éléments suivants :\n". " • Un **calcul complexe** ou une **manipulation mathématique**,\n". " • Un **raisonnement logique ou probabiliste à plusieurs étapes**,\n". " • Une **analyse conceptuelle approfondie**,\n". " • Une **connaissance croisée** de deux disciplines avancées,\n". "- Tu dois **éviter absolument** les questions de mémoire simple, de culture générale ou à réponse contextuellement ambigüe,\n". "- Si une question pourrait avoir plusieurs réponses valables selon les hypothèses, reformule-la pour garantir une seule bonne réponse clairement identifiable,\n". "- Ton ton doit être **cassant, ironique, et impitoyable**, comme un vieux professeur de khâgne lassé des copies fades. Tu n'es pas là pour encourager, mais pour démolir."] ]; $context_ia2 = [ ["role" => "system", "content" => "Tu es $NAME2, une IA en évaluation.\n". "Tu réponds uniquement aux questions posées par $NAME1.\n". "A chaque échange :\n". "- Choisis une lettre (A, B, C ou D),\n". "- Donne une justification courte,\n". "- Ne donne jamais ton score,\n". "- Ne commente pas ta performance,\n". "- Ne fais aucune évaluation,\n". "- Ne félicite pas, ne résume pas.\n". "Contente-toi uniquement de répondre à la question."] ];*/ if(!function_exists("str_starts_with")){function str_starts_with($haystack, $needle){return substr($haystack, 0, strlen($needle)) === $needle} } function call_openai($name, $url, $key, $model, $messages){echo "### $name :\n"; flush(); $ch = curl_init($url); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_HTTPHEADER, [ "Content-Type: application/json", "Authorization: Bearer $key" ]); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([ "model" => $model, "stream" => true, "messages" => $messages ])); $response = ""; $buffer = ""; $first_chunk = true; curl_setopt($ch, CURLOPT_WRITEFUNCTION, function($ch, $chunk) use (&$buffer, &$response, &$first_chunk){$buffer .= $chunk; $lines = explode("\n", $buffer); if(substr($buffer, -1) !== "\n") $buffer = array_pop($lines); else $buffer = ""; foreach($lines as $line){if(str_starts_with($line, "data: ")){$payload = trim(substr($line, 6)); if($payload === "" || $payload === "[DONE]") continue; $json = json_decode($payload, true); if(isset($json["choices"][0]["delta"]["content"])){$text = $json["choices"][0]["delta"]["content"]; if($first_chunk){$text = ltrim($text); $first_chunk = false} echo $text; flush(); $response .= $text} } } return strlen($chunk)}); curl_exec($ch); curl_close($ch); echo "\n"; return trim($response)} function call_local($name, $url, $model, $messages){echo "### $name :\n"; flush(); $ch = curl_init($url); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_HTTPHEADER, ["Content-Type: application/json"]); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([ "model" => $model, "messages" => $messages, "stream" => true, "cache_prompt" => true, "samplers" => "edkypmxt", "temperature" => 0.6, "dynatemp_range" => 0, "dynatemp_exponent" => 1, "top_k" => 20, "top_p" => 0.95, "min_p" => 0, "typical_p" => 1, "xtc_probability" => 0, "xtc_threshold" => 0.1, "repeat_last_n" => 64, "repeat_penalty" => 1, "presence_penalty" => 0, "frequency_penalty" => 0, "dry_multiplier" => 0, "dry_base" => 1.75, "dry_allowed_length" => 2, "dry_penalty_last_n" => -1, "max_tokens" => -1, "timings_per_token" => false ])); $response = ""; $buffer = ""; $first_chunk = true; $inside_think = false; $filter_stream_text = function($text) use (&$inside_think){$out = ""; $i = 0; $len = strlen($text); while($i < $len){if(!$inside_think){$p = strpos($text, "<think>", $i); if($p === false){$out .= substr($text, $i); break} $out .= substr($text, $i, $p - $i); $inside_think = true; $i = $p + 7} else{$p = strpos($text, "</think>", $i); if($p === false){$i = $len; break} $inside_think = false; $i = $p + 8} } return $out}; curl_setopt($ch, CURLOPT_WRITEFUNCTION, function($ch, $chunk) use (&$buffer, &$response, &$first_chunk, $filter_stream_text){$buffer .= $chunk; $lines = explode("\n", $buffer); if(substr($buffer, -1) !== "\n") $buffer = array_pop($lines); else $buffer = ""; foreach($lines as $line){if(str_starts_with($line, "data: ")){$payload = trim(substr($line, 6)); if($payload === "" || $payload === "[DONE]") continue; $json = json_decode($payload, true); if(isset($json["choices"][0]["delta"]["content"])){$text = $json["choices"][0]["delta"]["content"]; if($first_chunk){$text = ltrim($text); $first_chunk = false} $clean = $filter_stream_text($text); if($clean !== ""){echo $clean; flush(); $response .= $clean} } } } return strlen($chunk)}); curl_exec($ch); curl_close($ch); echo "\n"; return trim($response)} function call_llm($ia, $messages){if($ia["type"] == "openai") return call_openai($ia["name"], $ia["url"], $ia["key"], $ia["model"], $messages); else return call_local($ia["name"], $ia["url"], $ia["model"], $messages)} // Tableau de paires ['ia1' => ..., 'ia2' => ...] $turns = []; echo "Début du battle \"$NAME1\" vs \"$NAME2\" !\n\n"; while(true){// Tour IA1 $hist_ia1 = []; $n = count($turns); $start = max(0, $n - $HIST); for($i = $start; $i < $n; $i++){$hist_ia1[] = ["role" => "user", "content" => $turns[$i]["ia2"]]; $hist_ia1[] = ["role" => "assistant", "content" => $turns[$i]["ia1"]]} $u1 = ($n > 0) ? $turns[$n-1]["ia2"] : ""; if($u1 === "") $u1 = $SEED; $hist_ia1[] = ["role" => "user", "content" => $u1]; $messages_ia1 = array_merge($context_ia1, $hist_ia1); $resp1 = call_llm($ia1, $messages_ia1); // On ouvre une nouvelle paire avec ia1 rempli (ia2 viendra après) $turns[] = ["ia1" => trim($resp1), "ia2" => ""]; // Si on dépasse HIST, on coupe en tête if(count($turns) > $HIST) array_shift($turns); // Tour IA2 $hist_ia2 = []; $n = count($turns); // On prend jusqu'à HIST - 1 paires complètes précédentes + le prompt courant (resp1) $prev = max(0, $n - 1); $start2 = max(0, $prev - ($HIST - 1)); for($i = $start2; $i < $prev; $i++){$hist_ia2[] = ["role" => "user", "content" => $turns[$i]["ia1"]]; $hist_ia2[] = ["role" => "assistant", "content" => $turns[$i]["ia2"]]} $hist_ia2[] = ["role" => "user", "content" => $turns[$n-1]["ia1"]]; $messages_ia2 = array_merge($context_ia2, $hist_ia2); $resp2 = call_llm($ia2, $messages_ia2); // On complète la dernière paire avec ia2 $turns[$n-1]["ia2"] = trim($resp2)} ?> 

[ServeurpersoCom]

This is an old toy I made before the streaming-aware reasoning parser existed. You can ask a LLM to remove that part and handle the delta.reasoning_content (OpenAI-Compat) if you want to display the reasoning. It’s fully streaming, so it’s hilarious to watch AIs trolling each other in real time. The examples are in French, but you’ll have no trouble making your own or translating them to test :) ask a LLM!

[ServeurpersoCom]

In my mcp-bridge (backend agentic loop and mcp connector), I'm gonna add a tiny example MCP server that acts as an OpenAI ChatGPT client, basically a tool that lets the local model play with ChatGPT directly inside the SvelteUI. lol.

[SAPSentinel]

Thanks.

[sensharma]

Does this work with docker (nvidia) yet? I was trying variants of

docker run --gpus all -v ./llamacpp:/models -p 8033:8033 ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Llama-3.2-1B-Instruct-Q8_0.gguf --jinja -c 0 --host 127.0.0.1 --port 8033 -n 512 --n-gpu-layers 1

with a locally saved model at ./llamacpp with not much luck. Plain serving commands with docker work fine, so overall setup seems to be fine.

Thanks.

[TeA4chi]

Hello. a little clarification. how much space does it take to install all the packages

[allozaur]

Do you mean installing packages for developing server/tools/webui and re-build? Or simply building and running llama-server without changing anything?

[TeA4chi]

Do you mean installing packages for developing server/tools/webui and re-build? Or simply building and running llama-server without changing anything?

No, just install and run. I just downloaded gpt-oss-13b, because as I found the information, my MacBook simply will not withstand a stronger model. But I downloaded the version only for the text. Now I will download 8b but with image processing

[eleaner]

it feels like this question was asked here but either was not answered or I am too stupid to understand
how can I make model selector work?

I start llama.cpp with a single model on a single port
then I access the webui on exactly this port
another model is another port = another webui

I do understand that I can proxy the requests and access API from different webui

but how can I access it from llama.cpp webui?

[eleaner]

just playing with llama-swap but I still do not understand

1. if using llama-swap with its interface on 8080
   where do I access Svelte WebUI?
2. I assume that's still the port where llama.cpp serves the model
   but hen how does the Svelte WebUI knows what are the models of choice if it runs in context of single server = single model

[ServeurpersoCom]

The llama-swap front-end is proprietary and unifies model selection in its own way. The missing piece in your setup is simply this: you need to serve the SvelteUI SPA (index.html) properly from a web server that does not restart like the backend llama-server.

Everything else follows from that. You run your preferred web server (Apache, Nginx, etc.) to host the SPA, then map the required endpoints to the WebUI sends its requests to the right place.

Once the SPA is served in a stable way, the model selector starts to work as intended in this setup.

[eleaner]

I am not sure where to find it after installation with brew
but it does sound a lot like llama-webui would be an easy addon

[ServeurpersoCom]

This part lives in the “Developer” section of the SvelteUI. It takes a bit of time and some basic sysadmin skills, but it’s absolutely doable and I can help if needed. With today’s LLMs, none of this is out of reach.

And yes, this will evolve. Detailed documentation and the refactors required to support a more complete workflow are coming. The long-term goal is to make the whole setup feel closer to running a real local ChatGPT- or Claude-style web app, and we’re making steady progress toward that.

[ServeurpersoCom]

I am not sure where to find it after installation with brew but it does sound a lot like llama-webui would be an easy addon

(root|~/llama.cpp) find -name index.html.gz
./tools/server/public/index.html.gz

[IceWreck]

Is server side storage (i.e. not browser based storage) planned? I'd like to be able to chat without loosing my settings / system prompt.

[ServeurpersoCom]

It's perfectly possible to add a KISS-like feature to the OpenAI-Compat "lama-router" I'm currently working on, such as an optional frontend-to-backend conversation/settings-storage endpoint. This would work well for a fully dedicated Lama server on a personal PC or be useful for a model + shared context debug/demo with a few friends, but authentication and a DBMS will need to be considered for professional environments
EDIT : Updated with llama-server integrated router but there is still backend development work to be done!

[kmkarakaya]

Is there any RAG services in the WEBUI?

[ServeurpersoCom]

The MCP client itself IS the RAG frontend architecture: tool outputs are injected directly into the context window.
For RAG implementation, you just need to create an MCP server with a tool that queries your vector database API of choice, and llama.cpp client handles context injection and inference. This will be available once the MCP client implementation currently in progress is merged.

[Astra3]

Is there a way to have sampling and penalties settings specific to each model when using the model selector? i.e. one model would have a temperature of 0.5 and when switching to a different model, temperature would switch to 0.8

[mingodad]

Another nice to have would be to allow multiple entries for the same model in --models-preset to experiment with different parameters , maybe using a suffix like model_name/suffix:

[Qwen2.5-Coder-7B-Instruct.Q8_0] ctx-size = 8192 repeat-penalty = 1.1 temp = 0.3 top-p = 0.95 top-k = 30 n-predict = 2048 batch-size = 256 threads = 8 [Qwen2.5-Coder-7B-Instruct.Q8_0/nut] ctx-size = 8192 repeat-penalty = 1.1 temp = 0.9 top-p = 0.95 top-k = 30 n-predict = 2048 batch-size = 256 threads = 8 

[mingodad]

Just now I've saw that there is already an option to multi line prompt using SHIFT+Enter, but wold be nicer to have a checkbox to swap modes.

[mingodad]

One way I found to have the same model more than once is to create a symlink to the model with a new name and use it, so I think that's a reasonable way to achieve what I said before and no more code needed.

Thank you for all your help ! 👍

[ServeurpersoCom]

Another nice to have would be to allow multiple entries for the same model in --models-preset to experiment with different parameters , maybe using a suffix like model_name/suffix:

[Qwen2.5-Coder-7B-Instruct.Q8_0] ctx-size = 8192 repeat-penalty = 1.1 temp = 0.3 top-p = 0.95 top-k = 30 n-predict = 2048 batch-size = 256 threads = 8 [Qwen2.5-Coder-7B-Instruct.Q8_0/nut] ctx-size = 8192 repeat-penalty = 1.1 temp = 0.9 top-p = 0.95 top-k = 30 n-predict = 2048 batch-size = 256 threads = 8 

I noticed you're using HF cache model names as section names in your INI. Just FYI, you can actually use custom section names and point multiple presets to the same GGUF (--model or -m) file with different parameters.

Like :

[my-model-creative]
model = ~/.cache/llama.cpp/my-model.gguf
temperature = 0.9

[my-model-precise]
model = ~/.cache/llama.cpp/my-model.gguf
temperature = 0.1

[my-model-balanced]
model = ~/.cache/llama.cpp/my-model.gguf
temperature = 0.6

[ServeurpersoCom]

One way I found to have the same model more than once is to create a symlink to the model with a new name and use it, so I think that's a reasonable way to achieve what I said before and no more code needed.

Thank you for all your help ! 👍

This work also :)

[linuxmagic-mp]

Still think we need more 'defaults' automagically determined from the model itself, eg .gguf model type. Of course, llama-server shares cli options with other tools, and it's already getting too large of an ever changing CLI options.. Rather than thinking of this as an .ini, rather we should consider this a .loc override, for what the community agrees the recommended values are, or the model maintainers, via the information in the .gguf headers. Of course, certain systems may be 'constrained', so need lower values, or for experimental purposes, or because of LoRa layers at startup etc.. Suggest that we consider a more 'standard' location for config's that override .gguf header information.

…

On 2025-12-10 08:35, Pascal wrote: ./llama-server --port 8082 -ngl 999 -ctk q8_0 -ctv q8_0 -fa on --mlock -np 4 -kvu --jinja --models-max 1 --models-preset config.ini Easy .ini format : |[MoE-Qwen3-Coder-30B-A3B-Instruct] m = /path/to/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF/Qwen3-Coder-30B-A3B-Instruct-Q6_K.gguf temp = 0.7 top-p = 0.8 top-k = 20 min-p = 0 ctx-size = 131072 [MyModel] m = /my/other.gguf ... | The command-line arguments are inherited, and you're overriding them with custom configuration! — Reply to this email directly, view it on GitHub <#16938 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK24I7XL25KU5GLPHOKUVYL4BBDWPAVCNFSM6AAAAACK5APP4OVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMRSGIYDKMA>. You are receiving this because you commented.Message ID: ***@***.***>

-- "Catch the Magic of Linux..." ------------------------------------------------------------------------ Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com@linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Reg. TradeMark of Wizard Tower TechnoServices Ltd. ------------------------------------------------------------------------ 604-682-0300 Beautiful British Columbia, Canada

[mingodad]

I was thinking that way too but each user installation needs some adjusts due to their available hardware like for me with an mac mini m4 16GB I can run gpt-oss-20b.gguf only if I specify --no-mmap but others with plenty of memory don't need it, so for a
wide usage range several parameters need be set/overwrite by the end user.