[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗

[Vaibhavs10]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - brew install llama.cpp works
2. For Linux (CUDA) - they need to clone and install directly from github
3. For Windows - winget (?)

This adds barrier for non technically inclined people specially since in all the above methods users would have to reinstall llama.cpp to get upgrades (and llama.cpp makes releases per commit - not a bad thing, but becomes an issue since you need to upgrade more frequently)

Opening this issue to discuss what could be done to package llama.cpp better and allow users to maybe download an executable and be on their way.

More so, are there people in the community interested in taking this up?

[KhazAkar]

From this, IMO it only misses Linux+CUDA bundle to be useable as download & use.

Click to expand

If we want better packaging on Linux, we can also work on snap/bash installer when trying to use pre-built packages.

[michaelgiba]

On the first point this was a pain point I faced as well (#15249)

[maxwell-kalin]

"snap" the thing from ubuntu?

[mitkox]

It’s high time HuggingFace to copy Ollama’s packaging and GTM strategy, but this time, give credit to llama.cpp. Ideally, we should retain llama.cpp as the core component.

[KhazAkar]

For HF - they should follow similar path as Jan.ai does for example, a.k.a raw llama.cpp, not some weird forks like ollama.

[mitkox]

On technology- yes, on marketing strategy, GTM, and partnership lock - there is nothing like Ollama's way of doing it. I can't open any app without ollama support to be the "local AI" support.

[crodjer]

I use this as an amazing red flag.
Any project which has only Ollama as the AI client for local LLMs can be safely disregarded. Bonus points for not referring to Ollama at all.

[slaren]

Is the barrier the installation process, or the need to use a complex command line to launch llama.cpp?

[ericcurtin]

A bit of both.

I think packaging has improved in recent times, windows as an example. I kinda like the containers approach for Linux, it makes sense, I'm not sure how many Linux distros are going to tolerate separate llama-cpp-cuda, llama-cpp-rocm, llama-cpp-vulkan packages as per their packaging guidelines.

I do think at a minimum llama.cpp should document upstream what flags should be used with each backend CUDA, Vulkan, Metal (--threads, --cache-reuse, --flash-attention, etc.), as discussed here:

ollama/ollama#11714 (comment)

People use sub-optimal flags often when using llama.cpp and assume performance isn't up to scratch.

[qnixsynapse]

[slaren]

Are you using the GGML_CPU_ALL_VARIANTS option?

[qnixsynapse]

Nope. Separate instructions builds for CPU instructions(For example: using -DGGML_AVX512=ON for AVX512 build). We even found people use processors which doesn't even have avx2.

[slaren]

You should look into GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS, it is designed to solve this problem. Most of our releases use it.

[qnixsynapse]

Thank you!

[simonw]

For me the biggest thing is llama-server. That tool is fantastic... but it has very low discoverability. Until a few months ago I still thought it was just a demo because it lived in the examples/ folder (I just noticed it moved from there to the tools/ folder in May).

I'd love to see more emphasis placed on llama-server. It's really good! I think it's probably the most end-user appropriate way to interact with this project.

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux, which on Mac and Windows work like a desktop application: you get an icon you can use to launch the tool which start the server running and launch a window that shows that web UI.

Maybe include systray integration and a simple UI for selecting and downloading models too.

At that point llama-server would feel like an alternative to Ollama and LM Studio and Jan. I think it deserves that - it has most of what those tools offer implemented already, what's missing is an installer and a think desktop shell.

[t1u1]

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux,

An easy way to do this is to make it a PWA. Will work in most major browsers, including Safari, Chrome and its derivatives, Android and iOS, but not Firefox. It just takes a few lines of code to do this.

[allozaur]

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

[calculatortamer]

i hope they won't deprecate llama-server CLI for a llama-server GUI, it would be too annoying to use on termux or over SSH

[red-co]

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

I think I'm not that enough new to this, and I can compile llamacpp for a specific GPU, but I don't know how to use the llamacpp server for persistent chat-session data caching. Does it simply not have this feature? Or am I missing the parameter?

[linuxmagic-mp]

It's just moving to fast to be ready for a simple installer. Dont think a maintainer wants to keep up ;) But they have sure made it easy for anyone with basic skills in Linux. They have fixed so many compatability issues over the last year. It's pretty simple git pull, and cmake now. The hardest part if the python side of it. And yes, it takes some reading but it's about learning. You can even ask chatGPT on how to install it, the best parameters, what LLM's the would recommend for your hardware.. For most people, they want to learn.. Otherwise they would just get a subscription to some SaaS out there. So, rather than complaining, and putting the onus on the great ppl behind llama.cpp, roll up your sleeves.. However, in the end, a lot of people don't realize the heavy lifting comes in getting agentic tools working with your LLM..llams.cpp just made access to private LLM's possible to millions that want to 'play for free'. But for many, asl yourselves why you want your own LLM first.

[yaronsumel]

It will be cool if 'llama-server' would have auto configuration option to the machine/model like 'ollama' does it.

[exxocism]

For windows maybe choco and windows store would be a good idea? 🤔

[acbits]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - `brew install llama.cpp` works 2. For Linux (CUDA) - they need to clone and install directly from github 

I created a rpm spec to manage installation though I think flatpaks might be more user friendly and distribution agnostic.

[schnow265]

For Windows - winget (?)

The released Windows builds are available via Scoop.

Updates happen automatically. Old installed versions are kept, and current one symlinked into a folder „current“ which provides the executables on the path.

[damianofalcioni]

is it feasible to have a single release for OS including all the backend?

[slaren]

It's technically possible, but the size of the package would be quite big.

[SamuelMarks]

Easiest way is with an αcτµαlly pδrταblε εxεcµταblε

https://justine.lol/ape.html

[MaD70]

Easiest way is with an αcτµαlly pδrταblε εxεcµταblε

https://justine.lol/ape.html

The Actually Portable Executable (APE) for llama.cpp is llamafile, compiled with Cosmopolitan Libc: it can be bundled with the weights for an LLM (Windows has a maximum file size limit of 4GiB for executables, but the same APE can use an external .gguf), all in a single file that can run on six OSes (macOS, Windows, Linux, FreeBSD, OpenBSD, and NetBSD) and two CPU architectures (x86-64 and ARM64).

[tellsiddh]

For linux I just install the vulkan binaries and run the server from there. Maybe we can have a install script like ollama that detects the system and launches the server which can be controlled from an app as well as cli? The user then gets basic command line utillities like run start stop load list etc?

[leoxzhao]

On Mac, the easiest way (also arguably the safest way) from a user's perspective is to find it in App Store, and install from there. Because of apps from App Store are in a sandbox, so from a user's point of view, installing or uninstalling is simple and clean. Creating a build and passing the App Store review might take some efforts (due to the sandbox constraint), but it should be a one-time thing.

[O-J1]

Its my understanding that none of the automated installs support GPU acceleration. I might be wrong but its definitely the case for Windows, which makes it useless to install via winget.

[slaren]

The winget package comes with the Vulkan backend.

[O-J1]

Does that work reliably with CUDA cards and does it support the same kinds of models (Qwen2.5VL for instance)

[anderspitman]

My experience with the Vulkan backend on a RTX 3060 has been excellent.

[henk717]

To me the biggest advantage ollama currently has is that the optimal settings for a model are bundled, the gguf spec would allow for this to since its versatile enough to make this a metadata field inside the model. It would allow people to load the settings from a gguf and frontends can extract them and adapt them as they see fit. I think that part is going to be more valuable than obtaining the binary since downloading the binary from github is not that hard.

[logan-markewich]

This is the killer feature right here. Id use llama.cpp directly over any wrapper if this was added

[digitalspaceport]

My personal wishlist

1. Llama-swap integration
2. Catchall compiled binaries for linux, win, osx
3. GUI improvements

[neutrinotek]

I feel like point number 1 is widely undermentioned in this thread. The ease that Llama-Swap adds to being able to bounce around models for various tasks is invaluable. If it weren't for that, I would probably still be using Ollama. It still needs a bit of upfront setup, but once you get a model in, it basically becomes "set it and forget it."

Edit I just realized who I was replying to and felt the need to come back and say I love the YT channel man. Keep up the good work!

[ericcurtin]

Docker Model Runner provides 1 and 2 FWIW... They are trying to grow the community also... It uses llama.cpp upstream, contributes any changes required back... Number 3... I recommend AnythingLLM, but there are many options

[bugparty]

I think we should focus on what it is now, non-tech users will use Ollama or LMStudio anyway.

[angt]

Hey everyone!

I’ve been working on some scripts and tools to make it super easy to download and run llama-server on any Linux machine (not just Ubuntu) and for both aarch64 and x86_64.

Feature detection is done before downloading to reduce download time, and CUDA is supported as well, though not fully optimized yet. This is also the only way to provide optimized builds for aarch64.

Right now, models are downloaded with hf download, to make things as easy, like Ollama.
That means the hf CLI is required for now.

On Linux, you can try it with:

curl -s https://angt.github.io/installama.sh | sh -s -- qwen3-4b:Q4_0 

This should start a server running the requested model.

I’d love to hear your thoughts on this kind of packaging approach for Linux.
I’m currently exploring macOS with Metal support as the next step.

[cheery]

The scripts you've written look super clean and easy to understand. The script probes with target-features that is super simple. It's got a similar program called cuda-probe that does the same for cuda. And it's necessary that they're coming first. You've created lot of binaries for different platforms and likely have the scripts in place to update them automatically.

It looks like bare minimum to get it up and running. I personally like bare minimums. I would even throw away the 'hf' and let people decide how they download their huggingface models. You've already done the bulk difficult job when you've compiled it all for them and feature-recognize which version needs to be downloaded.

Then if the community creates frontends, maybe, for instance for linux with python rich & argparse -modules so that it's nice and colorful and running in terminal. That might make it full turn-key solution. Or if you really insist writing more complex bash scripts, then that stuff.

TL;DR I like it!

[0cc4m]

Please keep in mind that piping scripts directly from a web page into your shell is inherently unsafe and you (the user) are responsible for checking the contents of the script before running it. It's very easy to do something unwanted this way.

[rodrigovimieiro]

For Linux (CUDA) - they need to clone and install directly from github

It would be nice to have binaries for Ubuntu with CUDA, even though the one with Vulkan works pretty well

[angt]

This exactly what installama provides: https://huggingface.co/datasets/angt/installamacpp-cuda/tree/main.
They should work on all recent Linux distributions.

[pandruszkow]

It would be great if GGML_CPU_ALL_VARIANTS=ON binaries with CUDA support were included in this. I have an old Ivy Bridge machine for llama.cpp inference, and it would make my life so much easier if I could just download a ready-to-go binary that will work with Nvidia cards and not require AVX2 or FMA, but take advantage of AVX and F16C.

[angt]

Yes for CUDA i let the llama.cpp defaults to wait for comments and check if the current values were the good ones :)
I believe I can reduce to AVX without causing significant performance degradation (need to check).

[angt]

@pandruszkow I've updated the Linux CUDA binaries compiled with -march=x86-64-v2. To test them, please delete your ~/.installama directory and rerun the command. It would be awesome if you could check that they work on your setup :)

[angt]

Hey!!

Apple M1 through M4 are now supported with Metal. It turned out to be much easier than expected.
As before, just run:

curl -s https://angt.github.io/installama.sh | sh -s -- qwen3-4b:Q4_0 

and you’re good to go 🚀

Again hf is still required to download the models, i'm going to work on that.

[ggerganov]

Do you mean use libhttp for the file downloads?

Generally, dropping libcurl dependency would be great. But last time I looked into this, we still need libssl dependency which ironically seems even more difficult to provide compared to libcurl. But if you have ideas how to improve this, feel free to give it a try. Less dependencies is better.

[angt]

Yes, this is exactly what I had in mind. I've opened a PR to explore this: #16185

[angt]

All builds have been updated with the changes from PR #16185.
Now you can run:

curl -s https://angt.github.io/installama.sh | sh -s -- -hf unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_0 

[ggerganov]

Interesting approach! I'll take a look at the PR soon.

Btw, having an installer that does not require model to be specified would be also useful. I suppose this is currently like this for ease of use, but I think eventually the installer should run without setting a model.

[angt]

True! I've updated the script, so now you can run it without any arguments:

$ curl -s https://angt.github.io/installama.sh | sh Run ~/.installama/llama-server to launch the llama.cpp server 

It's still an early PoC, so everything can be changed or improved :)

[dlmsf]

in my nodejs project where have the llama.cpp in textgeneration module, i use like that

bash install.sh
ai | chat | generate | webgpt

working in ubuntu and alpine, maybe adding support to all distros can be a good option, direct and minimalistic

[deadprogram]

I just setup this build process for precompiled binaries of llama.cpp for Ubuntu 24.04 with NVidia 12.9 drivers.

Just in case it helps anyone else 😸

https://github.com/hybridgroup/llama-cpp-builder

[Danmoreng]

Maybe interesting for someone who wants to build from source under Windows. I made a Powershell Script which installs all the necessary prerequisites for a CUDA build, pulls the repo and builds it.
https://github.com/Danmoreng/llama.cpp-installer

[rosmur]

First thanks for this initiative - I think making it easier for more folk to use llama.cpp directly instead of using commercial middleware like LMStudio, ollama is a win-win for everyone.

I think the docs can use a lot of improvement to make them more accessible, organized, more complete, more examples etc. (this will in turn also help AI search) I started this a bit here: #15709

Happy to contribute more if it would be useful. Even asking AI to cleanup, reorganize etc. the current docs would be a net positive

[lakano]

As a downstream consumer, I would love to have a binary release of hip-radeon (ROCm) for ubuntu, as you already do for windows. In the same time, there is still a missing docker image for this backend ( #11913 ). So we are still forced to build manually the source at each release.

[lakano]

I've just tested, it's works! Thank a lot ! I'm very surprised because you said you didn't do the trick of copying libs. Which version of ROCm did you uses, may be the pre-release of 7.9 ?

And it's very pleasant because with exactly the same model and the same CLI options as on my builded version 7.0.2, my t/s have increased from 61 t/s to ~ 77 t/s !

[lakano]

@angt Do you believe that we can enjoy your work in the release page soon? :-)

[angt]

I’m not sure people will be happy to see hundreds of binaries in the release page and that’s a key part of the optimisation and a faster setup.

But I will try to make it more official with something like ggml.ai/installama.cpp, that would be great :)

Also, I think we cannot provide all releases, we need to be more selective and do updates every week or 2 weeks.

[lakano]

I understood your concerns and this will still be better than the current situation, thank you :-)

[Danmoreng]

From an end-user perspective, an installer that detects the current systems hardware and prompts the user which kind of version he wants to install would make sense imho.
For example, if it detects an AMD GPU, it prompts the user -> do you want to install ROCm, Vulkan or a CPU only build?

[SamuelTallet]

For users and in particular those using Windows (since there is already the LlamaBarn project, which is very user-friendly under macOS), it would be nice to have a homepage with a single download button for a Windows application.

One has to try to put in the users' shoes; selecting the right release on GitHub can already be a barrier for some. So once installed, this desktop application should detect the most powerful accelerator available locally and download the appropriate binary.

About curated GGUFs, a HuggingFace API could provide the models both trending and suitable for the computing power available locally.

TL;DR: It could be a WebView2-based project that:

• downloads the best (and latest) binary,
• asks the user to select a model (curated or not),
• starts a llama-server with the selected model,
• displays the WebUI embedded in the server.

[MarcusDunn]

I think delegating installation to external projects is entirely legitimate. The end "product" of llama.cpp being the C headers + implementation is an acceptable place for this project to be. I would much rather have the maintainers of llama.cpp not have to deal with the deluge of installation and packaging idiosyncrasies.

I understand a lot of people who are posting issues on this project are users of the server or CLI, but I think there's a sampling bias where most users of this repo are actually through llama-cpp-python, ollama, or some other wrapper (my own being llama-cpp-rs). I do not think this should be considered a problem to be solved.

[angt]

Hi everyone!

I’ve just done a major refactor of the build process and published all the work here:

https://github.com/angt/installama.sh

All fixes and new features (like the new downloader or the OpenSSL/BoringSSL backend) are either merged or on their way upstream. Let’s see how far we can go in this direction :)

For now, let’s use this new repo to track issues and feature requests.

[wtarreau]

Nice, but please oh please, don't encourage people to blindly run "curl foo | sh", as this is really bad and it allows to execute truncated contents (e.g. think about what happens with sudo rm -rf /tmp/${TEMP} when the transfer was interrupted just after the first '/'). Please better refer to the commands as curl foo > run.sh && sh run.sh that is not significantly longer and supports error checking.

[angt]

Thanks for the feedback!

I know some people disapprove the pipe to shell thing, but I believe the risks are often overstated and alternatives often just provide a false sense of security (but I dont want to debate here ^^).

That said, there is a real issue with trust in the current project but my ultimate goal is to host this on ggml.ai to address the trust aspect (at least, that’s my hope).

Regarding the issue you mentioned, you dont need to have the intermediate run.sh file. There is a much better way to handle that and you can check out the code to see it.

[wtarreau]

I have seen your script already and itself is not subject to this risk. It doesn't change the fact that the vast majority of people who get trained to blindly copy-pasting "curl|sh" don't have the skills to do that, and if they'd read the script in the first place they wouldn't need to download it again since they would already have it, and it's not responsible to encourage this bad practice of copy-pasting whatever this way without even checking. Instead, spreading the message that a download might fail and that it can still be checked in a one-liner at least raises their attention on this matter. It will not solve every risk but participates to not conveying a very bad incite to do bad things.

[angt]

I understand your points, but we cannot ignore that curl | sh has become a thing. It is now widely used, and the majority of users enjoy it.

Education, as always, is the key. People should be aware of the risk of piping untrusted things to the shell. That's the best. Even experts can be so badly tricked.

But the intermediate run.sh definitely won't help a beginner and doesn't improve security at all. While keeping a temporary file that the user could call later without being updated would be worse. I highly recommend not doing that...

[wtarreau]

we cannot ignore that curl | sh has become a thing. It is now widely used, and the majority of users enjoy it.

it's widely used among bad projects unfortunately, and users neither enjoy nor hate it, they copy-paste without understanding what they're doing.

But the intermediate run.sh definitely won't help a beginner

sure but it doesn't train them to do bad things without understanding. And if they're curious at the end they can read the script to understand how it works.

and doesn't improve security at all

not security but safety as a general case (not running a truncated script). And by now it has been sufficiently documented as a bad and harmful practice so that experienced developers cannot claim to ignore it anymore.

I highly recommend not doing that...

a lot of security-conscious people are trying hard to stop their users and friends from blindly running the curl|sh thing :-(

[rbberger]

Hey everyone, just want make you aware that I started adding llama.cpp into Spack. See spack/spack-packages#2437

[renet10]

Great to see the continued activity on this topic.

As an outside observer / newcomer, llama.cpp simply feels to me like hard-core development software. People like me who come from a development background but haven't seen a Makefile in decades are forced into LM Studio / Ollama because it takes way too long to retrain atrophied muscles, even when we prefer the elegance of minimally necessary software solutions (i.e. no shim layers).

We also run into continued misinformation. For example, this is what Google returns when I search for "how to install llama.cpp with gpu support" (excerpted)

Prerequisites
An NVIDIA GPU with up-to-date drivers.
The NVIDIA CUDA Toolkit installed. You can download it from the NVIDIA developer website.
Basic build tools like cmake, ninja, a C++ compiler (GCC on Linux, Visual Studio with C++ workload on Windows), and git.
A Python environment (e.g., using conda or a venv).
...

Even in this thread, a recent commenter thought (erroneously) that "none of the automated installs support GPU acceleration".

Reading through this thread and others (like #8188) it seems like llama.cpp is already most of the way to a decent packaging strategy. From what I can tell, it seems a few added lines into install.md plus some documentation structuring will go a long way.

[angt]

installama.sh supports CUDA acceleration for Linux without having to install CUDA.
I'll do Windows soon (after the Vulkan backend).

[renet10]

That's great news. So here's what I am seeing from various threads, please correct any misconceptions:

• Apple (the easy case): brew install will get you llama.cpp with GPU support via Metal
• Windows: winget installs with Vulkan backend support today. CUDA support via installama.sh upcoming. Any limitations for winget/installama.sh (e.g. CPU type, etc.)?
• Linux: installama.sh supports CUDA, unclear on nix

The documents I see that need updating include:

• README.md: add mention of installama.sh, or point to install.md
• Install.md: add sentences on GPU support, add installama.sh

Agree? I'd be happy to do the doc updates if someone will email me how (git wasn't even around when I stopped coding...😱)

[angt]

installama.sh is still quite new, we need more testing and Windows CUDA support before we can really brag about it :)

The list of supported systems and corresponding requirements can be found here:
https://github.com/angt/installama.sh/blob/master/REQUIREMENTS.md