On the first point this was a pain point I faced as well (#15249)

For HF - they should follow similar path as Jan.ai does for example, a.k.a raw llama.cpp, not some weird forks like ollama.

On technology- yes, on marketing strategy, GTM, and partnership lock - there is nothing like Ollama's way of doing it. I can't open any app without ollama support to be the "local AI" support.

I use this as an amazing red flag.
Any project which has only Ollama as the AI client for local LLMs can be safely disregarded. Bonus points for not referring to Ollama at all.

A bit of both.

I think packaging has improved in recent times, windows as an example. I kinda like the containers approach for Linux, it makes sense, I'm not sure how many Linux distros are going to tolerate separate llama-cpp-cuda, llama-cpp-rocm, llama-cpp-vulkan packages as per their packaging guidelines.

I do think at a minimum llama.cpp should document upstream what flags should be used with each backend CUDA, Vulkan, Metal (--threads, --cache-reuse, --flash-attention, etc.), as discussed here:

ollama/ollama#11714 (comment)

People use sub-optimal flags often when using llama.cpp and assume performance isn't up to scratch.

Are you using the GGML_CPU_ALL_VARIANTS option?

Nope. Separate instructions builds for CPU instructions(For example: using -DGGML_AVX512=ON for AVX512 build). We even found people use processors which doesn't even have avx2.

You should look into GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS, it is designed to solve this problem. Most of our releases use it.

Thank you!

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

i hope they won't deprecate llama-server CLI for a llama-server GUI, it would be too annoying to use on termux or over SSH

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

I think I'm not that enough new to this, and I can compile llamacpp for a specific GPU, but I don't know how to use the llamacpp server for persistent chat-session data caching. Does it simply not have this feature? Or am I missing the parameter?

It's just moving to fast to be ready for a simple installer. Dont think a maintainer wants to keep up ;) But they have sure made it easy for anyone with basic skills in Linux. They have fixed so many compatability issues over the last year. It's pretty simple git pull, and cmake now. The hardest part if the python side of it. And yes, it takes some reading but it's about learning. You can even ask chatGPT on how to install it, the best parameters, what LLM's the would recommend for your hardware.. For most people, they want to learn.. Otherwise they would just get a subscription to some SaaS out there. So, rather than complaining, and putting the onus on the great ppl behind llama.cpp, roll up your sleeves.. However, in the end, a lot of people don't realize the heavy lifting comes in getting agentic tools working with your LLM..llams.cpp just made access to private LLM's possible to millions that want to 'play for free'. But for many, asl yourselves why you want your own LLM first.

llama-server within docker

It's technically possible, but the size of the package would be quite big.

Easiest way is with an αcτµαlly pδrταblε εxεcµταblε

https://justine.lol/ape.html

Easiest way is with an αcτµαlly pδrταblε εxεcµταblε

https://justine.lol/ape.html

The Actually Portable Executable (APE) for llama.cpp is llamafile, compiled with Cosmopolitan Libc: it can be bundled with the weights for an LLM (Windows has a maximum file size limit of 4GiB for executables, but the same APE can use an external .gguf), all in a single file that can run on six OSes (macOS, Windows, Linux, FreeBSD, OpenBSD, and NetBSD) and two CPU architectures (x86-64 and ARM64).

The winget package comes with the Vulkan backend.

Does that work reliably with CUDA cards and does it support the same kinds of models (Qwen2.5VL for instance)

My experience with the Vulkan backend on a RTX 3060 has been excellent.

This is the killer feature right here. Id use llama.cpp directly over any wrapper if this was added

I feel like point number 1 is widely undermentioned in this thread. The ease that Llama-Swap adds to being able to bounce around models for various tasks is invaluable. If it weren't for that, I would probably still be using Ollama. It still needs a bit of upfront setup, but once you get a model in, it basically becomes "set it and forget it."

Edit I just realized who I was replying to and felt the need to come back and say I love the YT channel man. Keep up the good work!

Docker Model Runner provides 1 and 2 FWIW... They are trying to grow the community also... It uses llama.cpp upstream, contributes any changes required back... Number 3... I recommend AnythingLLM, but there are many options

Nice, but please oh please, don't encourage people to blindly run "curl foo | sh", as this is really bad and it allows to execute truncated contents (e.g. think about what happens with sudo rm -rf /tmp/${TEMP} when the transfer was interrupted just after the first '/'). Please better refer to the commands as curl foo > run.sh && sh run.sh that is not significantly longer and supports error checking.

Thanks for the feedback!

I know some people disapprove the pipe to shell thing, but I believe the risks are often overstated and alternatives often just provide a false sense of security (but I dont want to debate here ^^).

That said, there is a real issue with trust in the current project but my ultimate goal is to host this on ggml.ai to address the trust aspect (at least, that’s my hope).

Regarding the issue you mentioned, you dont need to have the intermediate run.sh file. There is a much better way to handle that and you can check out the code to see it.

I have seen your script already and itself is not subject to this risk. It doesn't change the fact that the vast majority of people who get trained to blindly copy-pasting "curl|sh" don't have the skills to do that, and if they'd read the script in the first place they wouldn't need to download it again since they would already have it, and it's not responsible to encourage this bad practice of copy-pasting whatever this way without even checking. Instead, spreading the message that a download might fail and that it can still be checked in a one-liner at least raises their attention on this matter. It will not solve every risk but participates to not conveying a very bad incite to do bad things.

I understand your points, but we cannot ignore that curl | sh has become a thing. It is now widely used, and the majority of users enjoy it.

Education, as always, is the key. People should be aware of the risk of piping untrusted things to the shell. That's the best. Even experts can be so badly tricked.

But the intermediate run.sh definitely won't help a beginner and doesn't improve security at all. While keeping a temporary file that the user could call later without being updated would be worse. I highly recommend not doing that...

we cannot ignore that curl | sh has become a thing. It is now widely used, and the majority of users enjoy it.

it's widely used among bad projects unfortunately, and users neither enjoy nor hate it, they copy-paste without understanding what they're doing.

But the intermediate run.sh definitely won't help a beginner

sure but it doesn't train them to do bad things without understanding. And if they're curious at the end they can read the script to understand how it works.

and doesn't improve security at all

not security but safety as a general case (not running a truncated script). And by now it has been sufficiently documented as a bad and harmful practice so that experienced developers cannot claim to ignore it anymore.

I highly recommend not doing that...

a lot of security-conscious people are trying hard to stop their users and friends from blindly running the curl|sh thing :-(

installama.sh supports CUDA acceleration for Linux without having to install CUDA.
I'll do Windows soon (after the Vulkan backend).

That's great news. So here's what I am seeing from various threads, please correct any misconceptions:

• Apple (the easy case): brew install will get you llama.cpp with GPU support via Metal
• Windows: winget installs with Vulkan backend support today. CUDA support via installama.sh upcoming. Any limitations for winget/installama.sh (e.g. CPU type, etc.)?
• Linux: installama.sh supports CUDA, unclear on nix

The documents I see that need updating include:

• README.md: add mention of installama.sh, or point to install.md
• Install.md: add sentences on GPU support, add installama.sh

Agree? I'd be happy to do the doc updates if someone will email me how (git wasn't even around when I stopped coding...😱)

installama.sh is still quite new, we need more testing and Windows CUDA support before we can really brag about it :)

The list of supported systems and corresponding requirements can be found here:
https://github.com/angt/installama.sh/blob/master/REQUIREMENTS.md

I actually like LM Studio, I never saw the point of Ollama. In some ways (env variables), it's harder to use than llama.cpp.

A couple of months ago I spent about a half a day going back and forth with Opus 4 to generate an LLM instruction document to load, update and test a llama.cpp installation and build in WSL2. I use it now about every 2-3 weeks and it has worked flawlessly. I originally built from source because the pre-built packages didn't fully support the Blackwell cards. Now I just continue this way because I can Vibe-Update without thinking about it much. The last time I updated, I used Gemini 3.0 Flash and it worked great, and now that GitHub copilot works with standard skills files, I will likely rebuild it to be a formal skill.md.

@ibehnam why not curl installama.sh | sh ?

..also, 2 hours?!

@ibehnam it because you have dual GPUs and not plain Jane single Apple Mx that you need to resort to a compile?

The final hitch to consider is the version. Using these methods, with the current release process, users would be updating the software 10 times a day. Especially with big binaries, that's not sustainable. The Homebrew package already skips every 10 releases because of this. I think llama.cpp should adopt a more formal tiered versioning system, where the dev branch releases constantly, but a "stable" branch releases maybe once a week at most. This will also help people understand what's changed and make it easier to say if some other software (or model) works with your llama.cpp or not. Bleeding edge 24/7/365 isn't a viable release strategy.

I still can't understand how, after so much time, people who have been able to build an engine are incapable of building a car, given that the engine is the main part of a car, but oh well.

It's better to stop dwelling on the issue, as there are many other people who have already solved this.

If we look at how Jan was built (around Llama.cpp), we simply have a frontend client that detects the characteristics of the hardware where it has been installed, and then downloads the updated binaries according to the hardware along with the version of llama.cpp compatible with those binaries.

If there are problems compiling these multiple binaries for specific hardware, it should be automated using prepared pipelines, along with three frontend client pipelines for the operating systems most used by the community (MacOS, Windows, Ubuntu/Debian).

If there are cost or time issues in making the releases (due to the speed of development), simply limit the number of distributions to the public to 1 or 2 per month and offer manual compilations to developers.

This problem has already been solved, and companies generally have a software architect in charge of aligning all of this.

The car engine is designed by car engineers to produce a given output, pretty much in isolation of the rest of the car. That's part of why you can put the same engine in a sports car, sedan, SUV and pickup truck. They need 2 things: 1) extremely simple, loosely coupled abstractions, and 2) an entire team to build every independent system, then a different team just to combine them all into one vehicle.

Most companies don't delve into the hardware very deeply because it's a huge amount of complexity. They might go into supporting SDK X, or driver Y, or version Z, as that's a lot less complex than tracking specific hardware.

Ollama's install script for example. They play fast and loose with what consistutes detection of a given driver, platform, hardware vendor. They depend on specific tools, specific vendor binaries (made for specific platforms), even specific PCI IDs. This works for a short time, but very soon it's out of date and you have to constantly keep up to date on everything, or have a very good support pipeline for all the bug reports coming in. You end up supporting a majority of whoever complains the most and dropping support for old platforms when people stop complaining. This requires more people time to keep updating things.

A single Linux distribution will burn hundreds (if not thousands) of hours every year to keep track of all the graphics hardware and software, just to try to keep baseline device support. And it's always behind, because you only find out what's broken and what needs supporting well after it's released and users try to use it and it breaks.

Things work easiest when all devices & software supports some abstraction, the platform ships it, and you shared-link against it. But graphics cards have always struggled with abstractions across platforms (especially within Linux). Vulkan is still changing and sort of new, and the rest is all vendor-specific, custom, changing. Since the abstractions are difficult, support is gonna be difficult too.

Llama.cpp is already shipping binaries in GitHub for a couple platforms, so they're kinda doing what you're suggesting? But it's not easy to use, and it doesn't support everything. (And it's definitely not a solved problem, or I wouldn't have had to spend hours and hours wrestling to figure out how my iGPU works enough to get Ollama to run correctly).

I think it can get better. But there's a certain amount of necessary complexity which requires a minimum amount of effort to overcome, and my guess is they don't have the staff (or volunteers) for it yet, plus a pinch of analysis paralysis and conflicting objectives (this project is clearly not intended to be easy to use; the command arguments alone, my "Bob"...).

There are a lot of good points in this post @peterwwillis, and I have even seen manufacturers themselves (never mind open source) struggle with their own software due to lack of abstractions with their own hardware!

The question in my mind however is your statement:

Llama.cpp is already shipping binaries in GitHub for a couple platforms, so they're kinda doing what you're suggesting? But it's not easy to use, and it doesn't support everything.

Do Llama.cpp binaries support enough of the fat head that non-developers with some programming background, ones who can twiddle shell scripts and understand brew, pip, etc. in more detail than parroting CLI commands, can install Llama.cpp with GPU support using a cookbook method (brew, winget, or 5-10 CLI commands?) Too often we shift problems to developers, when documentation goes most of the way.

Ideally the documentation is clear enough that Gemini / Claude / ChatGPT can generate the how to for the average user. Last time I checked it could not.

Do Llama.cpp binaries support enough of the fat head that non-developers [..] can install Llama.cpp with GPU support using a cookbook method

It seems the lack of Linux CUDA is the biggest stumbling block for initial adoption with a cookbook. Other people are providing builds, which is great, but it would be greater if llama.cpp did too 😄

But one focus of OP's thread is all the ways llama.cpp is harder to use than alternatives (Ollama, LMStudio). If llama.cpp wants to be that easy, there is a lot of work to do. (which I think the community would gladly help with, but there needs to be an official person saying 'yes please contribute XYZ in way ABC', and then someone can start working on a PR)

In the latest version of the official Debian / Ubuntu deb packages, we have added a systemd unit in order to start and monitor llama-server (disabled by default).

It means that running llama-server is now as simple as:

sudo apt install llama.cpp-tools
# sudo apt install libggml0-backend-vulkan # additional backends
sudo systemctl start llama-server.service

By default the service is in multi-models router mode and only available on localhost, but this can be configured in /etc/default/llama-server (along with all other options which support environment variables).

These packages are on their way to the various backports repositories. Meanwhile, you can use them from the Debian AI team repositories following these installation instructions. Feedback welcome in this discussion.

Debian packages of the ggml CUDA backend built against the Nvidia-provided repositories (vs the official Debian repositories) are now available. They can make deployment of llama.cpp easier in cloud instances or on recent Nvidia GPUs.

Installation instructions are available here:
https://salsa.debian.org/deeplearning-team/ggml/-/blob/debian/unstable/debian/README.vendor-cuda.md

More details, and feedback welcome in this discussion.

This seems unrelated to the discussion at hand. Please open a separate issue for better visibility. Please mention which CPU you're using too.

My current difficulty is integrating all the inference acceleration backends, but it seems quite challenging at the moment.
The core difficulty lies in the fact that libllama, libdiffusion, and Whisper all currently use static linking of ggml, especially stable-diffusion.cpp.I am adjusting the custom build