llama.cpp supports the new gpt-oss model in native MXFP4 format

[ggerganov]

The new gpt-oss model is fully supported in native MXFP4 format across all major ggml backends, including CUDA, Vulkan, Metal and CPU at exceptional performance. This virtually brings the unprecedented quality of gpt-oss in the hands of everyone - from local AI enthusiasts to enterprises doing inference at the edge or in the cloud. The unique inference capabilities of ggml unlock a vast amount of use cases for the entire spectrum of consumer-grade hardware available on the market today - use cases that are impossible to support with any other inference framework in existence. Today, gpt-oss trained with the MXFP4 format, effectively “leaps” over the existing resource barriers and allows us to experience SOTA AI quality on our own personal devices.

The era of natively trained 4-bit local models has officially began and ggml will continue to lead the way forward!

Over the past 2 years the open-source developer community behind ggml has significantly grown. Together we built a scalable software infrastructure capable of supporting all of the needs of modern, low-level ML inference. More and more engineers, product builders, hardware vendors and researchers continue to discover and adopt what we have created. All of this wouldn’t have been possible without the dedicated open approach that this community has embraced from the very beginning of the project. The main difference today, compared to 2 years ago is that we are past the “hacking” phase of the development and it is time to focus on architecting and maintaining the correct implementation that will become the foundation of most local AI applications and products in the near future.

The primary goal of ggml-org will continue to be to help the community grow and create opportunities for everyone involved. We are more than ever open for support from the leaders in the AI field and today’s release is a prime example of what is possible with such coordinated and aligned efforts.

Special thanks to all maintainers, collaborators and contributors of ggml and related projects. Looking forward to many new developments together. Have fun!

[ahmed-adly-khalil]

YESSSSSSS :D

[bugmaschine]

It's just amazing to see how far the project has come. Thanks to everyone that makes this possible!

[ejones]

Great work!

[RoadToNowhereX]

What's the advantage of MXFP4 Quant? Does it has better performance than Q4_K_M or IQ4_XS Quant?

[Green-Sky]

It is the native format of the model. Looks like applying any quantization makes the model perform significantly worse.

[MacJedi42]

This is incredible :D! I thank you to all the developers involved

[gsgxnet]

Great achievement! Many thanks to all involved.

[axion66]

Incredibly awesome work!! Thanks to all the people making this possible. :)

[ServeurpersoCom]

Yessss! Awesome work 😍

[Bixente-san]

Super, is it the release b6101 (https://github.com/ggml-org/llama.cpp/releases/tag/b6101) ?

[YannFollet]

Thanks for the great job

[ElectriPixie]

That is amazing! You are the best!

[sorasoras]

How does applied MXFP4 to other model compare to other GGML quant？

[marvin-0042]

Amazing! Thanks for the great job!

Just curious, has anyone done any accuracy test for gpt-oss-20B on non-Nvidia platforms? Thanks!

[gsgxnet]

See other "issue" my results on Mac Mini M4 Pro. #15396 (reply in thread)

[ggzy12345]

thank you for the great job! Does anyone know how to set the number of experts? I want to set it to 2 (default is 4) for better performance. I can set this number in lm studio. But I do not know how to set it with llama.cpp

[ggzy12345]

ok. i find out myself. --override-kv gpt-oss.expert_used_count=int:2. But not sure the side effect.

[ryzen88]

Maybe i am misunderstanding, although when not trained in it, would it be a good format to quant to from a source that is still fp16? or is it only beneficial when trained natively in it?

[SlavikCA]

Is there any update info about MXFP4 quants?

Can it be included to Wiki here: https://github.com/ggml-org/llama.cpp/wiki/Feature-matrix

This week GLM-4.6 was released, now quants are available:

• UD‑Q4_K_XL (204 GB) from unsloth
• MXFP4 (199GB) https://huggingface.co/sm54/GLM-4.6-MXFP4_MOE

Any guidance on PROs & CONs of MXFP4 quants for this new model?