support GLM-4.5V and GLM-4.1V vision models

[ddh0]

Add support for zai-org/GLM-4.5V and zai-org/GLM-4.1V-9B-Thinking vision models to llama.cpp. I currently only plan to support images + text, no video inputs in this PR.

The architecture is Glm4vMoeForConditionalGeneration ("model_type": "glm4v_moe") / Glm4vForConditionalGeneration ("model_type": "glm4v"). Internally, these consist of an LLM (text model) and a ViT (vision adapter / multimodal projector):

LLM

• Based on GLM-4.5-Air / GLM-4-9B-0414
• Tensor names start with model.language_model.
• Uses a "multimodal 3D RoPE" - in apply_multimodal_rotary_pos_emb, it applies rotary embeddings across temporal, height, and width dimensions for visual tokens

ViT

• Adapted from apple/aimv2-huge-patch14-336:
  • Architecture Aimv2VisionModel
  • ~681M params
  • 24 layers
  • hidden_size (n_embd): 1536
  • intermediate_size (n_ff): 4096
  • image_size: 336
  • patch_size: 14
  • num_channels: 3
  • depth: 24
• Tensor names start with model.visual.
• Its 2D positional embeddings are dynamically adapted via bicubic interpolation within the Glm4vMoeVisionEmbeddings module to handle varied image resolutions
• It also applies its own rotary position embeddings within the self-attention blocks (via apply_rotary_pos_emb_vision)
• Same for both models

Other notes:

• Native context length is 65,536 (as opposed to 131,072 for GLM-4.5-Air)
• RoPE theta (θ): 10,000.0 (as opposed to 100,000.0 for GLM-4.5-Air)
• The model supports video input, but I currently do not plan to support video input in this PR (images only)
• Tokenizer has video-related special tokens - need to handle these during conversion

References:

• The 🤗 reference implementations:
  • modeling_glm4v_moe.py
  • modular_glm4v_moe.py
• The 🤗 model card
• The 🤗 config.json

See also:

• arXiv:2507.01006
• arXiv:2411.14402

[ddh0]

So, it turns out that vision in this model is based on Qwen3-VL, which still needs support from llama.cpp. I am pretty familiar with llama.cpp in general but not with mtmd, so I may not be able to get this PR done on my own. I will keep trying to hack at it when I have time, and I would appreciate any help I could get. :)

Also just saw this thread (#16207) in which someone has posted a patch to get Qwen3-VL kinda-sorta-working in llama.cpp. I will take a look at that too and see if it is helpful - it might make more sense to get Qwen3-VL to a working state in llama.cpp first and only then start working on this PR on top of that. Not sure, just thinking out loud.

[rujialiu]

Thanks for your work! @ddh0
Based on the commit history, the imports of qwen3vl is the result of a "use qwen data class to avoid repeat again" refactor, so probably it's not quite "based on Qwen3-VL". But anyway, I'm planning to dive into Qwen3-VL and GLM-4.5V later this month and I hope I can help.

[ddh0]

I'm planning to dive into Qwen3-VL and GLM-4.5V later this month and I hope I can help.

Thank you @rujialiu! I suspect your understanding of the mtmd side of things is better than mine - I could use some guidance on what the next steps should be. I've paused working on this for now until I have a better understanding of what exactly needs to be done.

Also cc @ngxson (llama vision expert :))

[rujialiu]

Thank you @rujialiu! I suspect your understanding of the mtmd side of things is better than mine - I could use some guidance on what the next steps should be. I've paused working on this for now until I have a better understanding of what exactly needs to be done.

I have 0 understanding of mtmd before tackling the "inaccurate bbox" issue 😄
Then many people helped me along the way. So let's do/learn things together!

[rujialiu]

@ddh0 I asked Claude Sonnet 4.5 to carefully inspect transformers implementation and tell me the differences between Qwen2.5-VL, Qwen3-VL and GLM4V (not a single question. I asked several more specific questions and inspected the codes it gave me). I'm not knowledgable enough to check every detail, but it looks good to me. In short, GLM4V is very close to Qwen2.5-VL:

• Same chunked RoPE (though different names: MRople vs 2D-Rope/3D-Rope). glm4v's apply_multimodal_rotary_pos_emb even refers to qwenvl's paper
• Same max(t,h,w) logic
• Same window attention/patch merging (because it re-uses Qwen2_5_VLVisionAttention and Glm4vVisionPatchMerger. But I haven't checked this carefully)
• (new) Learnable embeddings + bicubic interpolation (search Perform bicubic interpolation comment)
• Few different constants, like hidden_size, rms_norm_eps etc

It's so similar to Qwen2.5-VL, but why the code re-uses qwen3_vl_moe? It's because Qwen2.5-VL doesn't have moe version 😄
Maybe we only need to wire up GLM4.5V's LLM with its vision encoder in the most obvious way and we're done.

So I guess it's ok to resume the work directly, based on https://github.com/FMayran/llama.cpp/tree/QwenVL-causal-fix

It should be easy to adapt to whatever "llama_batch improvement" is merged into master later.

BTW: Can we make sure the dense version (GLM-4.1V-9B-Thinking #14495 ) is working first? It's much smaller, easier to compare result with transformers, and it looks like GLM-4.5V is no different besides the LLM part.

[ddh0]

Thank you @rujialiu, that's all very helpful. I will take a look at GLM-4.1V-9B-Thinking and see if it can be incorporated into this PR.

Is there a PR associated with the branch you linked (FMayran:QwenVL-causal-fix)? If not, maybe we should start a separate PR to fix the vision implementation of these families of models in general and then come back to this PR to finish the model-specific parts.

[rujialiu]

Thank you @rujialiu, that's all very helpful. I will take a look at GLM-4.1V-9B-Thinking and see if it can be incorporated into this PR.

Is there a PR associated with the branch you linked (FMayran:QwenVL-causal-fix)? If not, maybe we should start a separate PR to fix the vision implementation of these families of models in general and then come back to this PR to finish the model-specific parts.

Of course! Hopefully @ngxson will find some time to fix the general problem (adding an internal token index for casual check). Since you're familiar with LLM part, you can take a look our discussion in #15474 (the quickiest way is to read in a bottom-up order until you understand). The issue and solution is conceptually very simple, but I'm not brave/skillful enough to touch llama-batch.h 😄

[FMayran]

Thank you @rujialiu, that's all very helpful. I will take a look at GLM-4.1V-9B-Thinking and see if it can be incorporated into this PR.

Is there a PR associated with the branch you linked (FMayran:QwenVL-causal-fix)? If not, maybe we should start a separate PR to fix the vision implementation of these families of models in general and then come back to this PR to finish the model-specific parts.

now there is: #16745

[ngxson]

• Its 2D positional embeddings are dynamically adapted via bicubic interpolation within the Glm4vMoeVisionEmbeddings module to handle varied image resolutions

• It also applies its own rotary position embeddings within the self-attention blocks (via apply_rotary_pos_emb_vision)

This is essentially the same thing that LFM2 use, you can copy most of the code from this model (already supported by mtmd)

The key difference is in the projection stage, GLM4V uses:

• one conv2d to merge output tokens (2x2 square kernel), see: https://github.com/huggingface/transformers/blob/8012f80f722044fd0dda45b4034f89fffc2ff344/src/transformers/models/glm4v/modeling_glm4v.py#L731
• then finally project to text embeddings using a simple FFN, see: https://github.com/huggingface/transformers/blob/8012f80f722044fd0dda45b4034f89fffc2ff344/src/transformers/models/glm4v/modeling_glm4v.py#L116

[ngxson]

normally, text model of "vision" model is just a normal text model, so you probably don't need to add a new arch for it (no need to change anything in main llama.cpp code). only thing need to change is mtmd

[ddh0]

Yeah, I am still not 100% sure if we need separate architectures for these vision models or not. The paper mentions:

To further enhance spatial awareness on the language side, we extend RoPE to 3D-RoPE in the LLM.

I think what they're referring to as "3D-RoPE in the LLM" is actually M-RoPE, which glm4 and glm4_moe do not use.

Maybe M-RoPE could be conditionally incorporated into the existing llm_build_glm4 and llm_build_glm4_moe graph, but I thought it would be cleaner for the implementation of the vision models to be separate. I also did it this way following the pattern of Qwen3 / Qwen3VL being separate, as I think GLM is not too dissimilar from those.

[fuutott]

https://huggingface.co/zai-org/GLM-4.6V

[eitelnick]

Can you also add to Ollama cloud with vision / multimodel support?

[eelbaz]

Almost there... In progress for community review..

[eelbaz]

GLM Support has landed and open for review: #17967 - Enjoy!

[ngxson]

@ddh0 are you still actively working on this PR? I'll have a look in upcoming days

[ngxson]

Nevemind, this PR doesn't have the tensor_mapping for mmproj that I need, so probably better for me to start from zero

[ddh0]

@ddh0 are you still actively working on this PR? I'll have a look in upcoming days

No, I sort of got stuck and wasn't sure how to proceed, and I also got a job so I have less free time now. As I'm sure you know there have been some MRoPE fixes/additions since I first started this PR so there is probably something I'm missing.

Nevemind, this PR doesn't have the tensor_mapping for mmproj that I need, so probably better for me to start from zero

Sure, I would appreciate it if you took over, you know what you're doing more than I do.

[eelbaz]

@ddh0 — Thanks for the great starting point, your work was super helpful! I just tried to contribute a working path to unblock the community while official maintainer approved support for glm4.6v is in consideration/progress. Please feel free to use in any manner #17998 (the take-or-leave code works), if helpful. Apologies for the noise/thrash!