[model] Add support for Plamo3#17304

mmnga · 2025-11-16T16:55:19Z

This PR adds support for the PLaMo-3 series (2B, 8B, 31B base models):

PLaMo-3 uses a hybrid architecture with Sliding Window Attention (SWA) and standard full attention layers, as well as a custom FFN layout. This PR wires those pieces into llama.cpp so that the official checkpoints can be converted to GGUF and run with the usual backends.

…amo-3

…lama.cpp into features/suppert-plamo-3

CISC · 2025-11-17T10:38:26Z

Any non-gated models available?

mmnga · 2025-11-17T10:57:39Z

There are no non-gated models available at the moment.

mmnga · 2025-11-17T13:34:22Z

Sorry, the checks failed, so I’m reverting it to draft for now.

CISC · 2025-11-17T13:37:49Z

Sorry, the checks failed, so I’m reverting it to draft for now.

The nvidia-vulkan-cm CI failures are unrelated if that's what you're referring to...

…lama.cpp into features/suppert-plamo-3

mmnga · 2025-11-18T15:41:44Z

I’ve reopened this PR. Thank you in advance.

mmnga · 2025-12-01T16:29:45Z

When you have time, I’d appreciate a quick look or any feedback on this PR.
Happy to update the code if needed. Thanks!

CISC

Sorry for the late review.

Make the changes, then rebase to resolve llama-arch.cpp changes.

CISC · 2025-12-17T19:38:58Z

convert_hf_to_gguf.py

+# PLaMo models use a custom tokenizer with a .jsonl file
+tokenizer_jsonl_path=self.dir_model/"tokenizer.jsonl"
+tokenizer_config_path=self.dir_model/"tokenizer_config.json"
+
+ifnottokenizer_jsonl_path.is_file():
+raiseFileNotFoundError(f"PLaMo tokenizer file not found: {tokenizer_jsonl_path}")
+
+# Load tokenizer config
+withopen(tokenizer_config_path, "r", encoding="utf-8") asf:
+tokenizer_config=json.load(f)
+
+# Load tokens from JSONL file (actually a list format)
+tokens= []
+scores= []
+toktypes= []
+
+withopen(tokenizer_jsonl_path, "r", encoding="utf-8") asf:
+forline_num, lineinenumerate(f):
+ifline.strip():
+token_data=json.loads(line)
+# Format: [token, score, type, ?, ?, ?, ?]
+token=token_data[0].encode("utf-8")
+score=float(token_data[1])
+token_type_str=token_data[2] iflen(token_data) >2else"NORMAL"
+
+tokens.append(token)
+scores.append(score)
+
+iftoken_type_str=="UNKNOWN":
+toktypes.append(gguf.TokenType.UNKNOWN)
+eliftoken_type_str=="CONTROL":
+toktypes.append(gguf.TokenType.CONTROL)
+eliftoken_type_str=="BYTE":
+toktypes.append(gguf.TokenType.BYTE)
+else:
+token_str=token_data[0]
+iftoken_str.startswith("<|plamo:") andtoken_str.endswith("|>"):
+toktypes.append(gguf.TokenType.CONTROL)
+else:
+toktypes.append(gguf.TokenType.NORMAL)
+
+vocab_size=self.hparams["vocab_size"]
+ifvocab_size>len(tokens):
+pad_count=vocab_size-len(tokens)
+logger.debug(f"Padding vocab with {pad_count} token(s) - [PAD1] through [PAD{pad_count}]")
+foriinrange(1, pad_count+1):
+tokens.append(bytes(f"[PAD{i}]", encoding="utf-8"))
+scores.append(-1000.0)
+toktypes.append(gguf.TokenType.UNUSED)
+
+self.gguf_writer.add_tokenizer_model("plamo2")
+self.gguf_writer.add_tokenizer_pre("default")
+self.gguf_writer.add_token_list(tokens)
+self.gguf_writer.add_token_scores(scores)
+self.gguf_writer.add_token_types(toktypes)
+
+if"bos_token"intokenizer_configandtokenizer_config["bos_token"] isnotNone:
+token_id=tokens.index(tokenizer_config["bos_token"].encode("utf-8"))
+self.gguf_writer.add_bos_token_id(token_id)
+if"eos_token"intokenizer_configandtokenizer_config["eos_token"] isnotNone:
+token_id=tokens.index(tokenizer_config["eos_token"].encode("utf-8"))
+self.gguf_writer.add_eos_token_id(token_id)
+if"pad_token"intokenizer_configandtokenizer_config["pad_token"] isnotNone:
+token_id=tokens.index(tokenizer_config["pad_token"].encode("utf-8"))
+self.gguf_writer.add_pad_token_id(token_id)
+if"sep_token"intokenizer_configandtokenizer_config["sep_token"] isnotNone:
+token_id=tokens.index(tokenizer_config["sep_token"].encode("utf-8"))
+self.gguf_writer.add_sep_token_id(token_id)
+if"unk_token"intokenizer_configandtokenizer_config["unk_token"] isnotNone:
+token_id=tokens.index(tokenizer_config["unk_token"].encode("utf-8"))
+self.gguf_writer.add_unk_token_id(token_id)
+
+# Add <|plamo:op|> as EOT to ensure appropriate end of generation
+self.gguf_writer.add_eot_token_id(4)
+
+self.gguf_writer.add_add_space_prefix(False)
+
+if"chat_template"intokenizer_configandtokenizer_config["chat_template"] isnotNone:
+self.gguf_writer.add_chat_template(tokenizer_config["chat_template"])


Suggested change
# PLaMo models use a custom tokenizer with a .jsonl file
tokenizer_jsonl_path=self.dir_model/"tokenizer.jsonl"
tokenizer_config_path=self.dir_model/"tokenizer_config.json"
ifnottokenizer_jsonl_path.is_file():
raiseFileNotFoundError(f"PLaMo tokenizer file not found: {tokenizer_jsonl_path}")
# Load tokenizer config
withopen(tokenizer_config_path, "r", encoding="utf-8") asf:
tokenizer_config=json.load(f)
# Load tokens from JSONL file (actually a list format)
tokens= []
scores= []
toktypes= []
withopen(tokenizer_jsonl_path, "r", encoding="utf-8") asf:
forline_num, lineinenumerate(f):
ifline.strip():
token_data=json.loads(line)
# Format: [token, score, type, ?, ?, ?, ?]
token=token_data[0].encode("utf-8")
score=float(token_data[1])
token_type_str=token_data[2] iflen(token_data) >2else"NORMAL"
tokens.append(token)
scores.append(score)
iftoken_type_str=="UNKNOWN":
toktypes.append(gguf.TokenType.UNKNOWN)
eliftoken_type_str=="CONTROL":
toktypes.append(gguf.TokenType.CONTROL)
eliftoken_type_str=="BYTE":
toktypes.append(gguf.TokenType.BYTE)
else:
token_str=token_data[0]
iftoken_str.startswith("<|plamo:") andtoken_str.endswith("|>"):
toktypes.append(gguf.TokenType.CONTROL)
else:
toktypes.append(gguf.TokenType.NORMAL)
vocab_size=self.hparams["vocab_size"]
ifvocab_size>len(tokens):
pad_count=vocab_size-len(tokens)
logger.debug(f"Padding vocab with {pad_count} token(s) - [PAD1] through [PAD{pad_count}]")
foriinrange(1, pad_count+1):
tokens.append(bytes(f"[PAD{i}]", encoding="utf-8"))
scores.append(-1000.0)
toktypes.append(gguf.TokenType.UNUSED)
self.gguf_writer.add_tokenizer_model("plamo2")
self.gguf_writer.add_tokenizer_pre("default")
self.gguf_writer.add_token_list(tokens)
self.gguf_writer.add_token_scores(scores)
self.gguf_writer.add_token_types(toktypes)
if"bos_token"intokenizer_configandtokenizer_config["bos_token"] isnotNone:
token_id=tokens.index(tokenizer_config["bos_token"].encode("utf-8"))
self.gguf_writer.add_bos_token_id(token_id)
if"eos_token"intokenizer_configandtokenizer_config["eos_token"] isnotNone:
token_id=tokens.index(tokenizer_config["eos_token"].encode("utf-8"))
self.gguf_writer.add_eos_token_id(token_id)
if"pad_token"intokenizer_configandtokenizer_config["pad_token"] isnotNone:
token_id=tokens.index(tokenizer_config["pad_token"].encode("utf-8"))
self.gguf_writer.add_pad_token_id(token_id)
if"sep_token"intokenizer_configandtokenizer_config["sep_token"] isnotNone:
token_id=tokens.index(tokenizer_config["sep_token"].encode("utf-8"))
self.gguf_writer.add_sep_token_id(token_id)
if"unk_token"intokenizer_configandtokenizer_config["unk_token"] isnotNone:
token_id=tokens.index(tokenizer_config["unk_token"].encode("utf-8"))
self.gguf_writer.add_unk_token_id(token_id)
# Add <|plamo:op|> as EOT to ensure appropriate end of generation
self.gguf_writer.add_eot_token_id(4)
self.gguf_writer.add_add_space_prefix(False)
if"chat_template"intokenizer_configandtokenizer_config["chat_template"] isnotNone:
self.gguf_writer.add_chat_template(tokenizer_config["chat_template"])
self._set_vocab_plamo()
tokenizer_config_path=self.dir_model/"tokenizer_config.json"
iftokenizer_config_path.is_file():
withopen(tokenizer_config_path, encoding="utf-8") asf:
tokenizer_config=json.load(f)
chat_template=tokenizer_config.get("chat_template")
chat_template_jinja=self.dir_model/"chat_template.jinja"
ifchat_template_jinja.is_file():
withopen(chat_template_jinja, encoding="utf-8") asf:
chat_template=f.read()
ifchat_template:
self.gguf_writer.add_chat_template(chat_template)
Move the rest of the code into TextModel._set_vocab_plamo and update Plamo2Model.set_vocab to just call `self._set_vocab_plamo().

CISC · 2025-12-17T19:40:06Z

convert_hf_to_gguf.py

+hparams=self.hparams
+block_count=hparams["num_hidden_layers"]
+
+self.gguf_writer.add_vocab_size(hparams["vocab_size"])
+self.gguf_writer.add_context_length(hparams["max_position_embeddings"])
+self.gguf_writer.add_embedding_length(hparams["hidden_size"])
+self.gguf_writer.add_feed_forward_length(hparams["intermediate_size"])
+self.gguf_writer.add_block_count(block_count)
+self.gguf_writer.add_head_count(hparams["num_attention_heads"])
+self.gguf_writer.add_head_count_kv(hparams["num_key_value_heads"])
+head_dim=hparams["head_dim"]
+self.gguf_writer.add_key_length(head_dim)
+self.gguf_writer.add_value_length(head_dim)
+self.gguf_writer.add_layer_norm_rms_eps(hparams["rms_norm_eps"])
+self.gguf_writer.add_rope_freq_base(hparams["rope_theta"])
+
+window_size=hparams.get("window_size") orhparams.get("sliding_window") or0
+self.gguf_writer.add_sliding_window(window_size)
+
+pattern=self._sliding_window_pattern(block_count)
+iflen(pattern) ==block_countandany(pattern):
+self.gguf_writer.add_sliding_window_pattern(pattern)
+
+self.gguf_writer.add_file_type(self.ftype)


Suggested change
hparams=self.hparams
block_count=hparams["num_hidden_layers"]
self.gguf_writer.add_vocab_size(hparams["vocab_size"])
self.gguf_writer.add_context_length(hparams["max_position_embeddings"])
self.gguf_writer.add_embedding_length(hparams["hidden_size"])
self.gguf_writer.add_feed_forward_length(hparams["intermediate_size"])
self.gguf_writer.add_block_count(block_count)
self.gguf_writer.add_head_count(hparams["num_attention_heads"])
self.gguf_writer.add_head_count_kv(hparams["num_key_value_heads"])
head_dim=hparams["head_dim"]
self.gguf_writer.add_key_length(head_dim)
self.gguf_writer.add_value_length(head_dim)
self.gguf_writer.add_layer_norm_rms_eps(hparams["rms_norm_eps"])
self.gguf_writer.add_rope_freq_base(hparams["rope_theta"])
window_size=hparams.get("window_size") orhparams.get("sliding_window") or0
self.gguf_writer.add_sliding_window(window_size)
pattern=self._sliding_window_pattern(block_count)
iflen(pattern) ==block_countandany(pattern):
self.gguf_writer.add_sliding_window_pattern(pattern)
self.gguf_writer.add_file_type(self.ftype)
super().set_gguf_parameters()
self.gguf_writer.add_vocab_size(self.hparams["vocab_size"])
if (sliding_window:=self.find_hparam(["window_size", "sliding_window"], optional=True)) isnotNone:
self.gguf_writer.add_sliding_window(sliding_window)
pattern=self._sliding_window_pattern(self.block_count)
iflen(pattern) ==self.block_countandany(pattern):
self.gguf_writer.add_sliding_window_pattern(pattern)

CISC · 2025-12-17T19:41:06Z

convert_hf_to_gguf.py

+results: list[tuple[str, Tensor]] = []
+
+if"gate_up_proj.weight"inname:
+name_up=name.replace("gate_up_proj.weight", "up_proj.weight")
+name_gate=name.replace("gate_up_proj.weight", "gate_proj.weight")
+gate_proj_weight, up_proj_weight=torch.chunk(data_torch, 2, dim=0)
+results.append((self.map_tensor_name(name_gate), gate_proj_weight))
+results.append((self.map_tensor_name(name_up), up_proj_weight))
+else:
+results.append((self.map_tensor_name(name), data_torch))
+
+returnresults


Suggested change
results: list[tuple[str, Tensor]] = []
if"gate_up_proj.weight"inname:
name_up=name.replace("gate_up_proj.weight", "up_proj.weight")
name_gate=name.replace("gate_up_proj.weight", "gate_proj.weight")
gate_proj_weight, up_proj_weight=torch.chunk(data_torch, 2, dim=0)
results.append((self.map_tensor_name(name_gate), gate_proj_weight))
results.append((self.map_tensor_name(name_up), up_proj_weight))
else:
results.append((self.map_tensor_name(name), data_torch))
returnresults
return [(self.map_tensor_name(name), data_torch)]

CISC · 2025-12-17T19:42:28Z

gguf-py/gguf/constants.py

+MODEL_TENSOR.ATTN_OUT,
+MODEL_TENSOR.ATTN_POST_NORM,
+MODEL_TENSOR.FFN_NORM,
+MODEL_TENSOR.FFN_GATE,


Suggested change
MODEL_TENSOR.FFN_GATE,

CISC · 2025-12-17T19:43:22Z

src/llama-arch.cpp

+{LLM_TENSOR_ATTN_POST_NORM, "blk.%d.post_attention_norm" },
+{LLM_TENSOR_FFN_NORM, "blk.%d.ffn_norm" },
+{LLM_TENSOR_FFN_POST_NORM, "blk.%d.post_ffw_norm" },
+{LLM_TENSOR_FFN_GATE, "blk.%d.ffn_gate" },


Suggested change
{LLM_TENSOR_FFN_GATE, "blk.%d.ffn_gate" },
Take care to update this accordingly after you rebase.

CISC · 2025-12-17T19:45:19Z

src/models/plamo3.cpp

+if (hparams.is_swa_any()){
+ inp_attn_iswa = build_attn_inp_kv_iswa();
+ } else{
+ inp_attn = build_attn_inp_kv();
+ }


Make build_plamo3 templated to handle this, like f.ex. smallthinker.

CISC · 2025-12-17T19:45:49Z

src/models/plamo3.cpp

+constint64_t head_dim_q = hparams.n_embd_head_k;
+constint64_t head_dim_v = hparams.n_embd_head_v;
+
+ ggml_tensor * inpL = build_inp_embd(model.tok_embd);


Suggested change
ggml_tensor * inpL = build_inp_embd(model.tok_embd);
ggml_tensor * cur;
ggml_tensor * inpL = build_inp_embd(model.tok_embd);

CISC · 2025-12-17T19:46:14Z

src/models/plamo3.cpp

+constfloat freq_scale_l = model.get_rope_freq_scale(cparams, il);
+ ggml_tensor * rope_factors = model.get_rope_factors(cparams, il);
+
+ ggml_tensor * cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);


Suggested change
ggml_tensor * cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);
cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);

CISC · 2025-12-17T19:46:42Z

src/models/plamo3.cpp

+ inpL = cur;
+ }
+
+ ggml_tensor * cur = inpL;


Suggested change
ggml_tensor * cur = inpL;
cur = inpL;

CISC · 2025-12-17T19:47:45Z

src/models/plamo3.cpp

+ ggml_tensor * ffn_up = build_lora_mm(model.layers[il].ffn_up, cur);
+cb(ffn_up, "ffn_up", il);
+
+ ggml_tensor * ffn_gate = build_lora_mm(model.layers[il].ffn_gate, cur);
+cb(ffn_gate, "ffn_gate", il);
+
+ ggml_tensor * ffn_act = ggml_swiglu_split(ctx0, ffn_gate, ffn_up);
+cb(ffn_act, "ffn_act", il);
+
+ cur = build_lora_mm(model.layers[il].ffn_down, ffn_act);
+cb(cur, "ffn_down", il);


Suggested change
ggml_tensor * ffn_up = build_lora_mm(model.layers[il].ffn_up, cur);
cb(ffn_up, "ffn_up", il);
ggml_tensor * ffn_gate = build_lora_mm(model.layers[il].ffn_gate, cur);
cb(ffn_gate, "ffn_gate", il);
ggml_tensor * ffn_act = ggml_swiglu_split(ctx0, ffn_gate, ffn_up);
cb(ffn_act, "ffn_act", il);
cur = build_lora_mm(model.layers[il].ffn_down, ffn_act);
cb(cur, "ffn_down", il);
cur = build_ffn(cur,
model.layers[il].ffn_up, NULL, NULL,
NULL, NULL, NULL,
model.layers[il].ffn_down, NULL, NULL,
NULL,
LLM_FFN_SWIGLU, LLM_FFN_SEQ, il);
cb(cur, "ffn_out", il);

CISC · 2025-12-17T21:59:07Z

src/llama-model.cpp

+ hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
+ ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, hparams.n_swa);
+if (hparams.n_swa == 0){
+ hparams.n_swa = 2048;
+ }
+ hparams.set_swa_pattern(8);


Suggested change
hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, hparams.n_swa);
if (hparams.n_swa == 0){
hparams.n_swa = 2048;
}
hparams.set_swa_pattern(8);
constbool found_swa = ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, hparams.n_swa, false);
if (found_swa && hparams.n_swa > 0){
hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
hparams.rope_freq_base_train_swa = 10000.0f;
hparams.rope_freq_scale_train_swa = 1.0f;
hparams.set_swa_pattern(8);
} else{
hparams.swa_type = LLAMA_SWA_TYPE_NONE;
}
Not sure where you were going with this, but since you seem to handle non-swa I'm guessing something like this?

mmngaysand others added 16 commits November 15, 2025 22:56

plamo3
f61ed44

fix plamo3
c3b6134

clean code
ce7a922

clean up the code
1ab3bba

fix diff
d9854cc

Merge branch 'ggml-org:master' into features/suppert-plamo-3
8dbbe79

clean up the code
967810d

clean up the code
74fa9d6

clean up the code
3391080

clean up the code
037d831

Merge branch 'ggml-org:master' into features/suppert-plamo-3
4d0be03

clean up the code
80c3418

clean up the code
9cecb26

Merge remote-tracking branch 'origin/master' into features/suppert-pl…
3873edb
…amo-3

Merge branch 'features/suppert-plamo-3' ofhttps://github.com/mmnga/l…
8b92852
…lama.cpp into features/suppert-plamo-3

clean up the code
0df5296

github-actionsbot added model Model specific python python script changes labels Nov 16, 2025

mmnga closed this Nov 16, 2025

add chat_template if exist
cdb1d2c

mmnga reopened this Nov 16, 2025

clean up the code
527c65a

mmnga marked this pull request as ready for review November 17, 2025 09:49

mmnga requested review from CISC and ggerganov as code owners November 17, 2025 09:49

mmnga marked this pull request as draft November 17, 2025 13:34

mmngaand others added 3 commits November 17, 2025 23:50

Merge branch 'ggml-org:master' into features/suppert-plamo-3
0f9d0a6

fix cpu-backend
5d52fe6

Merge branch 'features/suppert-plamo-3' ofhttps://github.com/mmnga/l…
dab7aaa
…lama.cpp into features/suppert-plamo-3

DajanaV mentioned this pull request Nov 17, 2025
UPSTREAM PR #17304: [model] Add support for Plamo3 auroralabs-loci/llama.cpp#237
Open

mmngaysand others added 2 commits November 18, 2025 16:53

chore: whitespace trim fix + typo fix
9bd33d0

Merge branch 'ggml-org:master' into features/suppert-plamo-3
67a6dda

mmnga marked this pull request as ready for review November 18, 2025 15:32

Merge branch 'ggml-org:master' into features/suppert-plamo-3
d965f02

Merge branch 'ggml-org:master' into features/suppert-plamo-3
c2aab05

CISC reviewed Dec 17, 2025
View reviewed changes

	ggml_tensor * inpL = build_inp_embd(model.tok_embd);
	ggml_tensor * cur;
	ggml_tensor * inpL = build_inp_embd(model.tok_embd);

	ggml_tensor * cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);
	cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);

[model] Add support for Plamo3#17304

Are you sure you want to change the base?

[model] Add support for Plamo3 #17304

Conversation

mmnga commented Nov 16, 2025

Uh oh!

CISC commented Nov 17, 2025

Uh oh!

mmnga commented Nov 17, 2025

Uh oh!

mmnga commented Nov 17, 2025

Uh oh!

CISC commented Nov 17, 2025

Uh oh!

mmnga commented Nov 18, 2025

Uh oh!

mmnga commented Dec 1, 2025

Uh oh!

CISC left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CISCDec 17, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CISCDec 17, 2025•
edited
Loading