Feature Request: Mapping model name to LoRA config

[ngxson]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I came across this idea while working on #10994

The idea is that we can maintain a list of model name mapped to LoRA config, for example:

{
 "llama-base": [{"id": 0, "scale": 0.0},{"id": 1, "scale": 0.0}],
 "llama-story": [{"id": 0, "scale": 1.0},{"id": 1, "scale": 0.0}],
 "llama-abliteration": [{"id": 0, "scale": 0.0},{"id": 1, "scale": 1.0}],
 "llama-story-abliteration": [{"id": 0, "scale": 0.5},{"id": 1, "scale": 0.5}]
}

Then, user can switch the model by specifying model in the request, for example:

# first user:
{
 "model": "llama-story-abliteration",
 "messages": [
{"role": "user", "content": "Write a NSFW story"}
 ]
}

# second user:
{
 "model": "llama-base",
 "messages": [
{"role": "user", "content": "Is this NSFW?"}
 ]
}

Motivation

N/A

Possible Implementation

No response

[joeamroo]

I can work on this if possible.

[ngxson]

@joeamroo Can your firstly describe how you will implement it?

Also, I'm thinking about a more general case. Maybe users want not just mapping list of lora to model name, but also generation params to model name. For example:

{
 "llama-base":{"lora": [{"id": 0, "scale": 0.0},{"id": 1, "scale": 0.0}], "top_k": 1},
 "llama-story":{"lora": [{"id": 0, "scale": 1.0},{"id": 1, "scale": 0.0}], "temperature": 0.8},
 "llama-abliteration":{"lora": [{"id": 0, "scale": 0.0},{"id": 1, "scale": 1.0}]},
 "llama-story-abliteration":{"lora": [{"id": 0, "scale": 0.5},{"id": 1, "scale": 0.5}]}
}

So probably we should implement the more generic way.

[brokad]

This feature would be useful.

Briefly reading through the code, here is a thought:

I see there already is a /props endpoint with POST not doing anything. Assuming model ids are url-safe, how about we expand this to /props/model-alias? Not sure if this is what was planned for this endpoint - or if it was destined for something else.

We'd add a mapping of model-alias -> common_params to the server ctx. The POST handler can be implemented like the /lora-adapters endpoints.

When the user POSTs a new model-alias, we can also start from the ambient server_context.params_base so that global defaults are applied unless otherwise overridden explicitly by the user.

Finally, in handle_completions_impl, instead of using ctx_server.params_base for defaults, we'd pass the alias-specific common_params.

What do you think @ngxson?

[ngxson]

IMO using a POST endpoint to set this will make it less reproducible. Many users expect to use llama-server on-demand, so it will be annoying having to set /props each time they start the server.

A better UX/DX is to provide an argument, for example --alias-presets-file my_preset.json and the user can simply put the config into the file:

{
 "llama-base":{"lora": [{"id": 0, "scale": 0.0},{"id": 1, "scale": 0.0}], "top_k": 1},
 "llama-story":{"lora": [{"id": 0, "scale": 1.0},{"id": 1, "scale": 0.0}], "temperature": 0.8},
 "llama-abliteration":{"lora": [{"id": 0, "scale": 0.0},{"id": 1, "scale": 1.0}]},
 "llama-story-abliteration":{"lora": [{"id": 0, "scale": 0.5},{"id": 1, "scale": 0.5}]}
}

[prd-tuong-nguyen]

I think this feature is really helpful. The configuration to map the LoRA name to its settings could also be configured in the environment variable. I am looking forward to this feature being released.