llama.cpp server now ships with router mode, which lets you dynamically load, unload, and switch between multiple models without restarting.

Reminder: llama.cpp server is a lightweight, OpenAI-compatible HTTP server for running LLMs locally.

Sources indicate that this feature was a popular request to bring Ollama-style model management to llama.cpp. It uses a multi-process architecture where each model runs in its own process, so if one model crashes, others remain unaffected.

Start the server in router mode by not specifying a model:.

In a recent development, this auto-discovers models from your llama.cpp cache (LLAMA_CACHE or ~/.cache/llama.cpp). If you've previously downloaded models via llama-server -hf user/model, they'll be available automatically.

You can also point to a local directory of GGUF files:.

It has been revealed that on the first request, the server automatically loads the model into memory (loading time depends on model size). Subsequent requests to the same model are instant since it's already loaded.

Returns all discovered models with their status (loaded, loading, or unloaded).

All model instances inherit settings from the router:.

All loaded models will use 8192 context and full GPU offload. You can also define per-model settings using presets:.

The built-in web UI also supports model switching. Just select a model from the dropdown and it loads automatically.

We hope this feature makes it easier to A/B test different model versions, run multi-tenant deployments, or simply switch models during development without restarting the server.

According to recent reports, have questions or feedback? Drop a comment below or open an issue on GitHub.

Supported via presets.ini, where you can specify the mmproj (and other long and short arguments) per model.

Awesome fresh feature! Can model selection be done on something other than requested model name? Like maybe specify the ranking in presets.ini, and then the highest ranked model that can satisfy the request will be the default. So maybe one model is best for short context, another (or the same with other settings) for when the context gets too long, and another when image input is required.

This is good addition, Thank you.

what is the best way to get and the tokens in between? openAI library is removing them.. i want to run llama-server in console and talk to it using a python library that does not remove the thinking tokens.

i checked the llama-cpp-python but it does not have that.

llama-server by default in most implementation keeps the reasoning content in reasoning_content variable in response attribute. You can get it from there. Otherwise use reasoning-format flag and pass DeepSeek value to get pure tokens.

Now I can use llama.cpp all the time. A big thank you to the devs.

In a recent development, is there presently a way to have a "default" model if the request doesn't specify? Could be the currently loaded model or a specific model. (Just noticed one of my apps broke because it's used to llama-server not requiring a model name.).

[DEFAULT]port = 8080n-gpu-layers = -1device = 0flash-attn = onchat-template = jinjamodels-max = 4.

Does it unload the current model if VRAM is full, to allow swapping to a new model?

fun ideas , add personal avatar and p2p social network also emule p2p models storage.

Hey there! Just wanted to drop a quick note saying I'm really digging the newly introduced router mode in llama.cpp server. It's a game-changer for me, especially when I need to switch between different models. The auto-discovery of models and LRU eviction is pretty neat – no more manual updates or restarts needed. It's like having a dynamic model manager on-the-fly. And the request routing part? Brilliant! Makes my workflow with dmenu smoother. Check out the full experience and check out my dmenu launcher script on the project's GitHub: https://gitea.com/gnusupport/LLM-Helpers/src/branch/main/bin/rcd-llm-dmenu-launcher.sh.

Recent findings suggest that thanks for the update! does it now behave like ollama?

Thank you so much for this, it's great!

I want to specifically pin models to a specific GPU (I have multiple) is that possible?

→ Read original article