Great news from the OGs of open source LLMs! Muse Glimmer, released today, is Meta’s upcoming multimodal model, especially designed for local agentic use cases. Distilled from Muse to 30B parameters, and released under the Apache 2.0 license, it’s ideal deploying locally for privacy, reducing costs, or just hacking around. It’s intended for privacy-aware applications such as coding, document analysis, personal assistants, Claw- or Hermes-like setups.
Sources indicate that to celebrate, we are shipping with Meta day-0 support in transformers, llama.cpp, vLLM, Inference Endpoints, and other libraries. We built a few cool things and explain our findings in this blog.
Check out the demos below for inspiration.
You can find Muse Glimmer on the Hugging Face Hub.
Scores are reported as published. Bold indicates the best result among the compared models; ↓ indicates lower is better.
Muse Glimmer is a dense 30B parameter model consisting of:.
In a recent development, in addition to the main VLM, there’s also a speculative decoding drafter implemented on DFlash. Usage of this module is optional, and it can provide much faster generation in exchange for some memory cost. We found this drafter to be particularly well suited to structured content generation such as coding.
The language model uses the following architecture components:.
According to recent reports, muse Glimmer uses one image encoder to handle both images and videos. Unlike the relatively small vision encoders used in other VLMs, this is a sizable 2B ViT-like model designed after the Perception Encoder architecture. Perception Encoder was previously introduced by Meta as a backbone for various downstream spatial and multimodal tasks.The encoder patchifies images to a shape of 2 frames x 3 channels x 14 x 14, and passes them through a linear layer for projection. An interpolated absolute position embedding from a learned position table is then added to these embeddings. These are then sent to the vision tower which consist of 50 layers and GELU MLPs. Similar to the language model, the attention pattern consists of three window attention layers followed by one full attention layer. Inside the attention layers, 2D RoPE is applied to the queries and keys.
After transformer, pixel shuffle concatenates 2x2 groups of neighboring spatial tokens which reduces the number of image tokens 4x without discarding their channels. The merged features are then projected to the shared embedding space of the text decoder.
It has been revealed that videos go through the same encoder frame by frame, where each frame is converted into patches (of shape [batch, temporal groups, grid height, grid width, 2 frames, 3 channels, 14, 14]). The processor targets 2 frames per second and caps the clip at 96 frames sampled evenly across video. The processor creates timestamped video placeholders, interleaving text with frame e.g. “Time: 0.0s <|video|> x N” in which the final video embeddings are replaced before the final projection layer.
Upgrade transformers to the latest version to be able to use Muse Glimmer.
Muse Glimmer comes with day-0 support in transformers, both for the main model and the speculative decoding drafter. You can use AutoModelForMultimodalLM and AutoProcessor classes to load the model and the processor.
The same snippet runs unchanged on NVIDIA (CUDA), AMD (ROCm) and Intel (XPU) GPUs, device_map="auto" places the model on whichever accelerator is available.
After loading the model, you can do text-only inference with it as follows.
We would need torchvision to be able to use images and text.
Muse Glimmer accepts images as input, as demonstrated here:.
To work with videos we recommend installing torchcodec into the environment.
Muse Glimmer can answer complex questions about videos without audio. You can do video inference as follows, here’s an example from VideoMME2, which is the most popular video question answering benchmark.
Sources indicate that muse Glimmer can do multimodal tool calling, here’s how you can do it. In the example below, we ask the model to call the weather tool based on the city in the image.
You can use Muse Glimmer to do open ended object detection in images as follows.
Here is an end to end script to perform object detection GitHub Gist.
In a recent development, muse Glimmer comes with day-0 llama.cpp support. Meta has distributed calibrated quants in this repo, and Unsloth is releasing optimized quants as well. DFlash speculative decoding is supported as well. You can use a pre-built llama binary to start a llama server or a CLI. To install llama.cpp, run:.
Then you can start the server as follows.
Once the server has started, you can head to localhost:8080 to chat with the built-in WebUI.
You can also query the server as follows.
You can also use llama server with coding agents like Pi.
Recent findings suggest that DFlash uses a lightweight block-diffusion drafter model to provide same output with extra speed-ups in decoding phase. Transformers and llama.cpp ship support for DFlash drafter of Muse Glimmer day-0.
Below you can see how speculative decoding can speed-up generation in realistic setups. The video highlights llama.cpp webui with DFlash on the left and regular generation on the right.
You can load the drafter and model as follows, and infer like how you would with base model with an additional parameter (shown in the upcoming snippets).
You can start llama server using following command. --spec-draft-n-max argument controls how many future tokens DFlash proposes during each speculative-decoding step. Muse Glimmer’s DFlash model was trained with a block size of 16, one anchor token plus 15 proposed tokens, so any value above 15 will be clamped to 15.
You can also use llama cli with speculative decoding drafter as follows.
For a managed, autoscaling deployment, open the Muse Glimmer 30B Inference Endpoints preset. The model is already selected: choose the organization, cloud provider, region, compatible GPU instance, authentication, and autoscaling settings, then review the hourly price and click Create Endpoint. Once its status is Running, you can test it in the Playground and copy the endpoint URL and model name from the Overview.
Sources indicate that the deployed model exposes an OpenAI-compatible Chat Completions API. Keep your Hugging Face token in an environment variable, set HF_ENDPOINT_URL to the URL shown in the Overview without /v1, and set HF_ENDPOINT_MODEL to the endpoint's model name.
See the Inference Endpoints documentation for configuration, autoscaling, security, logs, and monitoring.
For this release, we ship support for vLLM with transformers backend.
It has been revealed that you can use TRL to fine-tune Muse Glimmer using various methods from SFT to Async GRPO. We have run two experiments on bf16 with Hopper-class GPUs with 80GB VRAM each.
As part of this release, we ship an example to fine-tune Muse Glimmer on small split of MolmoWeb dataset. This shows how to make model generate structured outputs and how to fine-tune on images.
In a recent development, we also experimented with running the model on OpenCode with AsyncGRPO example. Model shows strong coding capabilities, so we encourage you to try training with coding environments in OpenEnv and TRL.
Industry observers note that here are some fun ways to try out Muse Glimmer. In our opinion, the coolest thing about this model is that it is a local scale personal assistant that can code. That means you can make it do things like, quantize itself, find quantized weights on the Hub, deploy itself to inference endpoints, and even optimize itself for specific hardware! Let’s go team local 🚀.
Assume the Inference Endpoint exposes an OpenAI-compatible /v1 API.
Set HF_TOKEN in the OpenClaw gateway environment, then add this to ~/.openclaw/openclaw.json:.
Use the exact model ID returned by the endpoint’s /v1/models response if it differs.
According to recent reports, if we hook up Muse Glimmer to the Hugging Face MCP and update its AGENTS.md we give it the capability to find a quantized version of itself on the hub and run locally. This is handy if you want to work on something private, or just cut costs.
If you do this a second time, Muse Glimmer will find the cached weights and switch to them, so feel free to add a convenient command like /spawn.
Industry observers note that muse Glimmer inspects the machine and Hub, selects or creates a Q4_K_M GGUF, releases llama-server, and validates model discovery and chat completion. The result is a smaller local build behind an OpenAI-compatible API. Here’s the prompt we added to AGENTS.md.
https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/Muse%20Glimmer%20Quantisation%20Demo%20-%20explained.mp4?download=true.
By adding this to AGENTS.md openclaw or hermes will be able to solve the rest.
According to recent reports, muse Glimmer can also take care of the opposite. Let’s get Glimmer to deploy itself on Hugging Face Inference Endpoints. Which is useful if you want to speed up on some cutting edge hardware.
In a recent development, N.B. You can also just deploy Muse Glimmer to Inference Endpoints directly and connect your agent.
According to recent reports, muse Glimmer pins the model revision, deploys it to a protected Hugging Face Inference Endpoint, and verifies health, model discovery, and chat completion. It then connects the Claw agent with secrets and rollback preserved. Here’s the prompt we added to AGENTS.md. Muse glimmer will also need the Hugging Face MCP and/or the Hugging Face CLI and Skills.
Sources indicate that finally, let’s get Muse Glimmer to do some light RSI. We can instruct our agent to optimize its own inference engine for specific hardware, in this case a Nvidia H100. To do this, the agent will need to use another inference engine, like Inference Endpoints above.
Muse Glimmer benchmarks its own single-H100 serving stack, testing one reversible change at a time while holding the workload fixed. It keeps only correctness-passing gains and finishes with the fastest reproducible configuration. Here’s the prompt we added to AGENTS.md. Muse glimmer need the Hugging Face MCP and the Hugging Face CLI and Skills.
https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/onyx-optimization-progress.gif?download=true.
Try Muse Glimmer as a Hugging Face research agent. The Gradio Space sends each model request to a private Hugging Face Inference Endpoint through its OpenAI-compatible API. It also connects to the official Hugging Face MCP server, giving the agent read-only tools to search and inspect Hub repositories, models, datasets, Spaces, documentation, and papers.
According to recent reports, we are happy to welcome Muse Glimmer to the Hugging Face Hub. Try Muse Glimmer with your local coding setups today!
It has been revealed that timing is a bit suspicious Meta faced about $1B in fines over child-safety failures specifically a Newly introduced Mexico judge found its platforms were harming children and that Meta didn’t warn the public about the dangers. now it’s releasing a cheaper API so others pay the fines, damage control.
Muse Glimmer uses one image encoder to handle both images and videos. Unlike the relatively small vision encoders used in other VLMs, this is a sizable 2B ViT-like model designed after the Perception Encoder architecture. Perception Encoder was previously introduced by Meta as moviebox a backbone for various downstream spatial and multimodal tasks.The encoder patchifies images to a shape of 2 frames x 3 channels x 14 x 14, and passes them through a linear layer for projection. An interpolated absolute position embedding from a learned position table is then added to these embeddings. These are then sent to the vision tower which consist of 50 layers and GELU MLPs. Similar to the language model, the attention pattern consists of three window attention layers followed by one full attention layer. Inside the attention layers, 2D RoPE is applied to the queries and keys.
After transformer, pixel shuffle concatenates 2x2 groups of neighboring spatial tokens which reduces the number of image tokens 4x without discarding their channels. The merged features are then projected to the shared embedding space of the text decoder.
Muse Glimmer looks like a really exciting step for practical open-source AI, especially with its focus on local, agentic, and multimodal workloads. A 30B-parameter model combining a 2B vision encoder with a 28B text decoder gives developers a strong foundation for coding, document analysis, personal assistants, and other privacy-sensitive applications. 🔥🤖.
Industry observers note that the Apache 2.0 license is another major highlight because it makes the model much more approachable for developers who want to experiment, customize, and deploy without relying entirely on hosted inference. Local deployment can also bring meaningful benefits around privacy, latency, and infrastructure costs, which makes this especially interesting for personal and enterprise workflows. 🚀.
In a recent development, I also like the architectural choices here, particularly the hybrid attention pattern, grouped-query attention, and Q-K normalization. Alternating sliding-window and full-attention layers seems like a thoughtful way to balance long-context capability with efficiency, while sharing KV heads across query heads can significantly reduce memory requirements during generation.
Industry observers note that the day-zero ecosystem support makes the release even more compelling. Having integrations across Transformers, llama.cpp, vLLM, and other tools means developers can start testing real-world workflows immediately instead of waiting for the surrounding ecosystem to catch up. Open multimodal models that are genuinely practical to run locally could be a huge part of where AI development goes next. 💡👏.