In a recent development, TL;DR: The transformers vLLM backend is now as fast (or faster) than custom vLLM implementations for many LLM architectures. Model authors can automatically leverage their transformers implementations to get ultra fast vLLM inference, for free.
Recent findings suggest that the transformers library has become the reference modeling library for Machine Learning. It supports 450+ architectures through consistent APIs, and is designed with the main goal that model implementations are self contained and easy to understand. Going through transformers code makes it easy for contributors to learn how an architecture works, and then port it to other frameworks such as vLLM, SGLang, MLX, llama.cpp, and many others.
Sources indicate that we have fully embraced this role in the ecosystem and are investing a lot of effort to make it easier. A big step in this direction was the integration last year of transformers as a modeling backend in vLLM. This has been allowing model authors to run transformers models (LLMs and VLMs alike) inside vLLM, without having to port anything. Transformers provides the modeling code, and vLLM provides extremely optimized inference techniques such as continuous batching and custom attention kernels.
This integration gets better now π!
We put the transformers modeling backend for vLLM head to head with vLLM's hand written native implementations across three very different Qwen3 models:.
Industry observers note that running any* Hugging Face model through the transformers modeling backend is a single flag β --model-impl transformers. It composes with the usual parallelism options, so nothing about your serving setup changes:.
In a recent development, *Models that use linear attention are not currently supported, but they will be soon! Custom models where the code lives in a Hub repo are unlikely to work as they will not have been written compliantly.
Each model is compared under three conditions that are identical in every way except the code path:.
The full, reproducible runner is available as a gist: benchmark.sh.
The transformers modeling backend for vLLM used to focus on attention as the bottleneck for inference. By plugging vLLMβs attention implementation at runtime, we could make a transformers model run efficiently inside the vLLM engine. But there are many dimensions to deployments that only a custom port can target to extract maximum inference performance. Parallelization across GPUs, compilation, fused kernels, and many more, all contribute to leveraging your hardware to achieve ultra-fast inference.
When model authors wanted the absolute best performance, they were still writing custom vLLM implementations.
The latest iteration of the transformers modeling backend for vLLM dynamically applies inference specific layer fusions at runtime to match the speed of custom code implementations, for compatible architectures.
In a recent development, the transformers modeling backend for vLLM now uses torch.fx to perform static analysis on the modelβs graph. This process searches for known patterns that can be optimised. After any patterns have been identified, it uses ast (abstract syntax tree) to manipulate the source code and rewrite some of the operations in place.
As shown above, this results in native vLLM inference speed for compatible models, without having to write a single line of code to optimize the model for inference.
We are in the process of writing a detailed blog post to dive deep inside these optimized inference methods and explain in detail how we manipulate the model to adapt to them.
One provenance implication here is that the model definition and the executed inference graph are no longer the same artifact boundary.
If the runtime rewrites model code and replaces modules with fused kernels, deployment evidence probably needs to preserve both the source model identity and the transformation/runtime path.