TL;DR: This work highlights how a lightweight vision–language model can acquire GUI-grounded skills and evolve into an agentic GUI coder. We release all training recipes, data-processing tools, resulting model, demo and datasets to enable full reproducibility and foster further research 🫡. Find the collection here.
Graphical User Interface (GUI) automation is one of the most challenging frontiers in computer vision. Developing models that see and interact with user interfaces enables AI agents to navigate mobile, desktop, and web platforms. This will reshape the future of digital interaction.
In a recent development, in this blog post, we present a comprehensive approach to training vision-language models for GUI automation through a multi-phase training strategy. We demonstrate how to transform a model with zero grounding capabilities into an agentic coder capable of understanding and interacting with graphical interfaces.
Rather than aiming for a SOTA model, our goal is to demonstrate the entire process, from data processing to model training, and, in doing so, show how to unlock GUI-grounding capabilities in VLMs.
GUI capabilities combine understanding of the interface and precise element localization. These abilities enable the model to translate high-level tasks into low-level GUI actions such as clicking, typing, ….
Our approach leverages SmolVLM2-2.2B-Instruct as the baseline model, a small powerful vision-language model that initially has no grounding capabilities for GUI tasks. This makes it an ideal candidate to demonstrate the effectiveness of our training methodology. Through our two-phase training process, we first instill grounding capabilities in the model, then enhance it with agentic reasoning abilities using Supervised Fine-Tuning (SFT).
We evaluate our approach on an established perception benchmark: ScreenSpot-v2, which tests the model’s ability to understand and locate elements within screenshots. Our process is inspired by the AGUVIS paper, and we leverage their carefully curated datasets to build upon their foundational work.
Evolution of ScreenSpot-v2 performance during the training phase of the base model SmolVLM2-2.2B-Instruct.
Sources indicate that this section explains how we convert heterogeneous GUI actions format from multiple datasets into a single unified format. By standardizing function names, signatures, and parameters, we create consistent, high-quality data that forms the foundation for effective model training.
Recent findings suggest that one of the primary challenges when working with multiple GUI automation datasets is the lack of standardization in action representations. Different datasets use varying function signatures, parameter naming conventions, and action taxonomies, making it difficult to train a unified model across diverse data sources.
We took the open-source datasets (xlangai/aguvis-stage1, xlangai/aguvis-stage2), originally used by AGUVIS, and implemented a comprehensive data transformation pipeline to create a unified action space. Our approach involved:.
Here are real examples from our action conversion system (preprocessing/action_conversion.py) showing how we transform heterogeneous action representations into our unified format (grounding coordinates normalized to [0,1]):.
Before (Original Action Dataset Formats):.
After (Unified Action Dataset Formats):.
This unification process was essential for creating coherent training data that allows the model to learn consistent action patterns across diverse GUI environments.
To maximize flexibility for different use cases, we developed the Action Space Converter (utils/action_space_converter.py), a tool that allows users to easily adapt from an action space to their own custom action vocabularies and naming conventions.
You can use this tool to transform one action signature (function names, parameter names, and parameter value changes, ...) into another:.
The Action Space Converter provides:.
This tool enables researchers and practitioners to:.
The Action Space Converter is particularly valuable for preparing datasets for training, as it ensures consistent action vocabularies across different deployment environments while maintaining compatibility with existing automation frameworks.
Through this pipeline, we transform the open-source datasets xlangai/aguvis-stage1, xlangai/aguvis-stage2 into our unified action space (see here). The output of this process is released as two latest fully formatted datasets: smolagents/aguvis-stage-1 and smolagents/aguvis-stage-2.
Sources indicate that phase 1 leverages the smolagents/aguvis-stage-1 dataset, which introduces GUI grounding by pairing low-level instructions with diverse executable actions (expressed in code form). For example, a user/assistant turn in smolagents/aguvis-stage-1 follows the structure:.
Recent findings suggest that each sample links a screenshot with multi-turn user/assistant interactions, enabling the model to learn fine-grained action grounding across dialogue turns. During fine-tuning, the data collator masks everything except the assistant’s answers when computing the loss.
Before proceeding with full-scale Phase 1 training, we conducted comprehensive ablation studies to determine optimal training configurations.
We experimented with different image sizes and coordinate representation systems to identify the optimal configuration for SmolVLM2:.
Some SOTA GUI VLMs (e.g., Qwen-VL) appear also to use a different normalized range (0–1000), which was not tested in this experiment.
Table 1: Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct (400k samples, aguvis-stage-1). Higher is better.
As demonstrated in our benchmark results, SmolVLM2-2.2B-Instruct base initially achieved 0% performance on perception benchmarks like ScreenSpot-v2. This complete lack of grounding capability provided us with a clean slate to evaluate the effectiveness of our training methodology.
From our experiments, we determined that:.
Using the optimal configuration (1152px resolution with normalized coordinates), we trained for 2 epochs on the smolagents/aguvis-stage-1 dataset. The results were remarkable, +41% improvement over baseline on ScreenSpot-v2.
This dramatic improvement demonstrates that our Phase 1 training successfully instilled fundamental grounding capabilities in the model, enabling it to understand and locate visual elements within screenshots.
Table 2: Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct (2 epochs, aguvis-stage-1).
Industry observers note that whereas Phase 1 provided grounding capabilities, Phase 2 targets agentic reasoning, the ability to deliberate and proposal before acting. This stage transforms the model from a reactive system identifying GUI elements into a proactive agent capable of executing complex, multi-step interactions.
Phase 2 uses the smolagents/aguvis-stage-2 dataset, which introduces agentic scenarios:.
Explicit reasoning about upcoming actions.
Context consistency across multiple interaction steps.
High-level instructions require multi-step, low-level actions.
For example, the smolagents/aguvis-stage-2 chat message is like this:.
In a recent development, each sample links a screenshot with a system/user/assistant turn. During fine-tuning, the data collator masks everything except the assistant’s answers when computing the loss.
Recent findings suggest that starting from the Phase 1 checkpoint (1152 px resolution, normalized coordinates), we fine-tuned the model for two epochs on smolagents/aguvis-stage-2. The accuracy on ScreenSpot-v2 increased from 41% to 61%, indicating that explicit reasoning improves GUI grounding performance.
Table 2: Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct after Phase 1 finetuning (2 epochs, aguvis-stage-1).
All training code, data processing pipelines, datasets and model are open-source!
Industry observers note that our experiments demonstrate that high-quality, reasoning-oriented data can substantially improve GUI grounding, even for small VLMs, using only supervised fine-tuning (SFT). Beyond raw performance gains, these results show that the GUI grounding capabilities are largely determined by the quality of the data. Carefully curated datasets teach models the structure and semantics of user interfaces, providing the grounding needed for accurate action prediction.
To support the development of GUI agents, we’re open-sourcing everything: our complete pipeline, datasets, and trained model. You can reproduce our results, experiment with different models and architectures, or adapt our approach to upcoming domains. The future of agentic AI depends on researchers like you pushing these boundaries further!
While SFT excels at supervised tasks, emerging methods such as Reinforcement Learning (RL) or Direct Preference Optimization (DPO) help develop stronger reasoning capabilities and enable real-time adaptation. These advances point toward a upcoming generation of GUI agents that learn and improve through interaction rather than relying solely on static datasets.
Let’s build the future of GUI agents together 🤗.
how do we use it for mobile devices offline ?
Recent findings suggest that FYI, we just released a model based on the little one SmolVLM2-500M, pushing ScreenspotV2 to 85.8 ! https://huggingface.co/vocaela/Vocaela-500M.
can this be converted into GGUF or any other format to run on mobile devices ?
Recent findings suggest that haven't converted yet. It hasn't changed arch or config of the base model (HuggingFaceTB/SmolVLM2-500M-Video-Instruct). I converted SmolVLM2-500M-Video-Instruct to gguf before using the tool provided in llamacpp codebase. So I suppose it can be converted smoothly -- but haven't done it this time.
ok, i have found something, VLM are like taking 1 min to decode the tokens and Imagesso i guess it is a terrible idea (for now) : ).
I am fine tuning smolvlm-256m for the gui/mobile operation based on datasets mentioned in the articlehttps://huggingface.co/datasets/smolagents/aguvis-stage-1 & https://huggingface.co/datasets/smolagents/aguvis-stage-2but below code function from https://github.com/huggingface/smol2operator/blob/main/recipe.ipynbis causing the training to go for more than 10 days , is there any way to optimize the collate_fn function !!?I am planning to release model in opensource after trainingthanks !
def create_collate_fn(processor, max_length: int): """Optimized collate function for VLM training that masks system prompt tokens.""".
can it be running on iOS device?
I created a model series, maybe close to, although not really tested on phones but on very old laptops. The latest one is Vocaela-2-500M-1024R2, it has GGUF provided as well. 3x faster than its previous version Vocaela-500M.
According to recent reports, there is an even smaller 256M version as well Vocaela-2-256M-512R2, speedwise much faster than the 500M ones (4-5x faster than Vocaela-2-500M-1024R2), however, quality not as good as them. Unfortunately, this one has no GGUF yet because LlamaCpp converted GGUF version's output is not consistent with the HF version. I am trying to figure out what happened there.