Stay up to date with AI tools, model releases, and developer workflows that matter.
Weekly. Free. One click to leave.
It has been revealed that 7 Day Free Trial. Cancel Anytime.
Running an AI agent on your own GPU sounds simple: point it at Ollama and stop paying per token. I set up OpenClaw, an open-source AI assistant, with a 9B model in Ollama on one RTX 3090 and connected it to Discord. Then I read the Ollama log for every request.
Recent findings suggest that the agent worked. The numbers were not what I expected, though. A one-line question turned into a prompt of more than 18,000 tokens, the first reply took over a minute, and the small model sometimes guessed instead of using its tools.
This article covers what I measured and what each number means if you initiative to run an agent on local hardware.
It has been revealed that openClaw ran in Docker and reached Ollama on the host at http://host.docker.internal:11434. Discord was the chat interface, and OpenClaw's web panel indicated the sessions and context use.
It has been revealed that this article is about the results, not the installation. The full step-by-step setup is in a separate guide to running OpenClaw in Docker with Ollama and Discord. That guide covers the Compose file, the Discord bot and how to keep the ports off your network.
According to recent reports, all timings below come from the Ollama log on the test machine. Ollama loaded the model with a 32,768-token context.
My first message in Discord was short: "Which model are you running on, and what is 17 * 23?" The bot answered correctly: it ran on ollama/qwen3.5:9b, and 17 × 23 is 391.
The Ollama log highlighted what that question cost. The model processed 18,415 prompt tokens and generated 33 tokens.
Industry observers note that the extra tokens come from the agent itself. With every request, OpenClaw sends:.
On the RTX 3090, reading that prompt took 10 seconds at 1,838 tokens per second. A slower GPU, or a CPU, needs much longer to read the same prompt.
Prompt tokens per model call, from the Ollama log. Only the first message had to process the whole prompt.
OpenClaw counts tokens with its own estimate, which was higher than the Ollama count. The gateway log estimated the prompt at 24,608 tokens, against a prompt budget of 24,576 tokens for this model. By OpenClaw's own count, the base prompt alone used up the budget before any conversation history.
The web panel showed a similar figure: 68 to 69 percent of the context window was in use for the Discord sessions after only a few messages.
The first message arrived while the model was not loaded in GPU memory. That reply took 1 min 10 s:.
The next requests were much faster:.
Reply times on the test machine. Loading the model took most of the first reply.
In a recent development, the speed-up comes from prompt caching. The system prompt, tool definitions and workspace files at the start of the prompt stay the same between requests. Ollama kept that part cached and processed only the new tokens. The model wrote at 91 to 96 tokens per second in every test.
Recent findings suggest that ollama's documentation says models stay in memory for 5 minutes by default before they are unloaded. After that, the next message pays the load time again. You can change this with the OLLAMA_KEEP_ALIVE environment variable on the Ollama server, or with the keep_alive parameter per request. A negative value, such as -1, keeps the model loaded. I did not change this setting in the test.
If you run Ollama in Docker, pass the variable with -e OLLAMA_KEEP_ALIVE=-1 or in the environment: section of your Compose file.
Two things used the GPU besides my chat messages.
A heartbeat. OpenClaw's web panel listed one scheduled automation, heartbeat-main, which runs every 30 minutes. The agent wakes on that schedule, so the model gets called even when nobody chats. With Ollama's 5-minute default, the model is unloaded between heartbeats unless someone chatted recently, so a heartbeat most likely starts with a model load. I did not measure this.
In a recent development, an embedding model. During the first message, OpenClaw also called Ollama's /api/embed endpoint twice to index its memory. It used qwen3-embedding:4b, which was already installed on the Ollama server. I did not set OpenClaw to use that model; it picked it on its own.
The two models took this much GPU memory:.
That is 11.4 GB for the models alone, almost half of the RTX 3090's 24 GB.
The rest of the stack was light:.
What this means for you: proposal GPU memory for the chat model, the embedding model and the context, not the chat model alone. If the GPU also runs other models, the heartbeat can compete with them.
The first two tests in Discord: a question, then a file written with a tool. The user name is blurred.
The mistakes highlighted up when the model had to check something:.
According to recent reports, after the restart, the bot claimed it had lost the earlier session. When asked to use its read tool, it read the file correctly.
Industry observers note that the fix for the first problem was simple. When I wrote "use your read tool to open homelab-test.md", the model opened the file and listed all three points correctly. Naming the tool made it check instead of guess.
The model listed four files and missed the test file and a folder, because it answered from its prompt.
Based on this test, a local 9B model on one consumer GPU works for:.
If you need better answers, OpenClaw also works with hosted models, but then your prompts leave your machine.
In this test, qwen3.5:9b used 6.3 GB and the qwen3-embedding:4b memory model used 5.1 GB. Together the two models needed 11.4 GB.
The model had to load into GPU memory first, which took 59 seconds of the 70-second reply. Ollama keeps a model loaded for 5 minutes by default, and the OLLAMA_KEEP_ALIVE setting changes that.
Sources indicate that every request includes OpenClaw's system prompt, its tool definitions and its workspace files. A one-line question became an 18,415-token prompt in this test.
It has been revealed that yes. A heartbeat automation runs every 30 minutes and calls the model even when nobody sends a message.
Industry observers note that it handled questions, arithmetic and file creation correctly in this test. It sometimes answered from its prompt instead of using its tools. When it wrongly said its memory was lost, asking it to use its read tool made it check the file and answer correctly.
7 Day Free Trial. Cancel Anytime.
