Can a local open model run an agent? Tool calling with open-weight models

Quick answer

Yes, with caveats. An open model of around 30B running locally handles a few well-described tools and a short loop well: pick a tool, fill in the arguments, read the result, answer. Benchmarks show that single calls are no longer a problem even for 8B models. The problem is long, multi-turn tasks where the model plans the next steps itself: that is where smaller models' scores halve. So an agent on a local model needs argument validation, a step limit, a loop breaker and human approval before actions with consequences. That is how we build agents at pimento, on models from the Qwen, Gemma, GLM and Mistral families served on our own NVIDIA Blackwell GPUs.

What a model has to do in an agent loop

"Tool calling" sounds like one skill, but it is several:

1. Deciding whether to call a tool at all. A "good morning" should not trigger a customer database query.
2. Choosing the right tool out of several, or several dozen.
3. Filling in arguments according to the schema: types, required fields, date formats, identifiers.
4. Reading the result and deciding what's next: another tool, a question for the user, or an answer.
5. Stopping when the task is done or can't be done.

Points 1-3 are a single call. Points 4-5 are the loop, and errors compound there. A model that gets 90% of single steps right can fail more often than it succeeds on a task with ten dependent steps. We cover what an agent is and how it differs from a chatbot in Agentic AI explained.

What the benchmarks show

The most widely cited function-calling leaderboard is the Berkeley Function Calling Leaderboard (BFCL), now at V4. Besides simple calls (single, multiple, parallel), it measures queries from real users ("live"), multi-turn tasks, recognising when no tool fits, and since V4 also agentic scenarios: web search and memory management.

Selected rows from the leaderboard as of 12 April 2026:

Three things follow from this table.

Single calls are solved. The simple-calls column is flat: 85-90% for almost everyone, from 8B up to the largest closed models. If your agent is "understand the question, call one tool, write the answer", a small local model is enough.

Multi-turn is where models differ. On multi-turn tasks the spread is huge: from about 10% to about 68%. The best open model, GLM-4.6, matches the best closed one here, but it is a large MoE model. Models that fit easily on a single GPU score 30-48%.

The call mode matters. BFCL tests some models twice: through the native function-calling format (FC) and through instructions in the prompt. DeepSeek-V3.2-Exp scores 34.85% on simple calls in FC mode and 85.52% in prompt mode with thinking. Same model. The difference comes from the format and how it is parsed, which is to say from how the model is served.

One caveat: the leaderboard lags behind releases. The April 2026 version does not include, for example, Gemma 4 or newer Qwen generations, even though both now support tools natively. Treat BFCL as a map of model families, not a verdict on a specific version.

The other important benchmark is τ-bench (Sierra), which simulates an agent talking to a user in retail and airline settings, with real business rules. Its authors introduced the pass^k metric: the probability that an agent solves the same task in k attempts in a row. In the original paper, even GPT-4o solved fewer than 50% of tasks, and pass^8 in the retail domain was below 25%. Its successor, τ²-bench, adds scenarios where the user also has to take actions. The takeaway for deployments is simple: average success hides instability, and in production what counts is whether the agent works every time.

Serving: where things usually break

A common cause of "the model can't call tools" lies in serving, not in the model. The model emits a call in its own format (XML tags, JSON inside special tokens, Python-style syntax), and the server has to recognise it and turn it into the tool_calls field of the OpenAI-compatible API. If the parser doesn't match the model, the call lands as plain text and the agent "does nothing".

In vLLM, tool calling is enabled with --enable-auto-tool-choice and --tool-call-parser, and the parser has to match the model family:

Parser names change between vLLM versions and new models get new parsers, so check the docs for the version you actually run, and the model card. A few more things worth knowing:

• tool_choice="required" and named functions are implemented in vLLM through structured outputs. The docs say plainly that this guarantees a syntactically valid call, not a good one.
• Strict mode. With tool_choice="auto", arguments are constrained by the schema only if the tool sets strict: true. Without it the model generates the call freely and the server only extracts it from the text. Schemas for strict mode: additionalProperties: false, all fields required, optional fields as nullable.
• Parallel calls behave differently across families. vLLM doesn't support them for Llama 3, but does for Llama 4.
• llama.cpp (llama-server) supports tools when started with --jinja. It has native formats for some families and a generic format for the rest, which uses more tokens. The docs warn against aggressive KV cache quantization (such as -ctk q4_0), because it noticeably degrades calls.
• Gemma 4 has its own tool declaration and call tokens in the chat template. Google's function-calling guide makes a point that is easy to lose: the model executes nothing itself, your code does, so your code has to validate the function name and arguments.

For a broader look at memory requirements for serving models, see our guide to how much VRAM a local LLM needs.

How to design an agent for a local model

A large closed model forgives a lot: vague tool descriptions, twenty tools in context, a long plan. A local model forgives less, so the agent design has to take some of the work off it. Principles that work well here:

1. Few tools at a time. Every tool in context is another chance for a mistake. If the agent has many, split them into groups and pick the group first.
2. Descriptions written for the model. The tool name says what it does. The description says when to use it and when not to. An example of the arguments helps more than a long paragraph.
3. Unambiguous schemas. Enums instead of free text, one date format, identifiers instead of names.
4. Validation in code. Every argument is checked before execution. A validation error goes back to the model as a readable message, not a server exception.
5. Deterministic code where possible. If the order of steps is fixed, the model doesn't plan it. The model fills the gaps that need language understanding.
6. Step budget and loop breaker. An agent that calls the same tool a third time with the same arguments gets stopped.
7. Human approval before consequences. The agent can read on its own. Sending a letter, changing data or entering a deadline waits for "Approve". No decision means no.
8. Audit log and test set. Every step goes into a log. Every new version of the model, prompt or tools runs through a test set built from real conversations, several times over, so you see the instability and not just the average.

Some of these principles are also security. An agent with tools is a prompt injection target, and a local model is no more resistant than a cloud one. More in our posts on AI agent and MCP security and the OWASP Top 10 for LLM applications.

When a local model is enough, and when it isn't

If data can't leave the company, the alternative to a small model isn't the cloud, it's a bigger open model on your own server. GLM-4.6 and the large Qwen MoE models are close to the top of BFCL, but need correspondingly more GPU memory. How we match the model and hardware to an agent is described on the AI agents page.

The business context matters too. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027 due to costs, unclear value or weak risk controls. An agent on a local model that does one job well and predictably has a better chance of surviving than an ambitious do-everything agent.

Checklist before deploying an agent on a local model

1. List the tools and mark which only read and which have consequences.
2. Pick 2-3 candidate models from families that score well on BFCL, and check their model cards for the tool format.
3. Run them with the right parser and chat template. Check on a few examples that calls land in tool_calls, not in the message content.
4. Build a test set of 30-100 real tasks with the expected flow.
5. Run each task several times. Measure the share of tasks solved every time, not just the average.
6. Add argument validation, a step budget, a loop breaker and approval for actions with consequences.
7. Turn on an audit log for every step and review failures weekly.
8. Repeat the tests on every change of model, parser or server version.

Sources

• Berkeley Function Calling Leaderboard (BFCL) V4
• BFCL V4, Part 1: Web Search (agentic categories)
• Yao et al.: τ-bench, A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)
• Barres et al.: τ²-Bench, Evaluating Conversational Agents in a Dual-Control Environment (2025)
• vLLM: Tool Calling
• llama.cpp: Function calling
• Google AI for Developers: Function calling with Gemma 4
• Gartner: Over 40% of agentic AI projects will be canceled by end of 2027 (2025)