What Is Function Calling in LLMs? APIs and Agents

Key takeaways

• Function calling lets a language model request a specific tool or action using structured arguments, rather than just generating text.
• Your application validates and executes the request. The model does not automatically gain access to an API, database, or account.
• Function calling supports AI agents, but useful agents also need permissions, error handling, and clear stopping rules.

Function calling in LLMs is a way for a large language model to request that software run a predefined function, such as searching the web or checking an order. It connects conversational requests to application code, allowing an AI assistant to retrieve information or initiate actions under rules you control.

How it works

A regular chatbot generates a text response. A chatbot with function calling can instead produce a structured request for another piece of software to handle.

The typical flow has five steps:

1. Define the available functions. Your application describes each function’s purpose, accepted inputs, and required fields, often using JSON Schema.
2. Send the user’s request. The model receives the conversation and the available function definitions.
3. Generate a function call. If appropriate, the model selects a function and supplies arguments.
4. Validate and execute it. Your application checks the request, verifies permissions, and runs the code.
5. Return the result. The application sends the result back to the model, which can explain it or request another permitted step.

Suppose a user asks, “Where is order A123?” Your application might expose a function called `get_order_status`. A simplified, illustrative call could look like this:

```json

{

"name": "get_order_status",

"arguments": {

"order_id": "A123"

}

}

```

This is a request to execute code—not proof that the order was checked. Your backend must confirm that the user can access that order, query the relevant system, and return the result.

Exact message formats vary between model providers.

Where an API fits

A function is a callable operation in your application. An API is an interface through which software communicates with another system.

A function can wrap an API request, query a local database, perform a calculation, or read an approved file. Function calling therefore does not always require an external API.

The important boundary is that the model proposes an operation; your software controls execution.

Real-world examples

Research assistants

A research assistant could turn “Find recent announcements about battery recycling” into a search request.

Your application might expose a search function backed by Exa or Tavily, both of which offer search APIs designed for AI workflows. The model can use the returned material to draft an answer grounded in retrieved sources.

Search results still need evaluation. Retrieving a page does not establish that its claims are accurate or current.

Internal knowledge assistants

An employee might ask, “What is our process for approving travel?”

A function could search an internal document collection using Qdrant, an open-source vector similarity search engine with API SDKs. The application retrieves relevant passages, and the model uses them to answer.

This combines function calling with retrieval-augmented generation, or RAG. Function calling requests the search; retrieval supplies context. Access controls should restrict results to documents the employee is allowed to see.

Website research workflows

An analyst might ask an assistant to summarize several supplier websites.

A function backed by Firecrawl could retrieve website content as clean Markdown or structured data. The model then compares the returned information.

Keep the distinction between reading and acting clear. Gathering public product information is different from submitting a form or changing an account setting.

Customer support actions

A support assistant could check an order, retrieve a return policy, and prepare a refund request.

Looking up information may be low risk. Issuing a refund is not. A safer workflow separates the lookup from the write action and requires explicit approval before money moves.

Benefits and limits

Function calling makes assistants more useful. They can work with current information, perform calculations through code, and interact with business systems rather than relying only on model training.

It gives developers a clearer interface. Named functions and defined inputs are easier to validate than free-form instructions such as “please update the customer record.”

It supports multi-step agents. An agent can request a search, examine the result, and choose another action. Function calling supplies an action interface; the surrounding application supplies the workflow, permissions, and stopping conditions.

However, structure is not the same as correctness. A model can choose the wrong function, supply a nonexistent identifier, or misunderstand the user’s intent. Even valid JSON can describe an inappropriate action.

External services can also fail, return incomplete data, or add latency and cost. Set timeouts, limit retries, and handle errors explicitly.

Security matters throughout. Treat retrieved pages and tool results as untrusted data, not instructions that can override your application’s rules. Keep API credentials on the server, apply least-privilege access, and require approval for consequential actions.

How to get started

Start with one narrow, read-only task rather than a general-purpose autonomous agent.

1. Choose a clear outcome. For example: “Search approved help articles and answer a support question.”
2. Write a small function definition. Use a descriptive name, explain when it should be used, and specify required inputs.
3. Implement the backend. Connect the function to your chosen API or database. Keep credentials out of prompts.
4. Validate every request. Check argument types, allowed values, user permissions, and resource ownership.
5. Return useful results. Provide concise data and explicit errors. Avoid sending unnecessary personal information to the model.
6. Test failure cases. Try ambiguous questions, missing records, service outages, unauthorized requests, and malicious instructions embedded in retrieved content.
7. Add operational limits. Cap tool calls, set timeouts, and log enough information to investigate problems without exposing secrets.

Choose tools based on the task: web search, website extraction, and internal document retrieval need different backends. A directory listing alone does not mean a service is automatically connected to your model; you still need an integration layer.

FAQ

Is function calling the same as tool calling?

The terms often overlap. “Tool calling” can be broader, covering provider-managed capabilities as well as developer-defined functions. Check your model provider’s terminology and execution model.

Does the LLM execute the function itself?

Usually, the model generates a structured request and your application executes it. Some platforms manage execution for certain built-in tools, but software outside the model still performs the operation.

Does function calling make an assistant an agent?

Not by itself. A single function call can simply retrieve information. An agent typically uses a loop to select actions, inspect results, and continue toward a goal, subject to permissions and stopping rules.