Definition
Function calling (tool use) is the API protocol in which a request includes tool definitions (a name, a description and a JSON Schema for the arguments), and the model may answer with a structured call, a tool name plus JSON arguments, instead of text. The client executes the call and returns the result in a tool message, and the model continues. OpenAI added it to the Chat Completions API in June 2023, and other providers adopted the same pattern. The research precursor is Toolformer (Schick et al., 2023), which taught a model to insert API calls into its own text.
A hosted model is reached through an HTTP endpoint. Each request carries the model id, the whole conversation as role-tagged messages (system, user, assistant, tool), sampling parameters and optional tools. The server keeps no conversation state between requests, so your application owns the history and re-sends it every time.
Tools turn the model from a text generator into a planner for your code. The model never executes anything. It writes a request to call a function, and your program decides whether to run it, runs it, and reports back. This article covers that wire protocol. Loops of many such calls are The agent loop and Tool use & function calling in the Agentic AI branch, and output-format guarantees are covered in Structured output (JSON mode).
Request and response
| Field | OpenAI-style Chat Completions | Anthropic Messages |
|---|---|---|
| Tool definition | {"type": "function", "function": {name, description, parameters}} | {name, description, input_schema} |
| Model asks for a call | message.tool_calls[] with id, function.name, function.arguments (a JSON string) | content block {"type": "tool_use", id, name, input} (a parsed object) |
| Stop signal | finish_reason: "tool_calls" | stop_reason: "tool_use" |
| Returning a result | message {"role": "tool", "tool_call_id", "content"} | user message with {"type": "tool_result", tool_use_id, content}, optional is_error: true |
| Choosing tools | tool_choice: auto, none, required or a named function | tool_choice: auto, none, any or a named tool (with extended thinking, only auto and none) |
| Token usage | usage.prompt_tokens, completion_tokens | usage.input_tokens, output_tokens |
Many self-hosted servers (vLLM, llama.cpp's server, Ollama) expose an OpenAI-compatible Chat Completions endpoint, so client code ports between them. Tool definitions are rendered into the prompt by the provider's chat template, so they are billed as input tokens on every request.
What a round trip costs
Each model turn is a full request, and each request includes everything before it. If a task needs tool calls, the first prompt is tokens, each call is output tokens and each result tokens, then:
- input tokens billed when the model makes one call per turn ( requests)
- input tokens when all calls come back in one assistant message (2 requests)
- system prompt, user message and tool schemas
Sequential cost grows quadratically in because the history is re-sent. Parallel tool calls also cut latency: the tools run concurrently, so the turn waits for the slowest one rather than the sum. Prompt caching reduces the price of the repeated prefix but not the number of requests.
Protocol details that matter
- Arguments are model output
- OpenAI returns
argumentsas a string that may be invalid JSON or off-schema;strict: trueon the function (Structured Outputs, August 2024) constrains it to the schema - Parallel calls
- OpenAI added parallel function calling in November 2023 (
parallel_tool_calls: falseturns it off); Anthropic asks for all results of one turn in a single user message - Ids pair calls with results
- every result must reference its call id; both providers reject (400) a history in which a call has no matching result
- Retryable errors
- 429 (rate limit, often with a
retry-afterheader), 500, 503, and Anthropic's 529overloaded_error - Not retryable
- 400 (invalid request), 401/403 (auth, permission), 404: change the request instead
Errors and retries
Transient failures are routine at scale. Retry them with exponential backoff and jitter, and honour retry-after when it is present. "Full jitter" (Brooker, AWS Architecture Blog, 2015) draws each wait uniformly up to an exponentially growing cap. This spreads the retries of many clients so they do not hit the server again in lockstep.
- wait before retry with base and cap
- probability that one request fails transiently (assumed independent)
- maximum retries per request
- model requests in the task; every turn of a tool loop is one
Tool failures are different. A timeout in get_weather should not raise in your loop. Return it to the model as the tool result (with is_error where supported), and the model can retry with other arguments, pick another tool or tell the user. Retrying a whole request is safe only if the tools it already triggered are idempotent. Never re-run a payment because the follow-up model call returned a 503.
Worked example
The diagram's task with three cities instead of two, using the figure's numbers: , , , .
| Request | Sequential: contents | Sequential input | Parallel: contents | Parallel input |
|---|---|---|---|---|
| 1 | prompt + schemas | 800 | prompt + schemas | 800 |
| 2 | + call 1 + result 1 | 990 | + 3 calls + 3 results | 800 + 3 × 190 = 1,370 |
| 3 | + call 2 + result 2 | 1,180 | — | — |
| 4 | + call 3 + result 3 | 1,370 | — | — |
| Total | 4 × 800 + 190 × 6 | 4,340 | 2 × 800 + 3 × 190 | 2,170 |
With a 20% transient error rate and no retries, the sequential task succeeds with probability . Allowing 3 retries makes each request fail only with , so the task succeeds with .
1import json, random, time23class Transient(Exception): # 429, 500, 503 (and 529 on some providers)4 def __init__(self, retry_after=None): self.retry_after = retry_after56def with_retries(send, max_retries=3, base=0.5, cap=8.0):7 for attempt in range(max_retries + 1):8 try:9 return send()10 except Transient as e: # never retry 400/401/403: fix the request11 if attempt == max_retries:12 raise13 wait = e.retry_after or random.uniform(0, min(cap, base * 2 ** attempt))14 time.sleep(wait) # full jitter unless the server says retry-after1516TOOLS = {"get_weather": lambda city: {"city": city, "temp_c": {"Paris": 18, "Oslo": 9}[city]}}1718def run(messages, call_model, tools_schema, max_rounds=8):19 for _ in range(max_rounds):20 msg = with_retries(lambda: call_model(messages, tools_schema))21 messages.append(msg)22 if not msg.get("tool_calls"): # finish_reason "stop"23 return msg["content"]24 for call in msg["tool_calls"]: # parallel calls: may run concurrently25 try:26 args = json.loads(call["function"]["arguments"])27 out = TOOLS[call["function"]["name"]](**args)28 except Exception as e: # bad JSON, unknown tool, tool failure29 out = {"error": f"{type(e).__name__}: {e}"}30 messages.append({"role": "tool", "tool_call_id": call["id"],31 "content": json.dumps(out)})32 raise RuntimeError("too many tool rounds")3334# A fake API: fails once with 429, asks for two parallel calls, then answers.35state = {"n": 0}36def fake_model(messages, tools):37 state["n"] += 138 if state["n"] == 1:39 raise Transient(retry_after=0.01)40 if messages[-1]["role"] == "user":41 return {"role": "assistant", "content": None, "tool_calls": [42 {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": '{"city": "Paris"}'}},43 {"id": "call_2", "type": "function", "function": {"name": "get_weather", "arguments": '{"city": "Oslo"}'}}]}44 temps = [json.loads(m["content"])["temp_c"] for m in messages if m["role"] == "tool"]45 return {"role": "assistant", "content": f"Paris is {temps[0]} °C and Oslo is {temps[1]} °C."}4647print(run([{"role": "user", "content": "Weather in Paris and Oslo?"}], fake_model, tools_schema=[]))48# Paris is 18 °C and Oslo is 9 °C.Where it's used
Data lookup
weather, orders, inventory or CRM records fetched on demand instead of stuffed into the prompt
Actions
creating tickets, sending messages or booking slots, ideally behind a human confirmation step
Extraction
a single tool whose parameters are the fields to extract; the call itself is the structured result
Routing
a classifier tool whose enum argument picks the downstream pipeline or model
Check yourself
The model returns two tool_calls in one assistant message. What should the next request contain?
Check yourself
Each request fails transiently with probability 0.1. A sequential tool loop makes 5 requests and you allow no retries. Roughly how often does the task fail?
Remember
- ✓Stateless HTTP: you re-send the full message history and the tool schemas with every request.
- ✓The model only writes calls (name + JSON arguments); your code validates, executes and returns results keyed by call id.
- ✓Sequential loops cost input tokens; parallel calls cost and run tools concurrently.
- ✓Retry 429/5xx with full-jitter exponential backoff and
retry-after; never retry 400/401; task success is . - ✓Return tool errors to the model as results, keep side-effecting tools idempotent or confirmed, and treat tool output as untrusted.