LLM APIs and function calling

A chat API is a stateless HTTP call: messages and JSON-Schema tool definitions in, text or tool calls out. Your code runs each call and sends the result back, so every round trip re-sends the history and costs one more request.

8 min read736 wordscore0 viewsupdated 2026-10-21

Definition

Function calling (tool use) is the API protocol in which a request includes tool definitions (a name, a description and a JSON Schema for the arguments), and the model may answer with a structured call, a tool name plus JSON arguments, instead of text. The client executes the call and returns the result in a tool message, and the model continues. OpenAI added it to the Chat Completions API in June 2023, and other providers adopted the same pattern. The research precursor is Toolformer (Schick et al., 2023), which taught a model to insert API calls into its own text.

A hosted model is reached through an HTTP endpoint. Each request carries the model id, the whole conversation as role-tagged messages (system, user, assistant, tool), sampling parameters and optional tools. The server keeps no conversation state between requests, so your application owns the history and re-sends it every time.

Tools turn the model from a text generator into a planner for your code. The model never executes anything. It writes a request to call a function, and your program decides whether to run it, runs it, and reports back. This article covers that wire protocol. Loops of many such calls are The agent loop and Tool use & function calling in the Agentic AI branch, and output-format guarantees are covered in Structured output (JSON mode).

Request and response

FieldOpenAI-style Chat CompletionsAnthropic Messages
Tool definition{"type": "function", "function": {name, description, parameters}}{name, description, input_schema}
Model asks for a callmessage.tool_calls[] with id, function.name, function.arguments (a JSON string)content block {"type": "tool_use", id, name, input} (a parsed object)
Stop signalfinish_reason: "tool_calls"stop_reason: "tool_use"
Returning a resultmessage {"role": "tool", "tool_call_id", "content"}user message with {"type": "tool_result", tool_use_id, content}, optional is_error: true
Choosing toolstool_choice: auto, none, required or a named functiontool_choice: auto, none, any or a named tool (with extended thinking, only auto and none)
Token usageusage.prompt_tokens, completion_tokensusage.input_tokens, output_tokens
The two common shapes (field names as documented by each provider; check current docs for additions)

Many self-hosted servers (vLLM, llama.cpp's server, Ollama) expose an OpenAI-compatible Chat Completions endpoint, so client code ports between them. Tool definitions are rendered into the prompt by the provider's chat template, so they are billed as input tokens on every request.

one tool-call round trip = two model requests (OpenAI-style chat completions)POST /v1/chat/completions{"model": "…", "messages": [{"role": "user", "content": "Weather in Paris and Oslo?"}], "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "required": ["city"], "properties": {"city": {"type": "string"}}}}}]}1 · app → APItools + messages"finish_reason": "tool_calls","message": {"role": "assistant", "content": null, "tool_calls": [ {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}}, {"id": "call_2", … "arguments": "{\"city\": \"Oslo\"}"}}]}2 · API → apptwo parallel callsjson.loads(arguments) → validate against the schema → callget_weather("Paris") ‖ get_weather("Oslo") (concurrently)→ {"temp_c": 18} {"temp_c": 9} errors go back as results3 · app runs toolsmodel runs nothingmessages = [user, assistant(tool_calls), tool results …]{"role": "tool", "tool_call_id": "call_1", "content": "{\"temp_c\": 18}"}{"role": "tool", "tool_call_id": "call_2", "content": "{\"temp_c\": 9}"}same tools list again: schemas are input tokens on every request4 · app → APIhistory re-sent"finish_reason": "stop","content": "Paris is 18 °C and Oslo is 9 °C."5 · API → appfinal answerarguments is a JSON string the model wrote: it can be malformed or off-schema unless strict mode is on
fig.One round trip for "Weather in Paris and Oslo?": request 1 carries the get_weather schema; the response stops with finish_reason "tool_calls" and two calls whose arguments are JSON strings; the app parses, validates and runs both; request 2 re-sends the history plus two tool messages keyed by tool_call_id; the response stops with "stop" and the answer. Two model requests, two tool executions.

What a round trip costs

Each model turn is a full request, and each request includes everything before it. If a task needs kk tool calls, the first prompt is PP tokens, each call is oo output tokens and each result tt tokens, then:

Iseq=(k+1)P+(o+t) k(k+1)2,Ipar=2P+k (o+t)I_{\text{seq}}=(k+1)P+(o+t)\,\frac{k(k+1)}{2},\qquad I_{\text{par}}=2P+k\,(o+t)
IseqI_{\text{seq}}
input tokens billed when the model makes one call per turn (k+1k+1 requests)
IparI_{\text{par}}
input tokens when all kk calls come back in one assistant message (2 requests)
PP
system prompt, user message and tool schemas

Sequential cost grows quadratically in kk because the history is re-sent. Parallel tool calls also cut latency: the kk tools run concurrently, so the turn waits for the slowest one rather than the sum. Prompt caching reduces the price of the repeated prefix but not the number of requests.

Protocol details that matter

Arguments are model output
OpenAI returns arguments as a string that may be invalid JSON or off-schema; strict: true on the function (Structured Outputs, August 2024) constrains it to the schema
Parallel calls
OpenAI added parallel function calling in November 2023 (parallel_tool_calls: false turns it off); Anthropic asks for all results of one turn in a single user message
Ids pair calls with results
every result must reference its call id; both providers reject (400) a history in which a call has no matching result
Retryable errors
429 (rate limit, often with a retry-after header), 500, 503, and Anthropic's 529 overloaded_error
Not retryable
400 (invalid request), 401/403 (auth, permission), 404: change the request instead

Errors and retries

Transient failures are routine at scale. Retry them with exponential backoff and jitter, and honour retry-after when it is present. "Full jitter" (Brooker, AWS Architecture Blog, 2015) draws each wait uniformly up to an exponentially growing cap. This spreads the retries of many clients so they do not hit the server again in lockstep.

wa∼U(0, min⁡(c, b⋅2a)),Pr⁡[task succeeds]=(1−f n+1)mw_a\sim U\big(0,\ \min(c,\ b\cdot2^{a})\big),\qquad \Pr[\text{task succeeds}]=\big(1-f^{\,n+1}\big)^{m}
waw_a
wait before retry a=0,1,…a=0,1,\dots with base bb and cap cc
ff
probability that one request fails transiently (assumed independent)
nn
maximum retries per request
mm
model requests in the task; every turn of a tool loop is one

Tool failures are different. A timeout in get_weather should not raise in your loop. Return it to the model as the tool result (with is_error where supported), and the model can retry with other arguments, pick another tool or tell the user. Retrying a whole request is safe only if the tools it already triggered are idempotent. Never re-run a payment because the follow-up model call returned a 503.

3 tool calls · sequential · tool 0.8 s · error rate f = 0.10 · ≤ 3 retriesmodeltools0123456789srun 1 of 2000 (first with a retry): done at 9.46 s · ■ model ■ 429/5xx ■ backoff ■ toolper task, 2000 seeded runssequentialparallelmodel requests42input tokens billed4,3402,170output tokens270270latency, no errors9.23 s6.92 smean latency (sim.)9.44 s7.03 sp95 latency (sim.)10.16 s7.55 sP(success) (1−fⁿ⁺¹)ᵐ99.96%99.98%success (sim.)100.00%100.00%attempts per task4.442.24input tokens per request (sequential)#1800#2990#31,180#41,370■ prompt + schemas 800 · ■ historyeach call adds 40 + 150 tokensmodel time = 0.25 s + 0.1 ms × input + 20 ms × output · final answer 150 tokensa failed request returns after 0.2 s; retry a waits U(0, 0.5 s × 2ᵃ) (full jitter)m = model requests per task, n = retries; failed attempts are not billed here
tool calling
fig.A task needing k tool calls, simulated 2,000 times with seeded transient errors. Defaults (k = 3, tool 0.8 s, f = 0.10, up to 3 retries): sequential takes 4 requests and 4,340 input tokens, 9.23 s with no errors and 9.44 s mean; parallel takes 2 requests, 2,170 tokens and 6.92 s (7.03 s mean). Set f = 0.20 and retries 0: sequential succeeds 40.96% of the time (0.8⁴), parallel 64%; with 3 retries, 99.36%. At k = 6 the token gap widens to 9,590 vs 2,740.

Worked example

The diagram's task with three cities instead of two, using the figure's numbers: P=800P=800, o=40o=40, t=150t=150, k=3k=3.

RequestSequential: contentsSequential inputParallel: contentsParallel input
1prompt + schemas800prompt + schemas800
2+ call 1 + result 1990+ 3 calls + 3 results800 + 3 × 190 = 1,370
3+ call 2 + result 21,180——
4+ call 3 + result 31,370——
Total4 × 800 + 190 × 64,3402 × 800 + 3 × 1902,170
Input tokens per request; each tool call adds o + t = 190 tokens of history

With a 20% transient error rate and no retries, the sequential task succeeds with probability 0.84=0.40960.8^4=0.4096. Allowing 3 retries makes each request fail only with 0.24=0.00160.2^4=0.0016, so the task succeeds with 0.99844=0.99360.9984^4=0.9936.

python
1import json, random, time
2
3class Transient(Exception): # 429, 500, 503 (and 529 on some providers)
4 def __init__(self, retry_after=None): self.retry_after = retry_after
5
6def with_retries(send, max_retries=3, base=0.5, cap=8.0):
7 for attempt in range(max_retries + 1):
8 try:
9 return send()
10 except Transient as e: # never retry 400/401/403: fix the request
11 if attempt == max_retries:
12 raise
13 wait = e.retry_after or random.uniform(0, min(cap, base * 2 ** attempt))
14 time.sleep(wait) # full jitter unless the server says retry-after
15
16TOOLS = {"get_weather": lambda city: {"city": city, "temp_c": {"Paris": 18, "Oslo": 9}[city]}}
17
18def run(messages, call_model, tools_schema, max_rounds=8):
19 for _ in range(max_rounds):
20 msg = with_retries(lambda: call_model(messages, tools_schema))
21 messages.append(msg)
22 if not msg.get("tool_calls"): # finish_reason "stop"
23 return msg["content"]
24 for call in msg["tool_calls"]: # parallel calls: may run concurrently
25 try:
26 args = json.loads(call["function"]["arguments"])
27 out = TOOLS[call["function"]["name"]](**args)
28 except Exception as e: # bad JSON, unknown tool, tool failure
29 out = {"error": f"{type(e).__name__}: {e}"}
30 messages.append({"role": "tool", "tool_call_id": call["id"],
31 "content": json.dumps(out)})
32 raise RuntimeError("too many tool rounds")
33
34# A fake API: fails once with 429, asks for two parallel calls, then answers.
35state = {"n": 0}
36def fake_model(messages, tools):
37 state["n"] += 1
38 if state["n"] == 1:
39 raise Transient(retry_after=0.01)
40 if messages[-1]["role"] == "user":
41 return {"role": "assistant", "content": None, "tool_calls": [
42 {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": '{"city": "Paris"}'}},
43 {"id": "call_2", "type": "function", "function": {"name": "get_weather", "arguments": '{"city": "Oslo"}'}}]}
44 temps = [json.loads(m["content"])["temp_c"] for m in messages if m["role"] == "tool"]
45 return {"role": "assistant", "content": f"Paris is {temps[0]} °C and Oslo is {temps[1]} °C."}
46
47print(run([{"role": "user", "content": "Weather in Paris and Oslo?"}], fake_model, tools_schema=[]))
48# Paris is 18 °C and Oslo is 9 °C.
A provider-neutral tool loop with retries and tool errors returned to the model. Runs as is against the fake API at the bottom; swap in your SDK's call for call_model

Where it's used

Data lookup

weather, orders, inventory or CRM records fetched on demand instead of stuffed into the prompt

Actions

creating tickets, sending messages or booking slots, ideally behind a human confirmation step

Extraction

a single tool whose parameters are the fields to extract; the call itself is the structured result

Routing

a classifier tool whose enum argument picks the downstream pipeline or model

Check yourself

The model returns two tool_calls in one assistant message. What should the next request contain?

Check yourself

Each request fails transiently with probability 0.1. A sequential tool loop makes 5 requests and you allow no retries. Roughly how often does the task fail?

Remember

  • ✓Stateless HTTP: you re-send the full message history and the tool schemas with every request.
  • ✓The model only writes calls (name + JSON arguments); your code validates, executes and returns results keyed by call id.
  • ✓Sequential loops cost (k+1)P+(o+t)k(k+1)/2(k+1)P+(o+t)k(k+1)/2 input tokens; parallel calls cost 2P+k(o+t)2P+k(o+t) and run tools concurrently.
  • ✓Retry 429/5xx with full-jitter exponential backoff and retry-after; never retry 400/401; task success is (1−fn+1)m(1-f^{n+1})^m.
  • ✓Return tool errors to the model as results, keep side-effecting tools idempotent or confirmed, and treat tool output as untrusted.
← back to Generative AI roadmap