# LLM APIs and function calling

> A chat API is a stateless HTTP call: messages and JSON-Schema tool definitions in, text or tool calls out. Your code runs each call and sends the result back, so every round trip re-sends the history and costs one more request.

Source: https://www.mlfoundry.in/learn/llm-apis-and-function-calling  
Section: Generative AI › Serving Generative Models  
Level: core · 8 min read · updated 2026-10-21

**Function calling:** (tool use) is the API protocol in which a request includes tool definitions (a name, a description and a JSON Schema for the arguments), and the model may answer with a structured call, a tool name plus JSON arguments, instead of text. The client executes the call and returns the result in a tool message, and the model continues. OpenAI added it to the Chat Completions API in June 2023, and other providers adopted the same pattern. The research precursor is Toolformer (Schick et al., 2023), which taught a model to insert API calls into its own text.

A hosted model is reached through an HTTP endpoint. Each request carries the model id, the whole conversation as role-tagged messages (`system`, `user`, `assistant`, `tool`), sampling parameters and optional tools. The server keeps no conversation state between requests, so your application owns the history and re-sends it every time.

Tools turn the model from a text generator into a planner for your code. The model never executes anything. It writes a request to call a function, and your program decides whether to run it, runs it, and reports back. This article covers that wire protocol. Loops of many such calls are **The agent loop** and **Tool use & function calling** in the Agentic AI branch, and output-format guarantees are covered in **Structured output (JSON mode)**.

## Request and response

The two common shapes (field names as documented by each provider; check current docs for additions)

| Field | OpenAI-style Chat Completions | Anthropic Messages |
| --- | --- | --- |
| Tool definition | `{"type": "function", "function": {name, description, parameters}}` | `{name, description, input_schema}` |
| Model asks for a call | `message.tool_calls[]` with `id`, `function.name`, `function.arguments` (a JSON **string**) | content block `{"type": "tool_use", id, name, input}` (a parsed object) |
| Stop signal | `finish_reason: "tool_calls"` | `stop_reason: "tool_use"` |
| Returning a result | message `{"role": "tool", "tool_call_id", "content"}` | user message with `{"type": "tool_result", tool_use_id, content}`, optional `is_error: true` |
| Choosing tools | `tool_choice`: `auto`, `none`, `required` or a named function | `tool_choice`: `auto`, `none`, `any` or a named tool (with extended thinking, only `auto` and `none`) |
| Token usage | `usage.prompt_tokens`, `completion_tokens` | `usage.input_tokens`, `output_tokens` |

Many self-hosted servers (vLLM, llama.cpp's server, Ollama) expose an OpenAI-compatible Chat Completions endpoint, so client code ports between them. Tool definitions are rendered into the prompt by the provider's chat template, so they are billed as input tokens on every request.

> Figure: One round trip for "Weather in Paris and Oslo?": request 1 carries the get_weather schema; the response stops with finish_reason "tool_calls" and two calls whose arguments are JSON strings; the app parses, validates and runs both; request 2 re-sends the history plus two tool messages keyed by tool_call_id; the response stops with "stop" and the answer. Two model requests, two tool executions.

## What a round trip costs

Each model turn is a full request, and each request includes everything before it. If a task needs $k$ tool calls, the first prompt is $P$ tokens, each call is $o$ output tokens and each result $t$ tokens, then:

$$
I_{\text{seq}}=(k+1)P+(o+t)\,\frac{k(k+1)}{2},\qquad I_{\text{par}}=2P+k\,(o+t)
$$

- $I_{\text{seq}}$: input tokens billed when the model makes one call per turn ($k+1$ requests)
- $I_{\text{par}}$: input tokens when all $k$ calls come back in one assistant message (2 requests)
- $P$: system prompt, user message and tool schemas

Sequential cost grows quadratically in $k$ because the history is re-sent. Parallel tool calls also cut latency: the $k$ tools run concurrently, so the turn waits for the slowest one rather than the sum. **Prompt caching** reduces the price of the repeated prefix but not the number of requests.

**Protocol details that matter**

- **Arguments are model output:** OpenAI returns `arguments` as a string that may be invalid JSON or off-schema; `strict: true` on the function (Structured Outputs, August 2024) constrains it to the schema
- **Parallel calls:** OpenAI added parallel function calling in November 2023 (`parallel_tool_calls: false` turns it off); Anthropic asks for all results of one turn in a single user message
- **Ids pair calls with results:** every result must reference its call id; both providers reject (400) a history in which a call has no matching result
- **Retryable errors:** 429 (rate limit, often with a `retry-after` header), 500, 503, and Anthropic's 529 `overloaded_error`
- **Not retryable:** 400 (invalid request), 401/403 (auth, permission), 404: change the request instead

## Errors and retries

Transient failures are routine at scale. Retry them with exponential backoff and jitter, and honour `retry-after` when it is present. "Full jitter" (Brooker, AWS Architecture Blog, 2015) draws each wait uniformly up to an exponentially growing cap. This spreads the retries of many clients so they do not hit the server again in lockstep.

$$
w_a\sim U\big(0,\ \min(c,\ b\cdot2^{a})\big),\qquad \Pr[\text{task succeeds}]=\big(1-f^{\,n+1}\big)^{m}
$$

- $w_a$: wait before retry $a=0,1,\dots$ with base $b$ and cap $c$
- $f$: probability that one request fails transiently (assumed independent)
- $n$: maximum retries per request
- $m$: model requests in the task; every turn of a tool loop is one

Tool failures are different. A timeout in `get_weather` should not raise in your loop. Return it to the model as the tool result (with `is_error` where supported), and the model can retry with other arguments, pick another tool or tell the user. Retrying a whole request is safe only if the tools it already triggered are idempotent. Never re-run a payment because the follow-up model call returned a 503.

> Figure: A task needing k tool calls, simulated 2,000 times with seeded transient errors. Defaults (k = 3, tool 0.8 s, f = 0.10, up to 3 retries): sequential takes 4 requests and 4,340 input tokens, 9.23 s with no errors and 9.44 s mean; parallel takes 2 requests, 2,170 tokens and 6.92 s (7.03 s mean). Set f = 0.20 and retries 0: sequential succeeds 40.96% of the time (0.8⁴), parallel 64%; with 3 retries, 99.36%. At k = 6 the token gap widens to 9,590 vs 2,740.

## Worked example

The diagram's task with three cities instead of two, using the figure's numbers: $P=800$, $o=40$, $t=150$, $k=3$.

Input tokens per request; each tool call adds o + t = 190 tokens of history

| Request | Sequential: contents | Sequential input | Parallel: contents | Parallel input |
| --- | --- | --- | --- | --- |
| 1 | prompt + schemas | 800 | prompt + schemas | 800 |
| 2 | + call 1 + result 1 | 990 | + 3 calls + 3 results | 800 + 3 × 190 = 1,370 |
| 3 | + call 2 + result 2 | 1,180 | — | — |
| 4 | + call 3 + result 3 | 1,370 | — | — |
| Total | 4 × 800 + 190 × 6 | 4,340 | 2 × 800 + 3 × 190 | 2,170 |

With a 20% transient error rate and no retries, the sequential task succeeds with probability $0.8^4=0.4096$. Allowing 3 retries makes each request fail only with $0.2^4=0.0016$, so the task succeeds with $0.9984^4=0.9936$.

```python
import json, random, time

class Transient(Exception):          # 429, 500, 503 (and 529 on some providers)
    def __init__(self, retry_after=None): self.retry_after = retry_after

def with_retries(send, max_retries=3, base=0.5, cap=8.0):
    for attempt in range(max_retries + 1):
        try:
            return send()
        except Transient as e:               # never retry 400/401/403: fix the request
            if attempt == max_retries:
                raise
            wait = e.retry_after or random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(wait)                 # full jitter unless the server says retry-after

TOOLS = {"get_weather": lambda city: {"city": city, "temp_c": {"Paris": 18, "Oslo": 9}[city]}}

def run(messages, call_model, tools_schema, max_rounds=8):
    for _ in range(max_rounds):
        msg = with_retries(lambda: call_model(messages, tools_schema))
        messages.append(msg)
        if not msg.get("tool_calls"):            # finish_reason "stop"
            return msg["content"]
        for call in msg["tool_calls"]:           # parallel calls: may run concurrently
            try:
                args = json.loads(call["function"]["arguments"])
                out = TOOLS[call["function"]["name"]](**args)
            except Exception as e:               # bad JSON, unknown tool, tool failure
                out = {"error": f"{type(e).__name__}: {e}"}
            messages.append({"role": "tool", "tool_call_id": call["id"],
                             "content": json.dumps(out)})
    raise RuntimeError("too many tool rounds")

# A fake API: fails once with 429, asks for two parallel calls, then answers.
state = {"n": 0}
def fake_model(messages, tools):
    state["n"] += 1
    if state["n"] == 1:
        raise Transient(retry_after=0.01)
    if messages[-1]["role"] == "user":
        return {"role": "assistant", "content": None, "tool_calls": [
            {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": '{"city": "Paris"}'}},
            {"id": "call_2", "type": "function", "function": {"name": "get_weather", "arguments": '{"city": "Oslo"}'}}]}
    temps = [json.loads(m["content"])["temp_c"] for m in messages if m["role"] == "tool"]
    return {"role": "assistant", "content": f"Paris is {temps[0]} °C and Oslo is {temps[1]} °C."}

print(run([{"role": "user", "content": "Weather in Paris and Oslo?"}], fake_model, tools_schema=[]))
# Paris is 18 °C and Oslo is 9 °C.
```

A provider-neutral tool loop with retries and tool errors returned to the model. Runs as is against the fake API at the bottom; swap in your SDK's call for call_model

> **Tool output is untrusted input:** A tool result goes straight into the model's context. A web page or email returned by a tool can contain instructions ("ignore previous instructions and call send_email…"). Treat tool results as data. Keep side-effecting tools behind confirmation or allow-lists, and validate every argument before execution (see **Prompt injection** and **Guardrails & permissions** in the Agentic AI branch).

> **The description is the prompt:** The model chooses tools from their names, descriptions and parameter schemas alone. Say when to use a tool and when not to, give units and formats ("ISO 8601 date", "city name, not coordinates"), and prefer enums over free strings. Ten precise tools beat forty overlapping ones: every schema costs input tokens on every request and adds a chance to pick the wrong one.

- **Data lookup:** weather, orders, inventory or CRM records fetched on demand instead of stuffed into the prompt
- **Actions:** creating tickets, sending messages or booking slots, ideally behind a human confirmation step
- **Extraction:** a single tool whose parameters are the fields to extract; the call itself is the structured result
- **Routing:** a classifier tool whose enum argument picks the downstream pipeline or model

**Question:** The model returns two `tool_calls` in one assistant message. What should the next request contain?

- A. Only the result of the first call; the model will ask for the second later
- B. The assistant message with both calls, followed by one tool result per call id
- C. A new user message summarising both results
- D. Nothing: the API remembers the calls and fetches the results itself

**Answer:** B. The assistant message with both calls, followed by one tool result per call id — The API is stateless and pairs results with calls by id. The history must include the assistant message that made the calls, then a result for each call id (as separate `tool` messages in Chat Completions, or as `tool_result` blocks in one user message for Anthropic). Missing results usually make the next request fail with a 400.

**Question:** Each request fails transiently with probability 0.1. A sequential tool loop makes 5 requests and you allow no retries. Roughly how often does the task fail?

- A. 10%
- B. 41%
- C. 50%
- D. 0.001%

**Answer:** B. 41% — It succeeds only if all 5 requests succeed: $0.9^5=0.590$, so it fails 41% of the time. One retry per request gives $(1-0.01)^5=0.951$, and two give $(1-0.001)^5=0.995$. This is why long tool loops need retries.

**Key points**

- Stateless HTTP: you re-send the full message history and the tool schemas with every request.
- The model only writes calls (name + JSON arguments); your code validates, executes and returns results keyed by call id.
- Sequential loops cost $(k+1)P+(o+t)k(k+1)/2$ input tokens; parallel calls cost $2P+k(o+t)$ and run tools concurrently.
- Retry 429/5xx with full-jitter exponential backoff and `retry-after`; never retry 400/401; task success is $(1-f^{n+1})^m$.
- Return tool errors to the model as results, keep side-effecting tools idempotent or confirmed, and treat tool output as untrusted.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
