# Conversational assistants

> A chat assistant rebuilds its whole prompt each turn: system prompt, memory, recent turns, tools. Resending full history costs tokens quadratic in turns; windows, summaries and retrieval cap the cost but decide which facts survive.

Source: https://www.mlfoundry.in/learn/conversational-assistants  
Section: Generative AI › Applications  
Level: core · 8 min read · updated 2026-10-21

**Conversational assistant:** is a multi-turn application around an instruction-tuned LLM. On every turn it assembles a prompt from a system prompt, conversation memory, retrieved documents and tool definitions, calls the model, executes any tool calls, checks the output and updates its state. The modern form dates from InstructGPT (Ouyang et al., 2022) and ChatGPT (November 2022), which tuned models with **RLHF** to follow instructions in dialogue.

Before LLMs, task-oriented assistants were pipelines: intent and slot classification, a dialogue-state tracker, a policy that picked the next action, and template text. They were reliable inside their domain and useless outside it. An LLM assistant replaces most of that pipeline with one model, and keeps the engineering in what goes into the prompt and what the model may do.

The model itself is stateless. A chat API takes a list of messages with roles (`system`, `user`, `assistant`, `tool`) and returns one new message. The server flattens the list with the model's **chat template**, for example ChatML's `<|im_start|>user … <|im_end|>` markers, which the model saw during **Instruction tuning**. Everything the assistant "remembers" is whatever the application chooses to resend.

## Anatomy of a turn

> Figure: Turn 23 of a travel conversation. The prompt is rebuilt from five parts totalling 2,745 tokens: a 600-token system prompt with tool schemas, a 190-token summary of old turns, one retrieved earlier turn, the last six turns verbatim and the new message. The model calls a booking tool, gets the result and is called again; the final text passes an output check, streams to the user, and the turn is folded into memory for the next call.

What goes into the prompt, and why

| Part | Contents | Design note |
| --- | --- | --- |
| System prompt | role, policies, tone, output format, tool schemas | stable across turns; put it first so it can be cached (see **System prompts and role prompting**) |
| Memory | rolling summary, user profile, retrieved past turns | decides which old facts survive |
| Grounding | retrieved documents, account data | see **RAG architecture**; cite sources in the reply |
| Recent turns | last W user/assistant pairs, verbatim | keeps pronouns and follow-ups resolvable |
| Tool results | JSON returned by functions the model called | see **LLM APIs and function calling** |

## The cost of remembering

If the application resends the full history, turn $t$ carries every earlier turn. With a system prompt of $s$ tokens and $u$ user and $a$ assistant tokens per turn, the prompt grows linearly and the total over a conversation grows quadratically:

$$
P_t=s+(t-1)(u+a)+u,\qquad \sum_{t=1}^{T}P_t=T(s+u)+(u+a)\,\frac{T(T-1)}{2}
$$

- $P_t$: prompt (input) tokens sent at turn t
- $s,\;u,\;a$: system-prompt tokens, user tokens per turn, assistant tokens per turn
- $T$: number of turns in the conversation

$$
P_t^{\text{win}}=s+\min(t-1,W)\,(u+a)+u,\qquad \sum_{t=1}^{T}P_t^{\text{win}}=T(s+u)+(u+a)\Big(WT-\frac{W(W+1)}{2}\Big)\ \ (T\ge W)
$$

- $W$: number of recent turns kept verbatim; the total now grows linearly in T

A window caps the cost but forgets everything older than $W$ turns. The usual fixes add a small memory that grows slowly. A **rolling summary** folds each turn that leaves the window into a short text, at the price of an extra LLM call and lossy compression. **Retrieval memory** embeds past turns and fetches the few most similar to the new message, at the price of misses when the wording differs. MemGPT (Packer et al., 2023) lets the model itself page facts in and out of an external store.

Long context does not remove the problem. Liu et al. (2023, "Lost in the middle") found that models use information at the start and end of a long input better than in the middle, so a fact from turn 3 buried in 30k tokens of history can be present yet ignored.

**Reference points**

- **InstructGPT (2022):** labelers preferred outputs of the 1.3B-parameter InstructGPT over those of the 175B GPT-3
- **Llama 2-Chat (2023):** Touvron et al. introduced Ghost Attention (GAtt), fine-tuning data that keeps a system instruction in force over many turns
- **MultiWOZ (2018):** about 10,000 human–human task dialogues across 7 domains (hotel, restaurant, train, …); the standard pre-LLM benchmark for dialogue-state tracking
- **MT-Bench (2023):** Zheng et al.: 80 two-turn questions in 8 categories, scored by an LLM judge; the second turn tests follow-ups (see **LLM-as-a-judge**)
- **Chat APIs:** stateless message lists with `system`/`user`/`assistant`/`tool` roles; the client resends history on every call

## Memory strategies in a simulation

> Figure: A 40-turn conversation (600-token system prompt, about 65 user and 201 assistant tokens per turn) in which the user states 8 facts and later asks about each. Default (full history, 8k limit): the prompt hits the limit at turn 29 and the oldest turns are dropped, so the name from turn 1 is lost when asked at turn 33: 7/8 recalled, 217,360 prompt tokens. A 32k limit keeps all 8 at 238,143 tokens (peak 11,055). A 6-turn window costs 84,781 tokens but recalls 2/8. Summary + window recalls 7/8 (the 5-fact summary evicts the oldest) for 90,451 + 18,212 summarizer tokens; retrieval + window recalls 7/8 for 112,138, missing the Lisbon turn.

## Worked example

Total prompt tokens, s = 600, u = 60, a = 200 (from the equations above)

| Turns T | Full history | Window W = 6 | Ratio |
| --- | --- | --- | --- |
| 40 | 26,400 + 260 × 780 = 229,200 | 26,400 + 260 × 219 = 83,340 | 2.75× |
| 100 | 66,000 + 260 × 4,950 = 1,353,000 | 66,000 + 260 × 579 = 216,540 | 6.25× |

Going from 40 to 100 turns multiplies the full-history cost by 5.9 but the windowed cost by only 2.6. At turn 100 a full-history prompt is $600+99\cdot260+60=26{,}400$ tokens, which no longer fits an 8k or 16k context window. **Prompt caching** makes a repeated prefix cheaper but does not change which facts fit.

```python
def n_tokens(text):                         # swap in the model's tokenizer
    return max(1, round(len(text) / 4))

class Conversation:
    """Sliding window of recent turns plus a rolling summary of older ones."""

    def __init__(self, system, llm, window=6, budget=8000):
        self.system, self.llm = system, llm
        self.window, self.budget = window, budget
        self.turns, self.summary = [], ""        # turns: list of (user, assistant)

    def messages(self, user_msg):
        msgs = [{"role": "system", "content": self.system}]
        if self.summary:
            msgs.append({"role": "system", "content": "Earlier in this conversation: " + self.summary})
        recent = self.turns[-self.window:]
        while recent and sum(n_tokens(m["content"]) for m in msgs) + sum(
                n_tokens(u) + n_tokens(a) for u, a in recent) + n_tokens(user_msg) > self.budget:
            recent = recent[1:]                  # hard cap: drop oldest verbatim turns
        for u, a in recent:
            msgs += [{"role": "user", "content": u}, {"role": "assistant", "content": a}]
        return msgs + [{"role": "user", "content": user_msg}]

    def send(self, user_msg):
        reply = self.llm(self.messages(user_msg))
        self.turns.append((user_msg, reply))
        if len(self.turns) > self.window:        # fold the turn leaving the window
            u, a = self.turns[-self.window - 1]
            self.summary = self.llm([
                {"role": "system", "content": "Update the summary. Keep names, numbers, "
                 "preferences and open tasks. At most 120 words."},
                {"role": "user", "content": f"Summary: {self.summary}\nUser: {u}\nAssistant: {a}"},
            ])
        return reply

echo = lambda msgs: "ok: " + msgs[-1]["content"][:40]   # offline stand-in for an API call
chat = Conversation("You are a travel assistant.", echo, window=2)
for m in ["I'm Priya.", "I'm vegetarian.", "Find flights from Lisbon.", "Book the 9:40 one."]:
    chat.send(m)
print([m["role"] for m in chat.messages("Thanks!")])
# ['system', 'system', 'user', 'assistant', 'user', 'assistant', 'user']
```

Window plus rolling-summary memory with a token budget (plain Python; run with an offline stand-in for the LLM call)

> **Summaries drift:** A rolling summary is rewritten by the model many times, and each rewrite can drop or distort a detail ("order 4417" becomes "an order"). Store hard facts such as IDs, names and preferences as structured fields the code maintains, not only as prose, and keep the original turns in a database so they can be retrieved or audited.

> **Order the prompt for caching:** Put what never changes first (system prompt, tool schemas), then what changes slowly (summary, profile), then the recent turns and the new message. Providers that cache prompt prefixes can then reuse most of each call, which matters because every turn resends that prefix (see **Prompt caching**).

- **Customer support:** answer from help-centre documents, look up orders through tools, hand off to a human with the conversation summary
- **Internal assistants:** search company knowledge with retrieval, respecting each user's document permissions
- **Task agents:** multi-turn tool use for booking, scheduling or data entry; see **The agent loop** in the Agentic AI branch

**Question:** A conversation with s = 600, u = 60, a = 200 runs for 40 turns. Roughly how many prompt tokens does full-history resending cost compared with keeping only the last 6 turns?

- A. The same
- B. About 1.5×
- C. About 2.75×
- D. About 40×

**Answer:** C. About 2.75× — Full history: 26,400 + 260 × 40 × 39 / 2 = 229,200. Window of 6: 26,400 + 260 × (240 − 21) = 83,340. The ratio is 2.75, and it keeps growing with T because one sum is quadratic and the other linear.

**Question:** Users report that the assistant forgets their dietary restriction after a long chat, even though the model's context window is large enough for the whole history. What is the most likely cause and fix?

- A. The temperature is too low; raise it
- B. The fact sits in the middle of a long prompt and is under-used; store it as a structured profile field and place it near the start of every prompt
- C. The chat template is wrong; switch models
- D. The tokenizer splits the word; spell it differently

**Answer:** B. The fact sits in the middle of a long prompt and is under-used; store it as a structured profile field and place it near the start of every prompt — Being in the context is not the same as being used: models attend less reliably to the middle of long inputs. Keeping key facts in a short, always-sent memory block near the system prompt makes them both present and salient.

**Key points**

- The model is stateless; the application rebuilds the prompt every turn from system prompt, memory, grounding, recent turns and tool results.
- Full history costs $T(s+u)+(u+a)T(T-1)/2$ tokens: quadratic in turns. A window of $W$ turns makes it linear.
- Windows forget, summaries drift and retrieval misses; most assistants combine a window, a summary and structured facts.
- A large context window does not guarantee use: facts in the middle of long prompts are used less reliably.
- Put stable content first for prompt caching; authorise tool calls and check outputs in code, not in the prompt.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
