Conversational assistants

A chat assistant rebuilds its whole prompt each turn: system prompt, memory, recent turns, tools. Resending full history costs tokens quadratic in turns; windows, summaries and retrieval cap the cost but decide which facts survive.

8 min read618 wordscore0 viewsupdated 2026-10-21

Definition

Conversational assistant is a multi-turn application around an instruction-tuned LLM. On every turn it assembles a prompt from a system prompt, conversation memory, retrieved documents and tool definitions, calls the model, executes any tool calls, checks the output and updates its state. The modern form dates from InstructGPT (Ouyang et al., 2022) and ChatGPT (November 2022), which tuned models with RLHF to follow instructions in dialogue.

Before LLMs, task-oriented assistants were pipelines: intent and slot classification, a dialogue-state tracker, a policy that picked the next action, and template text. They were reliable inside their domain and useless outside it. An LLM assistant replaces most of that pipeline with one model, and keeps the engineering in what goes into the prompt and what the model may do.

The model itself is stateless. A chat API takes a list of messages with roles (system, user, assistant, tool) and returns one new message. The server flattens the list with the model's chat template, for example ChatML's <|im_start|>user … <|im_end|> markers, which the model saw during Instruction tuning. Everything the assistant "remembers" is whatever the application chooses to resend.

Anatomy of a turn

the API is stateless: every turn rebuilds the whole promptprompt for this call · 2,745 tokenssystem prompt600role, policies, tone,tool schemasmemory summary190Priya; vegetarian;budget €800; child 6retrieved turn 11260"…flying from Lisbonon the 14th…"last 6 turns1,650user/assistant pairs17–22, verbatimnew user message45"Book the 9:40 one."user · turn 23Book the 9:40one.LLM callchat template:roles → tokenstool call (JSON)book_flightflight: TP1350seat: 14C← {status: ok}output checkmoderation, policy,PII redactionstream reply"Booked TP1350, seat14C, total €412."update stateappend turn,refresh summaryfinal textnextturntool result is appendedand the model is calledagain
fig.Turn 23 of a travel conversation. The prompt is rebuilt from five parts totalling 2,745 tokens: a 600-token system prompt with tool schemas, a 190-token summary of old turns, one retrieved earlier turn, the last six turns verbatim and the new message. The model calls a booking tool, gets the result and is called again; the final text passes an output check, streams to the user, and the turn is folded into memory for the next call.
PartContentsDesign note
System promptrole, policies, tone, output format, tool schemasstable across turns; put it first so it can be cached (see System prompts and role prompting)
Memoryrolling summary, user profile, retrieved past turnsdecides which old facts survive
Groundingretrieved documents, account datasee RAG architecture; cite sources in the reply
Recent turnslast W user/assistant pairs, verbatimkeeps pronouns and follow-ups resolvable
Tool resultsJSON returned by functions the model calledsee LLM APIs and function calling
What goes into the prompt, and why

The cost of remembering

If the application resends the full history, turn tt carries every earlier turn. With a system prompt of ss tokens and uu user and aa assistant tokens per turn, the prompt grows linearly and the total over a conversation grows quadratically:

Pt=s+(t−1)(u+a)+u,∑t=1TPt=T(s+u)+(u+a) T(T−1)2P_t=s+(t-1)(u+a)+u,\qquad \sum_{t=1}^{T}P_t=T(s+u)+(u+a)\,\frac{T(T-1)}{2}
PtP_t
prompt (input) tokens sent at turn t
s,  u,  as,\;u,\;a
system-prompt tokens, user tokens per turn, assistant tokens per turn
TT
number of turns in the conversation
Ptwin=s+min⁡(t−1,W) (u+a)+u,∑t=1TPtwin=T(s+u)+(u+a)(WT−W(W+1)2)  (T≥W)P_t^{\text{win}}=s+\min(t-1,W)\,(u+a)+u,\qquad \sum_{t=1}^{T}P_t^{\text{win}}=T(s+u)+(u+a)\Big(WT-\frac{W(W+1)}{2}\Big)\ \ (T\ge W)
WW
number of recent turns kept verbatim; the total now grows linearly in T

A window caps the cost but forgets everything older than WW turns. The usual fixes add a small memory that grows slowly. A rolling summary folds each turn that leaves the window into a short text, at the price of an extra LLM call and lossy compression. Retrieval memory embeds past turns and fetches the few most similar to the new message, at the price of misses when the wording differs. MemGPT (Packer et al., 2023) lets the model itself page facts in and out of an external store.

Long context does not remove the problem. Liu et al. (2023, "Lost in the middle") found that models use information at the start and end of a long input better than in the middle, so a fact from turn 3 buried in 30k tokens of history can be present yet ignored.

Reference points

InstructGPT (2022)
labelers preferred outputs of the 1.3B-parameter InstructGPT over those of the 175B GPT-3
Llama 2-Chat (2023)
Touvron et al. introduced Ghost Attention (GAtt), fine-tuning data that keeps a system instruction in force over many turns
MultiWOZ (2018)
about 10,000 human–human task dialogues across 7 domains (hotel, restaurant, train, …); the standard pre-LLM benchmark for dialogue-state tracking
MT-Bench (2023)
Zheng et al.: 80 two-turn questions in 8 categories, scored by an LLM judge; the second turn tests follow-ups (see LLM-as-a-judge)
Chat APIs
stateless message lists with system/user/assistant/tool roles; the client resends history on every call

Memory strategies in a simulation

full history · context limit 8,000prompt tokens per turn · ■ system ■ memory ■ history ■ new message04k8k12klimit 8,000: oldest turns dropped (faded bars)110203040turnstated → asked aboutname: Priyalostvegetarianseenorder #4417seenbudget €800seenflying from Lisbonseenallergic to nutsseenaisle seatseenchild aged 6seenfacts recalled7 / 8prompt tokens, 40 turns217,360+ summarizer0peak prompt7,997
memory
context limit
fig.A 40-turn conversation (600-token system prompt, about 65 user and 201 assistant tokens per turn) in which the user states 8 facts and later asks about each. Default (full history, 8k limit): the prompt hits the limit at turn 29 and the oldest turns are dropped, so the name from turn 1 is lost when asked at turn 33: 7/8 recalled, 217,360 prompt tokens. A 32k limit keeps all 8 at 238,143 tokens (peak 11,055). A 6-turn window costs 84,781 tokens but recalls 2/8. Summary + window recalls 7/8 (the 5-fact summary evicts the oldest) for 90,451 + 18,212 summarizer tokens; retrieval + window recalls 7/8 for 112,138, missing the Lisbon turn.

Worked example

Turns TFull historyWindow W = 6Ratio
4026,400 + 260 × 780 = 229,20026,400 + 260 × 219 = 83,3402.75×
10066,000 + 260 × 4,950 = 1,353,00066,000 + 260 × 579 = 216,5406.25×
Total prompt tokens, s = 600, u = 60, a = 200 (from the equations above)

Going from 40 to 100 turns multiplies the full-history cost by 5.9 but the windowed cost by only 2.6. At turn 100 a full-history prompt is 600+99⋅260+60=26,400600+99\cdot260+60=26{,}400 tokens, which no longer fits an 8k or 16k context window. Prompt caching makes a repeated prefix cheaper but does not change which facts fit.

python
1def n_tokens(text): # swap in the model's tokenizer
2 return max(1, round(len(text) / 4))
3
4class Conversation:
5 """Sliding window of recent turns plus a rolling summary of older ones."""
6
7 def __init__(self, system, llm, window=6, budget=8000):
8 self.system, self.llm = system, llm
9 self.window, self.budget = window, budget
10 self.turns, self.summary = [], "" # turns: list of (user, assistant)
11
12 def messages(self, user_msg):
13 msgs = [{"role": "system", "content": self.system}]
14 if self.summary:
15 msgs.append({"role": "system", "content": "Earlier in this conversation: " + self.summary})
16 recent = self.turns[-self.window:]
17 while recent and sum(n_tokens(m["content"]) for m in msgs) + sum(
18 n_tokens(u) + n_tokens(a) for u, a in recent) + n_tokens(user_msg) > self.budget:
19 recent = recent[1:] # hard cap: drop oldest verbatim turns
20 for u, a in recent:
21 msgs += [{"role": "user", "content": u}, {"role": "assistant", "content": a}]
22 return msgs + [{"role": "user", "content": user_msg}]
23
24 def send(self, user_msg):
25 reply = self.llm(self.messages(user_msg))
26 self.turns.append((user_msg, reply))
27 if len(self.turns) > self.window: # fold the turn leaving the window
28 u, a = self.turns[-self.window - 1]
29 self.summary = self.llm([
30 {"role": "system", "content": "Update the summary. Keep names, numbers, "
31 "preferences and open tasks. At most 120 words."},
32 {"role": "user", "content": f"Summary: {self.summary}\nUser: {u}\nAssistant: {a}"},
33 ])
34 return reply
35
36echo = lambda msgs: "ok: " + msgs[-1]["content"][:40] # offline stand-in for an API call
37chat = Conversation("You are a travel assistant.", echo, window=2)
38for m in ["I'm Priya.", "I'm vegetarian.", "Find flights from Lisbon.", "Book the 9:40 one."]:
39 chat.send(m)
40print([m["role"] for m in chat.messages("Thanks!")])
41# ['system', 'system', 'user', 'assistant', 'user', 'assistant', 'user']
Window plus rolling-summary memory with a token budget (plain Python; run with an offline stand-in for the LLM call)

Where it's used

Customer support

answer from help-centre documents, look up orders through tools, hand off to a human with the conversation summary

Internal assistants

search company knowledge with retrieval, respecting each user's document permissions

Task agents

multi-turn tool use for booking, scheduling or data entry; see The agent loop in the Agentic AI branch

Check yourself

A conversation with s = 600, u = 60, a = 200 runs for 40 turns. Roughly how many prompt tokens does full-history resending cost compared with keeping only the last 6 turns?

Check yourself

Users report that the assistant forgets their dietary restriction after a long chat, even though the model's context window is large enough for the whole history. What is the most likely cause and fix?

Remember

  • ✓The model is stateless; the application rebuilds the prompt every turn from system prompt, memory, grounding, recent turns and tool results.
  • ✓Full history costs T(s+u)+(u+a)T(T−1)/2T(s+u)+(u+a)T(T-1)/2 tokens: quadratic in turns. A window of WW turns makes it linear.
  • ✓Windows forget, summaries drift and retrieval misses; most assistants combine a window, a summary and structured facts.
  • ✓A large context window does not guarantee use: facts in the middle of long prompts are used less reliably.
  • ✓Put stable content first for prompt caching; authorise tool calls and check outputs in code, not in the prompt.
← back to Generative AI roadmap