Definition
Conversational assistant is a multi-turn application around an instruction-tuned LLM. On every turn it assembles a prompt from a system prompt, conversation memory, retrieved documents and tool definitions, calls the model, executes any tool calls, checks the output and updates its state. The modern form dates from InstructGPT (Ouyang et al., 2022) and ChatGPT (November 2022), which tuned models with RLHF to follow instructions in dialogue.
Before LLMs, task-oriented assistants were pipelines: intent and slot classification, a dialogue-state tracker, a policy that picked the next action, and template text. They were reliable inside their domain and useless outside it. An LLM assistant replaces most of that pipeline with one model, and keeps the engineering in what goes into the prompt and what the model may do.
The model itself is stateless. A chat API takes a list of messages with roles (system, user, assistant, tool) and returns one new message. The server flattens the list with the model's chat template, for example ChatML's <|im_start|>user … <|im_end|> markers, which the model saw during Instruction tuning. Everything the assistant "remembers" is whatever the application chooses to resend.
Anatomy of a turn
| Part | Contents | Design note |
|---|---|---|
| System prompt | role, policies, tone, output format, tool schemas | stable across turns; put it first so it can be cached (see System prompts and role prompting) |
| Memory | rolling summary, user profile, retrieved past turns | decides which old facts survive |
| Grounding | retrieved documents, account data | see RAG architecture; cite sources in the reply |
| Recent turns | last W user/assistant pairs, verbatim | keeps pronouns and follow-ups resolvable |
| Tool results | JSON returned by functions the model called | see LLM APIs and function calling |
The cost of remembering
If the application resends the full history, turn carries every earlier turn. With a system prompt of tokens and user and assistant tokens per turn, the prompt grows linearly and the total over a conversation grows quadratically:
- prompt (input) tokens sent at turn t
- system-prompt tokens, user tokens per turn, assistant tokens per turn
- number of turns in the conversation
- number of recent turns kept verbatim; the total now grows linearly in T
A window caps the cost but forgets everything older than turns. The usual fixes add a small memory that grows slowly. A rolling summary folds each turn that leaves the window into a short text, at the price of an extra LLM call and lossy compression. Retrieval memory embeds past turns and fetches the few most similar to the new message, at the price of misses when the wording differs. MemGPT (Packer et al., 2023) lets the model itself page facts in and out of an external store.
Long context does not remove the problem. Liu et al. (2023, "Lost in the middle") found that models use information at the start and end of a long input better than in the middle, so a fact from turn 3 buried in 30k tokens of history can be present yet ignored.
Reference points
- InstructGPT (2022)
- labelers preferred outputs of the 1.3B-parameter InstructGPT over those of the 175B GPT-3
- Llama 2-Chat (2023)
- Touvron et al. introduced Ghost Attention (GAtt), fine-tuning data that keeps a system instruction in force over many turns
- MultiWOZ (2018)
- about 10,000 human–human task dialogues across 7 domains (hotel, restaurant, train, …); the standard pre-LLM benchmark for dialogue-state tracking
- MT-Bench (2023)
- Zheng et al.: 80 two-turn questions in 8 categories, scored by an LLM judge; the second turn tests follow-ups (see LLM-as-a-judge)
- Chat APIs
- stateless message lists with
system/user/assistant/toolroles; the client resends history on every call
Memory strategies in a simulation
Worked example
| Turns T | Full history | Window W = 6 | Ratio |
|---|---|---|---|
| 40 | 26,400 + 260 × 780 = 229,200 | 26,400 + 260 × 219 = 83,340 | 2.75× |
| 100 | 66,000 + 260 × 4,950 = 1,353,000 | 66,000 + 260 × 579 = 216,540 | 6.25× |
Going from 40 to 100 turns multiplies the full-history cost by 5.9 but the windowed cost by only 2.6. At turn 100 a full-history prompt is tokens, which no longer fits an 8k or 16k context window. Prompt caching makes a repeated prefix cheaper but does not change which facts fit.
1def n_tokens(text): # swap in the model's tokenizer2 return max(1, round(len(text) / 4))34class Conversation:5 """Sliding window of recent turns plus a rolling summary of older ones."""67 def __init__(self, system, llm, window=6, budget=8000):8 self.system, self.llm = system, llm9 self.window, self.budget = window, budget10 self.turns, self.summary = [], "" # turns: list of (user, assistant)1112 def messages(self, user_msg):13 msgs = [{"role": "system", "content": self.system}]14 if self.summary:15 msgs.append({"role": "system", "content": "Earlier in this conversation: " + self.summary})16 recent = self.turns[-self.window:]17 while recent and sum(n_tokens(m["content"]) for m in msgs) + sum(18 n_tokens(u) + n_tokens(a) for u, a in recent) + n_tokens(user_msg) > self.budget:19 recent = recent[1:] # hard cap: drop oldest verbatim turns20 for u, a in recent:21 msgs += [{"role": "user", "content": u}, {"role": "assistant", "content": a}]22 return msgs + [{"role": "user", "content": user_msg}]2324 def send(self, user_msg):25 reply = self.llm(self.messages(user_msg))26 self.turns.append((user_msg, reply))27 if len(self.turns) > self.window: # fold the turn leaving the window28 u, a = self.turns[-self.window - 1]29 self.summary = self.llm([30 {"role": "system", "content": "Update the summary. Keep names, numbers, "31 "preferences and open tasks. At most 120 words."},32 {"role": "user", "content": f"Summary: {self.summary}\nUser: {u}\nAssistant: {a}"},33 ])34 return reply3536echo = lambda msgs: "ok: " + msgs[-1]["content"][:40] # offline stand-in for an API call37chat = Conversation("You are a travel assistant.", echo, window=2)38for m in ["I'm Priya.", "I'm vegetarian.", "Find flights from Lisbon.", "Book the 9:40 one."]:39 chat.send(m)40print([m["role"] for m in chat.messages("Thanks!")])41# ['system', 'system', 'user', 'assistant', 'user', 'assistant', 'user']Where it's used
Customer support
answer from help-centre documents, look up orders through tools, hand off to a human with the conversation summary
Internal assistants
search company knowledge with retrieval, respecting each user's document permissions
Task agents
multi-turn tool use for booking, scheduling or data entry; see The agent loop in the Agentic AI branch
Check yourself
A conversation with s = 600, u = 60, a = 200 runs for 40 turns. Roughly how many prompt tokens does full-history resending cost compared with keeping only the last 6 turns?
Check yourself
Users report that the assistant forgets their dietary restriction after a long chat, even though the model's context window is large enough for the whole history. What is the most likely cause and fix?
Remember
- ✓The model is stateless; the application rebuilds the prompt every turn from system prompt, memory, grounding, recent turns and tool results.
- ✓Full history costs tokens: quadratic in turns. A window of turns makes it linear.
- ✓Windows forget, summaries drift and retrieval misses; most assistants combine a window, a summary and structured facts.
- ✓A large context window does not guarantee use: facts in the middle of long prompts are used less reliably.
- ✓Put stable content first for prompt caching; authorise tool calls and check outputs in code, not in the prompt.