How Systems Like ChatGPT Manage Long Conversations Efficiently | Engineering Blog

1 // The Core Problem: Stateless Models vs Stateful Conversations

At a fundamental hardware and math level, Large Language Models (LLMs) are completely stateless functions. Given an input sequence of tokens, the transformer architecture computes attention scores across that sequence and predicts the probability distribution for the next token. When you ask a follow-up question in a chat session, the model itself has zero biological or server-side memory of your previous turn.

To create the seamless experience of ongoing dialogue in applications like ChatGPT, systems must resend previous conversation history back to the model on every single turn.

The Quadratic Cost Bottleneck

Standard self-attention mechanisms scale quadratically O(N²) in compute and memory with respect to prompt sequence length N. As chat sessions extend from 5 turns to 50 turns, reprocessing entire uncompressed transcripts causes exponential spikes in latency (Time to First Token) and VRAM consumption.

graph LR T1[Turn 1: Prompt 1] --> M1[Model Inference] T2[Turn 2: Prompt 1 + Ans 1 + Prompt 2] --> M2[Model Inference] TN[Turn N: Prompt 1 + Ans 1 + ... + Prompt N] --> MN[Model Inference] M1 --> M2 M2 -.-> MN style T1 fill:#0f172a,stroke:#3b82f6,color:#f8fafc style T2 fill:#0f172a,stroke:#3b82f6,color:#f8fafc style TN fill:#0f172a,stroke:#3b82f6,color:#f8fafc style M1 fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style M2 fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style MN fill:#1e1b4b,stroke:#6366f1,color:#f8fafc

2 // The 300-Page PDF Mental Model (RAG vs Pure Memory)

To understand how production systems manage massive histories without blowing up GPU memory, consider a helpful engineering mental model: Asking a question about a 300-page technical manual.

There are two ways an AI application can answer a query about page 240:

Approach A: Naive Full Ingestion

Stuffing all 300 pages (~150,000 tokens) into a 1M token context window on every prompt. High cost, 10x slower inference, massive compute waste.

Approach B: RAG & Chunk Retrieval

Chunking the PDF into small 500-token vectors. Retrieving only the 3 most relevant pages (~1,500 tokens) and injecting them into the prompt. Fast, cheap, exact.

graph TD PDF[300-Page PDF] --> Chunk[Chunk & Embed] Chunk --> DB[(Vector Database Store)] Query[User Query] --> Search[Semantic Search] DB --> Search Search -->|Top 3 Matches ~1k Tokens| Inject[Final Prompt Injection] Query --> Inject Inject --> LLM[Target LLM] style PDF fill:#0f172a,stroke:#3b82f6,color:#f8fafc style Chunk fill:#0f172a,stroke:#3b82f6,color:#f8fafc style DB fill:#064e3b,stroke:#10b981,color:#f8fafc style Query fill:#0f172a,stroke:#3b82f6,color:#f8fafc style Search fill:#0f172a,stroke:#3b82f6,color:#f8fafc style Inject fill:#0f172a,stroke:#3b82f6,color:#f8fafc style LLM fill:#1e1b4b,stroke:#6366f1,color:#f8fafc

3 // The Multi-Tiered Context Management Pipeline

Production systems like ChatGPT use a hybrid, multi-tier strategy that combines the immediacy of short-term memory with the efficiency of vector retrieval and semantic summarization.

Tier 1

Sliding Window Buffer (Immediate Attention)

Keeps the last N turns (e.g., last 10 messages) raw and uncompressed in the prompt. This ensures 100% exact fidelity for immediate back-and-forth context, pronouns, and quick follow-ups.

Tier 2

Hierarchical Summarization (Background Rollup)

As older turns fall out of the sliding window, background worker processes condense them into structured bullet points or executive summaries. A 5,000-token exchange is compressed into a 200-token summary block.

graph TD Raw[Raw Dialogue 100 Turns / 30,000 Tokens] --> W1[Block Summarizer Worker] Raw --> W2[Block Summarizer Worker] Raw --> W3[Block Summarizer Worker] Raw --> Active[Active Window Raw Turns 76-100] W1 --> SA[Summary Chunk A Turns 1-25] W2 --> SB[Summary Chunk B Turns 26-50] W3 --> SC[Summary Chunk C Turns 51-75] SA --> Rollup[Master Rollup 500 Tokens] SB --> Rollup SC --> Rollup Rollup --> Prompt[Final Injected Prompt ~2,500 Tokens] Active --> Prompt style Raw fill:#0f172a,stroke:#3b82f6,color:#f8fafc style W1 fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style W2 fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style W3 fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style Active fill:#064e3b,stroke:#10b981,color:#f8fafc style SA fill:#0f172a,stroke:#3b82f6,color:#f8fafc style SB fill:#0f172a,stroke:#3b82f6,color:#f8fafc style SC fill:#0f172a,stroke:#3b82f6,color:#f8fafc style Rollup fill:#064e3b,stroke:#10b981,color:#f8fafc style Prompt fill:#064e3b,stroke:#10b981,color:#f8fafc

Tier 3

Persistent Memory & Vector Profile Retrieval

Explicit user preferences (e.g., "Always write code in TypeScript", "I live in San Francisco") are extracted asynchronously by a secondary LLM worker, embedded, and saved in a vector store. When relevant, these facts are fetched and dynamically injected into the system prompt.

graph TD subgraph User Session Window[Sliding Active Window<br>Recent Turns N-5 to N] Persist[Persistent Memory Engine<br>Extracted User Profile] end Window --> Builder[Dynamic Prompt Builder] Persist --> Builder style Window fill:#0f172a,stroke:#3b82f6,color:#f8fafc style Persist fill:#064e3b,stroke:#10b981,color:#f8fafc style Builder fill:#1e1b4b,stroke:#6366f1,color:#f8fafc

4 // System Optimization: KV Caching & vLLM PagedAttention

Summarizing and retrieving context reduces token counts, but at the GPU infrastructure level, processing long prompts still incurs heavy memory overhead due to key-value (KV) activations.

When a model generates tokens sequentially, calculating key and value vectors for past tokens on every step is redundant. Modern serving frameworks (vLLM, TensorRT-LLM) use KV Caching and PagedAttention:

How PagedAttention Prevents VRAM Waste

Non-Contiguous Allocation: Similar to OS virtual memory paging, PagedAttention stores KV caches in non-contiguous physical memory blocks, reducing VRAM fragmentation from 60% down to under 4%.
Prefix Caching: Common system prompts and initial conversation turns are shared across parallel user sessions in GPU memory, avoiding duplicated matrix calculations.

5 // Full End-to-End System Flow

Bringing all components together results in a robust, low-latency production architecture for long-running conversational AI:

graph TD UserQuery[User Input Prompt] --> CacheCheck{KV Cache Hit?} CacheCheck -- Yes --> InjectCache[Reuse Cached Prompt Keys & Values] CacheCheck -- No --> WindowPlanner[Context Window Planner] WindowPlanner --> VectorStore[(Vector Store Long-Term Profile)] WindowPlanner --> SummaryDB[(Background Summary Rollup Store)] WindowPlanner --> SlidingBuf[Sliding Window Raw Dialogue History] VectorStore --> PromptAssembler[System Prompt & Memory Assembler] SummaryDB --> PromptAssembler SlidingBuf --> PromptAssembler InjectCache --> PromptAssembler PromptAssembler --> LLM[LLM Inference Core Engine] LLM --> UserResponse[Stream User Response] UserResponse --> AsyncLoop[Async Background Worker Loop] AsyncLoop --> MemoryExtraction[Extract Entity Facts to Vector DB] AsyncLoop --> RollupWorker[Update Hierarchical Summary Rollup] style UserQuery fill:#0f172a,stroke:#3b82f6,color:#f8fafc style CacheCheck fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style InjectCache fill:#064e3b,stroke:#10b981,color:#f8fafc style WindowPlanner fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style VectorStore fill:#0f172a,stroke:#3b82f6,color:#f8fafc style SummaryDB fill:#0f172a,stroke:#3b82f6,color:#f8fafc style SlidingBuf fill:#0f172a,stroke:#3b82f6,color:#f8fafc style PromptAssembler fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style LLM fill:#1e1b4b,stroke:#6366f1,color:#f8fafc style UserResponse fill:#064e3b,stroke:#10b981,color:#f8fafc style AsyncLoop fill:#0f172a,stroke:#a855f7,color:#f8fafc style MemoryExtraction fill:#0f172a,stroke:#a855f7,color:#f8fafc style RollupWorker fill:#0f172a,stroke:#a855f7,color:#f8fafc

6 // Key Engineering Takeaways

Context window management is an active systems engineering challenge, not just a model capacity limit.

Combining stochastic retrieval with deterministic cache policies is essential for high-fidelity agent sessions.

KV-caching requires rigorous garbage collection to prevent GPU VRAM fragmentation during parallel user sessions.