What Actually Happens When You Type "hi" Into an LLM
It feels instant, but a single "hi" triggers memory retrieval, reranking, and a second hidden process that decides what to remember. A walkthrough of the memory layers behind an AI agent.
You type "hi" into an LLM. It feels instant.
Here's what's actually happening under the hood.
1. Your message doesn't go in alone
It's bundled with the system prompt, your recent chat history, and a memory retrieval step. The system pulls your top-5 most relevant memories about you from storage: facts, preferences, past conversations.
2. Three kinds of memory get consulted
- Procedural: how the agent should behave (its instructions, skills)
- Semantic: durable facts about you (a structured DB plus a vector index, not just embeddings)
- Episodic: what happened before (timestamped events, ranked by relevance, recency and importance)
3. All of that gets reranked
Not everything gets dumped into context. A top-K search narrows thousands of candidate memories down to the handful that actually matter for this message.
4. Then, and only then, the LLM responds
5. After it replies, a second invisible process kicks in
A memory extraction agent looks at the conversation and asks: did anything durable just change?
It logs the episode, pulls out any new facts, and flags candidate preferences. Those go through a review gate before they're saved as "truth".
So "hi" isn't just a string
It is a message that triggers retrieval, reranking, generation, and memory extraction, all before you've finished reading the reply.