How AI Chat Systems Work
How AI Chat Systems Work: Models, Memory, and Tools
What happens behind the chat box and how understanding it can improve the way we use AI.
Ask an AI assistant to rewrite a paragraph, and it produces text. Ask it to investigate a problem, inspect files, run tests, and prepare a report, and it may perform several operations before answering.
Both requests begin in the same chat box. What happens next depends on the model, the info it receives, and the capabilities of the application around it.
Consider a practical request:
Review this ServiceNow story and identify what needs clarification before development starts.
The answer will depend on more than the wording of that prompt. Does the assistant have the approved role matrix? Can it retrieve related requirements? Does it follow a consistent review procedure? Can it distinguish a documented rule from an assumption?
To understand the result, we need to look at the system behind it.
The application coordinates the model, context, and tools behind a single chat request.
What the model does
A language model is a trained neural network. Training adjusts its numerical parameters, or weights, so it can capture patterns in data. For the text-generating models used in many assistants, predicting the next token is a central training task. Further training improves capabilities such as following instructions.
Tokens are units of encoded text: words, parts of words, punctuation, or other fragments. They are not necessarily whole words. [1]
Given “Roses are red” a model might continue with “violets are blue.” It generates tokens based on its input and what it has already generated. Depending on the decoding method, it may choose the highest-probability next token or sample from possible alternatives. [2]
This mechanism supports much more than completing familiar phrases. Models can generate code, compare arguments, and work through problems. Some also process images or audio. But a fluent answer can still be wrong: generating a plausible explanation does not establish its accuracy.
For our ServiceNow story, the model can use the supplied text to identify vague acceptance criteria. It cannot reliably determine your organization's undocumented access rules.
Context, conversation history, and memory
During ordinary use, a model's weights generally remain fixed. Correcting an assistant does not normally retrain it on the spot. Yet its next answer can reflect your correction because the new information is available in its context.
Context is the information supplied for a model call. It can include instructions, conversation messages, documents, and tool results. Adapting to examples or information in that input is commonly called in-context learning.
Three mechanisms help explain what people mean when they say an AI “learns” or “remembers”:
| Mechanism | What changes? | Example |
|---|---|---|
| In-context adaptation | Information available during a model call | You provide a naming convention, and the model applies it. |
| Application memory | Stored information that can be supplied in later calls | The app saves a preference and uses it in another conversation. |
| Training or fine-tuning | The model’s weights | A training process updates the model using additional examples. |
Using new information, remembering it, and updating model weights are different mechanisms.
Conversation continuity also depends on context. An application may supply earlier messages, retrieve selected information, or summarize a long exchange. The model does not necessarily receive everything visible in your chat history.
The context window limits the tokens available during a model call, generally including input and generated output. Summaries and retrieval can support longer tasks, but they can omit details. Even information that fits within the window may not be used reliably. [3]
This explains a familiar experience: the assistant follows a decision for several exchanges, then appears to forget it. The relevant detail may have been omitted, compressed, or overlooked.
For important work, maintain a clear source of truth. In the ServiceNow review, that could be the current story, an approved role matrix, and a short decision log. These are easier to inspect and correct than an assumption that the assistant remembers every exchange.
If a conversation starts drifting, provide a brief handover: the current goal, authoritative version, active constraints, decisions, and unresolved questions. Continue in the existing chat or use the handover to start another.
Product note: memory and project boundaries depend on the application and its settings. A new chat does not necessarily remove account-level personalization. Temporary modes may have separate controls for personalization and retention. Check the current documentation for the product you use. [4–6]
How the application manages instructions
An AI application assembles the information the model receives and manages what happens around each call. In agent systems, the software coordinating model calls, tools, and execution is often called a harness or runtime.
It may retrieve documents, select conversation history, store progress, and enforce permissions. A single user request can therefore involve several model calls before an answer appears.
The application can also distinguish instruction sources. Platform rules, application instructions, project guidance, and user requests do not necessarily have equal priority.
For example, OpenAI describes an instruction hierarchy in which system instructions take priority over developer instructions, which take priority over user instructions. A later user message does not automatically override a higher-priority rule. This is intended behavior, rather than a guarantee that the model will always follow the correct instruction. [7]
As a user, place stable preferences in the appropriate settings, project requirements with the project, and the immediate task in your request.
For the story review, a useful instruction would be:
Identify missing acceptance criteria, access-control ambiguities, and gaps in testability. Separate documented requirements from assumptions. Present unresolved business decisions as questions.
This gives the assistant a concrete objective and a way to handle uncertainty. Supplying approved references then gives it evidence to work with.
Retrieval, tools, and reusable skills
Three additions can make our story reviewer more useful: relevant sources, access to operations, and a repeatable procedure.
Retrieval supplies relevant information. A system can search a document collection and include selected passages in the model's context. This pattern is commonly called retrieval-augmented generation, or RAG.
For our review, retrieval might supply an approved role matrix or a related process document. It can provide current or private info without retraining the model. But the system can still retrieve the wrong document or miss a decisive passage.
Ask for sources that directly support the findings. A citation helps only if it supports the associated claim and applies to the relevant version or configuration.
Tools let the system perform operations. These might include querying an API, searching records, executing code, or editing a file.
In a typical interaction, the model proposes a structured tool call. The runtime checks and executes it, then returns the result. Generating a command does not itself give the model access to a computer.
A connector provides integration with a service and may expose several operations. The Model Context Protocol, or MCP, standardizes communication between AI applications and servers exposing capabilities. Tools can also be integrated through ordinary APIs or other interfaces. [8]
An authorized ServiceNow connector could let our reviewer retrieve related records. Its permissions determine which records it can access and whether it can change them.
Skills package reusable procedures or workflows. Under the Agent Skills specification, a skill contains a SKILL.md instruction file and may include scripts, references, or assets. [9]
A story-review skill could define the inputs, review steps, output format, and handling of missing information. It makes the procedure reusable; its usefulness still depends on the quality of those instructions and how reliably the system follows them.
Together, these additions give the assistant evidence to consult, operations to perform, and a procedure to follow.
Workflows and agents
A workflow follows predefined rules or paths, which can include branches and loops. An agent gives the model more freedom to choose its next steps and tools. This is a useful architectural distinction, although terminology varies across products. [10]
Our story-review workflow might:
- Read the story and approved references.
- Identify the five most consequential clarification questions.
- Produce a compact test matrix covering roles, states, and negative cases.
- Suggest suitable Automated Test Framework (ATF) candidates and their prerequisites.
- Present unresolved decisions for human review.
An agent could decide that a finding needs more evidence, retrieve a related story, and revise its assessment before returning the result.
That flexibility can help with less predictable tasks. It also creates more opportunities to take an unhelpful path. Define a stopping condition, time or cost limits, and what counts as a successful result.
Scheduling is a separate concern: it determines when work begins. A scheduled review can run a fixed workflow or an agent. Recurring work exists only if a scheduling service actually registers and runs the task.
What improves the ServiceNow review?
The same request can produce very different results depending on the surrounding system:
| Available capability | What it adds |
|---|---|
| Story text | A review of the requirements as written. |
| Approved references | A basis for checking consistency with documented rules. |
| Retrieval | A way to find relevant supporting material. |
| Review skill | A reusable procedure and output format. |
| Authorized tools | Access to related records and other permitted operations. |
| Verification | Evidence that the findings are supported and useful. |
These capabilities need to earn their place. A short, self-contained story may need only a clear prompt. A complex review spanning several sources may benefit from retrieval and tools.
Evaluate the output against real work. Did the assistant catch an ambiguity that would have caused rework? Did it invent a requirement? Could a developer act on the questions? Were its ATF suggestions feasible in the relevant instance?
A polished response is a starting point. Useful, supported findings are the measure of success.
Permissions and verification
As assistants gain access to more information and operations, access controls become more consequential.
Retrieved documents and tool results may contain malicious instructions. A page might tell the assistant to ignore its task or disclose private information. This is a form of prompt injection: an attempt to make untrusted content act like an authoritative instruction.
Permissions must therefore be enforced by the application and connected services. Restrict access, validate operations, and provide appropriate review for consequential actions. The MCP tool specification explicitly discusses human oversight and the ability to deny operations. [11]
In ServiceNow, enforce access through the integration account, API permissions, and platform controls. An instruction to avoid production changes cannot replace restrictions on the ability to make them.
Verification should also match the task. A story-review finding needs supporting requirements. A file edit needs inspection. A claimed system change needs evidence that the change succeeded.
Where to look when the answer disappoints
A poor result can have several causes: an ambiguous request, missing context, an unsuitable source, a failed tool call, or an incorrect inference.
Before rewriting the prompt, identify which part failed. The improvement might be a clearer instruction, an approved reference, a better retrieval step, a simpler procedure, or a check on the result.
The model generates outputs. The application determines what information and capabilities it receives. Understanding both helps you build more useful AI systems and judge their work more carefully.
Source:
Sources and further reading
- Hugging Face: How do Transformers work?
- Hugging Face: Generation strategies
- Anthropic: Effective context engineering for AI agents
- OpenAI: Memory in ChatGPT
- OpenAI: Projects in ChatGPT
- OpenAI: Temporary chat in ChatGPT
- OpenAI: Improving instruction hierarchy in frontier LLMs
- MCP: Architecture overview
- Agent Skills: Specification
- Anthropic: Building effective agents
- MCP: Tools and security considerations
Comments
Post a Comment