How Does AI Work? A Plain-English Guide to 20 Core Concepts

A large language model, commonly abbreviated as LLM, is a transformer-based model trained on very large volumes of text. Training data typically includes books, websites, source code, encyclopedias, forums, and other publicly available material, often totaling trillions of tokens.

The training objective is to predict the next token in a sequence. Although this task appears modest, repeating it across trillions of examples produces models that exhibit a wide range of capabilities, including grammar, reasoning, code generation, translation, and mathematical problem-solving. These capabilities emerge from scale rather than from explicit instruction.

The term "large" reflects parameter counts in the hundreds of billions and training costs measured in millions of dollars. ChatGPT, Claude, and Gemini are all examples of large language models.

Every AI model has a finite working memory known as the context window. The context window defines the maximum number of tokens the model can consider at one time, including the user prompt, the model response, and prior conversation history.

Context windows have grown substantially. Early versions of GPT supported approximately 4,000 tokens. GPT-4 supports 128,000. Claude 3.5 supports 200,000. Gemini 1.5 Pro supports up to 1 million tokens.

A larger context window does not guarantee perfect recall. Models tend to give more weight to information near the beginning and end of the context, while information placed in the middle is often underweighted. This phenomenon is known as the "lost in the middle" problem and explains why a model may appear to forget content that was clearly provided earlier in a long input.

When a language model generates text, it does not always select the single most probable next token. The degree of randomness in token selection is controlled by a parameter called temperature.

A temperature of zero produces deterministic, conservative output. A temperature near one produces more varied and creative output. Higher temperature values can produce incoherent results.

Lower temperature settings are appropriate for tasks that demand precision, such as code generation, factual summarization, and structured extraction. Higher temperature settings are appropriate for tasks that benefit from variety, such as brainstorming, creative writing, and ideation. Most consumer applications set this parameter automatically, but understanding it helps explain variability in model output.

Large language models are capable of producing confident, well-formed responses that are factually incorrect. This behavior is referred to as hallucination.

An LLM does not retrieve verified information. It generates the next token based on patterns observed during training. If a fabricated statement matches the statistical pattern of what might plausibly come next, the model may produce it. This is true regardless of whether the underlying fact is real.

Common hallucinations include nonexistent citations, fabricated software functions, and incorrect historical claims presented as fact. Mitigations include independent verification of outputs and the use of retrieval-augmented generation, which is covered later on this page. From a governance perspective, hallucination is one of the most significant risks associated with deploying LLMs into workflows that involve regulated information or material business decisions.

The structure and content of a prompt materially affect the quality of a model response. The same model can produce substantially different output depending on how a question is framed.

Effective prompts typically share several characteristics. They provide context about the user and the task. They assign a role to the model. They include examples of the desired output format. They specify constraints such as length, structure, or tone. They decompose complex requests into discrete steps.

Prompt engineering is best understood as a form of clear written communication rather than as a technical specialty. It is the primary mechanism by which users interact with and direct large language models. For a deeper guide, see our Prompt Engineering Guide.

Retrieval-Augmented Generation, abbreviated as RAG, is an architectural pattern that addresses the hallucination problem by grounding model responses in retrieved source material.

When a user submits a question, the system first searches a knowledge base for relevant documents. Those documents are then provided to the model as context, and the model generates a response that draws on the retrieved information rather than on memorized training data.

The closed-book versus open-book analogy is useful. Without RAG, the model answers from memory. With RAG, the model consults verified sources. RAG also allows organizations to update model behavior simply by updating the underlying documents, without retraining. Most production AI systems used in regulated industries rely on some form of retrieval to manage accuracy and traceability.

Retrieval-augmented systems require the ability to search large document collections by meaning rather than by exact keyword match. Vector databases provide this capability.

Each document is converted to an embedding and stored in the database. When a query arrives, it is also converted to an embedding, and the database returns documents whose embeddings are closest to the query embedding in vector space. This allows a query about "heart disease treatment" to return documents about "cardiac care protocols" even when there is no exact word overlap.

Common vector databases include Pinecone, Qdrant, Weaviate, and the pgvector extension for PostgreSQL. Vector search is what enables AI systems to operate over the meaning of content rather than its literal text.

An AI agent extends a language model with the ability to take actions and pursue goals. Whereas a conventional model responds to a single message, an agent operates in a continuous loop of thinking, acting, observing results, and adjusting its approach.

A coding agent assigned a defect might read the issue description, examine relevant source files, formulate a hypothesis, propose a code change, execute the test suite, evaluate the results, and iterate until the issue is resolved. The model functions as the reasoning component, while tools provide the means to interact with external systems.

Common tools available to agents include web search, code execution, file system access, API calls, scheduling systems, and databases. Agents transform AI from a question-answering interface into a workflow participant. From a security perspective, agents introduce significant new considerations related to authorization, auditability, and least privilege, which warrant deliberate governance before deployment.

Chain-of-thought prompting is a technique that improves the accuracy of model output on complex problems by directing the model to reason explicitly through intermediate steps before producing a final answer.

For mathematical, logical, or multi-step questions, prompting a model to provide its reasoning produces more reliable results than asking only for the final answer. Phrases such as "think step by step" or "reason through this carefully" are sufficient to elicit this behavior in most modern models.

The technique reflects a broader principle. Providing a model with structured space to work through a problem typically yields better results than demanding an immediate response.

Diffusion models are the architecture behind most modern AI image generation systems. The training process is counterintuitive. The model does not learn to draw. It learns to remove noise.

During training, the model is shown real images that have been progressively corrupted with random noise. It is taught to reverse this corruption step by step. During generation, the model begins with pure random noise and progressively refines it into an image, guided by a text prompt.

The name reflects the underlying mathematical concept, in which particles diffuse randomly through a medium. The model learns to reverse this diffusion. The same architecture also underpins video generation systems such as Sora and Runway, and is being applied to audio, three-dimensional content, and the design of novel molecules for pharmaceutical research.