Field guide · Layer 01

LLM fundamentals, for people building on top of one

Not a machine-learning course — a working model of what's actually happening inside the box, deep enough to explain why the rest of the AI Factory Stack (RAG, vector DBs, agents, guardrails, evals) exists at all.

00 — What is an LLM

A very good next-token predictor

Strip away the branding and a large language model is a function trained on an enormous amount of text and code to do one thing: given the tokens so far, predict which token comes next. Everything you experience as "understanding," "reasoning," or "conversation" is that single mechanism, applied one token at a time, at a scale large enough for genuinely useful patterns to emerge.

“Autocomplete on steroids” undersells it — the patterns an LLM learns encode grammar, facts, style, code structure, and a rough model of how arguments and instructions tend to resolve — but it doesn't mis-describe the mechanism. Nothing in that loop involves the model checking a fact against the world, running a calculation, or knowing whether what it just said is true. It's producing the statistically most plausible continuation given everything it learned during training.

That single sentence explains most of what's counterintuitive about working with LLMs: why they're extraordinary at fluent, structured language and pattern completion, and why they need the rest of this stack — retrieval, tools, guardrails, evals — to become a reliable product rather than a very convincing autocomplete.

01 — Tokens, not words

The model never sees a word

Before any prediction happens, text is broken into tokens — subword chunks somewhere between a character and a whole word. “Evaluation” might be one token or three, depending on the model's tokenizer; common words are usually a single token, rare or made-up ones get split.

This isn't a minor implementation detail — it's the unit everything else in this guide is measured in. Context window limits, API pricing, and latency are all denominated in tokens, not characters or words, and the exchange rate isn't fixed: the same passage can tokenize to a meaningfully different count depending on which model's tokenizer is doing the splitting, and non-English text often tokenizes less efficiently than English.

Rule of thumb

For English text, one token ≈ 4 characters, or roughly ¾ of a word. It's a rough guide for budgeting, not a guarantee — always check actual token counts for anything cost- or limit-sensitive.

02 — Embeddings & attention

How the model tracks what matters

Each token gets converted into a high-dimensional vector — an embedding — that captures learned meaning, plus a positional signal so the model can tell “dog bites man” from “man bites dog.” From there, the transformer architecture's core mechanism, self-attention, takes over.

At every layer, every token compares itself against every other token in the sequence, deciding how much to “attend to” each one when updating its own representation. Multiple attention heads run this comparison in parallel, each specializing in different kinds of relationships — one head might track grammatical agreement, another might link a pronoun back to the noun it refers to several sentences earlier. Stack enough of these layers and the model builds a representation rich enough to continue the sequence plausibly, including following instructions, maintaining a persona, or writing syntactically valid code.

This is also the mechanism responsible for the cost of long inputs: comparing every token to every other token means the compute grows quadratically with sequence length, which is the direct cause of the context-window trade-offs in the next section.

03 — Context window

Working memory, not long-term memory

The context window is the maximum number of tokens a model can process in a single call — your system prompt, conversation history, any retrieved documents, and the space reserved for its reply, all sharing one budget.

What eats the budgetWhy it matters
System prompt & instructionsFixed overhead on every single call
Conversation historyGrows every turn unless actively trimmed or summarized
Retrieved documents (RAG)Often the largest and most variable chunk
Reserved output spaceBudget the model needs left over to actually answer

Two things surprise people moving from “what's the spec sheet number” to “what actually works.” First, quality doesn't hold steady as you approach the limit — models reliably show a “lost in the middle” effect, where information buried in the center of a long context is used less reliably than information near the start or end, and accuracy on real tasks often degrades well before the nominal limit is reached. Second, a bigger context window isn't free: attention cost grows quadratically with length, so doubling the input more than doubles the compute, which shows up directly as higher latency and cost.

This is precisely the gap Layer 02 (RAG) and Layer 03 (Vector DB) exist to close: instead of stuffing more into the window and hoping, you retrieve only the pieces that are actually relevant to the current query and feed those in — better accuracy, and a cost that doesn't scale with how much data you have sitting behind the system.

04 — How training happens

Three stages, three different jobs

A model you can actually talk to is the product of a pipeline, not a single training run — and each stage exists to fix something the previous one leaves incomplete.

Stage 1
Pretraining
Next-token prediction over a huge, largely unlabeled corpus of text and code. Builds broad world/language knowledge cheaply, at massive scale.
Stage 2
Supervised fine-tuning
Trained on curated instruction → response pairs, so the model shifts from “continue this text” to “follow this instruction.”
Stage 3
Alignment
Human (or AI-generated) preference data pushes outputs toward helpful, safe, on-policy behavior — classically RLHF, increasingly its cheaper alternatives.

Alignment specifically is where the field has moved fastest. Classic RLHF trains a separate reward model from ranked human responses, then uses reinforcement learning to optimize the base model against it — effective, but expensive and finicky to get right. Newer approaches like DPO fold preference optimization into a more direct supervised step without a separate reward model, Constitutional AI uses a written set of principles plus model self-critique instead of large volumes of human labeling, and RLAIF replaces human raters with AI-generated preference judgments. All are trying to hit the same target more cheaply: a model that's not just capable, but reliably does what you actually wanted.

05 — Generation is sampling

The model doesn't choose the right answer — it samples one

At inference time, the model produces a probability distribution over every possible next token, then a sampling step picks one. Temperature controls how sharply that distribution is followed: near zero, the model almost always takes the highest-probability token (deterministic, conservative); higher values flatten the distribution, so lower-probability — more varied, more surprising — tokens get picked more often.

This single fact is worth sitting with, because it's the root cause of a lot of what feels like unpredictable LLM behavior: there is no step anywhere in this process where the model checks whether a token is true. It's checking whether the token is plausible, given everything it learned. Most of the time those two things line up closely enough that the distinction doesn't matter. When they don't — sparse training data on a topic, an ambiguous or leading question, a request that pushes past what the model actually knows — the model still samples a plausible-sounding continuation. That's a hallucination: not a bug in the sampling process, but the sampling process working exactly as designed, applied to a case where “plausible” and “true” have quietly come apart.

06 — Structural limits

What's actually wrong with LLMs — and who fixes it

Every limitation below follows directly from the mechanism described so far. None of them are solved by a cleverer prompt, because the prompt doesn't change how the model was trained or how it generates — which is exactly why the rest of the stack exists.

LimitationWhy it happensWhat compensates
Knowledge cutoffOnly knows what was in its training data, frozen at a point in timeRAG — retrieves current, external information at query time
HallucinationSampling a plausible token isn't the same as sampling a true one (§05)RAG for grounding, Guardrails and Evals to catch what still slips through
No persistent memoryOnly "remembers" what fits in the current context windowVector DB-backed memory, application-level state
Can't take action on its ownIt predicts text — it has no built-in way to call an API or run codeAI Agents give it a loop; MCP gives it a standard way to reach tools
No guaranteed safe/valid outputNothing in training guarantees a given output is policy-compliant or well-formedGuardrails — input/output filtering and structured validation
No way to know if it's workingFluent output looks the same whether it's right or wrongEvals — the only layer that actually measures the rest
07 — Where it sits in the stack

The LLM is the engine, not the car

Every other layer in the AI Factory Stack exists because the LLM, on its own, is a stateless, actionless, occasionally-confident text predictor with a hard knowledge cutoff. None of that is a design flaw to be fixed — it's the trade-off that makes the core model general-purpose and reusable. The surrounding layers are what turn that general-purpose engine into a specific, reliable product.

RAG + Vector DB

Compensate for the knowledge cutoff and context-window limits (§03, §06) by retrieving exactly the relevant, current information at query time instead of hoping it's already in the weights. See the RAG guide and Vector DB guide.

AI Agent + MCP

Compensate for “can't take action” by putting the LLM in a loop with tools, and standardizing how it reaches those tools and data sources. See the AI Agent guide and MCP guide.

Guardrails

Compensate for “no guaranteed safe or valid output” by checking inputs and outputs against explicit rules the model itself has no built-in way to enforce. See the Guardrails guide.

Evals

Compensate for “no way to know if it's working” — the measurement layer that turns every claim above into something you can actually verify. See the Evals guide.

All seven layers are now published — back to the hub for the full series.

08 — Choosing a model

What actually differs between models

Once you're past “an LLM predicts tokens,” most practical decisions come down to a handful of trade-offs, not a single “best” model.

  • Capability vs. cost vs. latency — larger, more capable models generally cost more per token and respond more slowly; plenty of tasks (classification, extraction, simple rewrites) are handled perfectly well by smaller, cheaper, faster models.
  • Context window size — matters most for workloads with large documents or long conversations, but a bigger number on a spec sheet doesn't guarantee reliable use of that whole window (§03).
  • Open-weight vs. closed / hosted — self-hosting an open-weight model buys control over data residency, fine-tuning, and cost at scale, at the price of owning the infrastructure; hosted APIs trade that control for simplicity and someone else's uptime.
  • Prompting is the primary interface, with real but bounded power — clear instructions, examples, and structure measurably improve output quality, because they shift the distribution the model is sampling from (§05) toward what you want. What a prompt can't do is give the model current information it was never trained on, a persistent memory it doesn't have, or a guarantee that it won't hallucinate — those are jobs for the other layers, not for a better-worded instruction.
09 — Getting started

Before you build on an LLM

A short checklist for framing any new LLM-backed feature realistically, before writing the prompt.

  • Write down what the task actually needs: fresh/private knowledge (→ RAG), multi-step action (→ Agent + MCP), or just fluent language generation from what the model already knows.
  • Estimate your realistic token budget per request — prompt, history, retrieved context, and reply — and check it against the model's context window with headroom, not right up to the limit.
  • Assume hallucination is possible on every call that touches facts, and decide up front how you'll catch it (grounding, guardrails, human review) rather than after a user finds one.
  • Never treat the model's internal knowledge as current or private-data-aware — anything time-sensitive or proprietary belongs behind retrieval, not baked into the prompt from memory.
  • Pick temperature deliberately: low for anything that should be consistent and checkable, higher only where variety is actually the goal.
  • Before shipping, read the Evals guide — you'll want a way to know whether a prompt or model change made things better before your users tell you it didn't.
An LLM is a very capable, very confident guesser. Everything else in this stack exists to give that guess somewhere solid to stand on.
10 — Sources

What this guide draws on

Read in full and used to shape the structure and claims of this guide.

SourcePublisherUsed for
How LLMs actually work: transformers, tokens, and attention explained Knovo Tokenization, embeddings, attention mechanism, generation/sampling (§01–02, §05)
LLM Training: RLHF and Its Alternatives Sebastian Raschka, Ahead of AI The pretraining / SFT / alignment pipeline, DPO, Constitutional AI, RLAIF (§04)
LLM context windows: what they are & how they work Redis Context window mechanics, quadratic attention cost, lost-in-the-middle effect, RAG as the practical answer (§03)

Compiled 2026. Model-specific figures (context sizes, pricing, benchmark scores) move fast — treat this guide's mechanisms and trade-offs as durable, and check current provider docs for today's numbers.