How large language models actually work

Here is the whole trick. A large language model reads the text so far, then guesses what comes next. It adds its guess to the text and does the same thing again. And again, hundreds of times a second, until the answer is finished. That is the entire machine.

This sounds far too simple to produce anything useful, so it is worth sitting with why it is not. To guess the last word of "the capital of France is ___", a model has to have stored that fact somewhere. To guess the next line of a mathematical proof, it has to have picked up how proofs are built. Guessing well, across more or less everything humans have ever written down, turns out to require something that behaves a great deal like knowing things.

Hold onto one sentence and the rest of this lesson has somewhere to attach: a model is a fluent guesser, and a fluent guesser needs checking.

A model never sees the letters you typed. Before anything else happens, your text is chopped into chunks called tokens. A token is usually a common word, a piece of a longer word, or a punctuation mark. Each model has a fixed vocabulary of these chunks, somewhere between fifty thousand and two hundred thousand of them.

The chopping is done by an algorithm called Byte-Pair Encoding, usually shortened to BPE, and the idea behind it is straightforward. Start with single characters, then repeatedly glue together whichever pair turns up together most often. Do that enough times and common words like "the" end up as a single chunk, while rare words get broken into several pieces. As a rough guide, one token is about four characters, or about three quarters of an English word.

Three things follow from this

• 01You pay by the token, and speed depends on tokens too. Every request is billed as tokens in plus tokens out. A fifty-page PDF pasted into a prompt might be thirty thousand tokens, and that number is both your bill and your waiting time. Shortening what you send is the most direct lever you have on either.
• 02The chunking favours English. The vocabulary was learned mostly from English text, so the same sentence in Arabic or Hindi can take two to four times as many tokens. It costs more, runs slower, and leaves less room for everything else. This is a concrete reason a system that tests beautifully in English can quietly perform worse elsewhere, and why multilingual work needs testing in the actual language rather than a translation of the English tests.
• 03Models cannot reliably see individual letters. The classic example is asking how many times the letter r appears in "strawberry" and getting a confident wrong answer. The model sees chunks, not letters. So when someone proposes using a model to check whether a reference number is formatted correctly, that is almost always a job for ordinary code instead.

Once your text is in chunks, the model converts each chunk into a long list of numbers. That list is called a vector, or an embedding. It might be 1,536 numbers long, or 4,096.

Think of it as a coordinate. A map locates a place with two numbers, latitude and longitude. An embedding locates a piece of meaning with a few thousand numbers. The property that makes this useful is that things meaning similar things land near each other.

The classic demonstration is arithmetic on meaning. Take the numbers for "king", subtract the numbers for "man", add the numbers for "woman", and you land near "queen". Directions in that space turn out to correspond to ideas like royalty, gender, and plurality. Nobody programmed that in. It falls out of the training.

Two kinds are worth telling apart. Token embeddings live inside the model and represent each chunk as it moves through the layers. Document embeddings come from a separate, smaller model that turns a whole passage into one vector. Document embeddings are what make search-by-meaning work, and they return in lesson two.

One thing I have run into repeatedly in real projects: general-purpose embeddings are poor at specialist vocabulary. Terms from oncology, or genomics, or one particular piece of legislation all collapse into roughly the same spot, because the model was never taught to separate them. Embeddings trained on that specific domain put them where they belong. Which is why "we will just use an embedding model" is not yet a plan.

Every modern language model is built on a design called the transformer, introduced in a 2017 paper called "Attention Is All You Need". You do not need the maths. You need the mechanism, and why it was a breakthrough.

Here is the problem it solved. Older models read text strictly left to right, one word at a time, and had effectively forgotten the beginning by the time they reached the end of a long passage. They also could not be spread across many processors efficiently. Transformers fixed both at once.

Now attention, in one example. Read this sentence: "The council rejected the proposal because it was too expensive." You knew instantly that "it" means the proposal, not the council. You did not apply a rule. You weighed the words against each other and the answer was obvious. Attention is the mechanism that lets a model do the same thing. For every chunk of text, it looks at all the other chunks and decides how much each one matters right now. Self-attention just means the text is doing this to itself.

Then it gets layered up. The model runs many of these attention operations side by side, called heads, and each one drifts towards tracking a different kind of relationship: one follows grammar, another follows what a pronoun points at, another follows the topic. Picture several people marking up the same page, each with a different coloured highlighter. Stack dozens of those layers on top of one another and the early ones pick up surface patterns like grammar, while the deeper ones pick up meaning and reasoning.

At the end of each step, the model produces a score for every chunk in its vocabulary. Those raw scores are called logits. A function called softmax turns them into proper probabilities that add up to one hundred percent. Then the system picks one chunk, adds it to the text, and starts again.

The settings controlling that pick are genuine product decisions, not obscure engineering knobs.

• Temperature controls how strongly the model favours its top guess. Near zero it almost always takes the most likely chunk, which gives you consistent, factual, slightly repetitive answers. Higher, towards 1 and beyond, and the less likely options get a real chance, which gives you variety and creativity and more risk. Low for anything factual. Higher for brainstorming.
• Top-p, also called nucleus sampling, keeps only the most likely options until their probabilities add up to a set figure such as 0.9, then picks from those. It trims off the long tail of strange choices.
• Top-k does the same job by keeping a fixed number of options instead of a probability budget.
• Greedy decoding always takes the single highest-scoring option. Maximally consistent, and it can get stuck repeating itself.

The context window is the maximum number of tokens a model can consider in one go. Your instructions, the conversation so far, any documents you have supplied, and the answer currently being written all count against the same limit. Older models had four to eight thousand tokens. Current ones run from around a hundred and twenty-eight thousand to over a million.

The picture that works is a desk. Everything the model is using for this one request has to fit on the desk at the same time. Fill the desk and things get pushed off the edge, or the request fails outright.

Three things people get wrong about it

• 01A bigger window is not free. Cost and waiting time both rise with how much you put in. A two-hundred-thousand-token prompt is slow and expensive on every single request, not just the first one.
• 02Models get lost in the middle. They pay most attention to the beginning and the end of a long context and can miss something buried in the middle. This is the practical argument for fetching the right few pages rather than tipping in the whole manual and hoping.
• 03Context is not memory. Once a conversation scrolls past the window it is gone, unless somebody has built a system to store it. When an assistant seems to remember you from last week, that is engineering, not something the model does by itself.

This is the part most people are hazy on, and a little precision here goes a long way in a meeting. Modern models are built in three stages, and they are not equally expensive or equally famous.

A few more terms come up often enough that knowing them signals you are current.

• Reasoning models are trained to think at length before answering, working the problem through in a hidden internal draft. They are far better at multi-step problems like maths, code, and planning. They are also slower and dearer, because you pay for all that hidden thinking. Use them where a wrong answer is costly and the problem genuinely has steps. Do not use them to sort emails into two categories.
• Parameters are the numbers a model learned during training, quoted as 8 billion, 70 billion, and so on. More parameters generally means more capable, and also slower, dearer, and hungrier for hardware. Small models in the one to eight billion range run on hardware you own, and they are very good indeed when trained for one narrow job.
• Distillation means training a small model to copy a big one. You keep much of the quality at a fraction of the cost and waiting time. It is the standard way to ship something affordable once an expensive model has proved the idea works.
• Quantization means storing the model's numbers less precisely, at 8 bits or 4 bits rather than 16, so it runs faster on cheaper hardware. Quality drops slightly. It is the usual way to fit a capable model into a constrained environment.

Models you can download, and models you can only rent

Closed models such as GPT, Claude, and Gemini are reached through an API. They tend to be the strongest available, your data leaves your building to reach them, and you cannot run them yourself. Open-weight models such as Llama, Mistral, Qwen, and DeepSeek can be downloaded and run inside your own cloud account, your own data centre, or a machine with no internet connection at all. When data legally cannot leave a country or an organisation, that single distinction decides the entire architecture. It comes back in the safety lesson.

Inference means running a trained model to get an answer, as opposed to training it in the first place. It is worth understanding because cost and speed questions come up in every project, and unlike a lot of AI questions they have concrete answers.

A request happens in two phases. Prefill is the model reading your prompt. It happens in parallel, it is fast per token, it scales with how long your prompt is, and it decides how long you stare at a blank screen before the first word appears. Decode is the model writing the answer, one token at a time, in sequence, and it decides how quickly the words arrive after that. Total waiting time is roughly the reading phase, plus the number of tokens in the answer multiplied by the time each one takes.

Once you can see that shape, the levers are obvious. Send shorter prompts. Fetch fewer and better documents. Show the words to the user as they are produced. Cap how long the answer can run. Send easy requests to a smaller model. Cache anything you are paying to compute twice.

That is the full path: text becomes chunks, chunks become numbers, numbers move through layers of attention, out comes a score for every possible next chunk, one gets picked, repeat. Cost and speed fall out of how many tokens are involved and which phase they are in. That single mental model clears most of the fog around this technology. The next lesson is how you teach one of these things your own knowledge.

Let us walk the whole thing through with a concrete example, because the pieces stick better connected than separate. Imagine an assistant that helps a planning officer work out whether a proposed house extension breaks a boundary distance or fire access rule.

• 01Documents become chunks. The application PDF, the labels on the drawings, the zoning rules, and the fire safety clauses all get split into tokens.
• 02Chunks become numbers. The system turns the officer's question, the facts pulled off the drawing, and the relevant regulations into embeddings. "Setback", "boundary distance", and "rear clearance" land near each other even though the applicant and the regulation use different words for the same thing.
• 03The right material gets fetched. The system pulls in only the clauses, measurements, and source pages that matter. It does not paste the entire planning code into the prompt, because a giant context is slower, dearer, and quite capable of burying the one decisive clause in the middle where the model reads least carefully.
• 04Attention connects the facts. Inside the transformer, the chunk for "extension" can weigh heavily towards "rear boundary", "3m setback", and the fire access clause that was fetched. This is how the model links the officer's ordinary wording to the formal rule.
• 05The answer gets built one chunk at a time. The model scores the possible next chunks, turns those scores into probabilities, and picks. For a government workflow you would run a low temperature and force a fixed output format: issue found, the clause it comes from, how confident the system is, and what a human should check.
• 06Training explains what it can do, inference explains what it costs. Pre-training gave it general language ability. Instruction tuning and preference training made it behave like an assistant. At the moment of use, the reading phase takes in the prompt and the writing phase produces the answer, and both cost money and time.