Teaching a model your stuff: prompting, RAG, and fine-tuning

Learn this sentence now and spend the rest of the lesson understanding why it is true. RAG changes what the model knows. Fine-tuning changes how the model behaves. Prompting is where you always start.

Nearly every architecture question in applied AI comes down to choosing between those three, and usually to combining them. The rest of this lesson is the reasoning behind that sentence, ending with a decision tree you can reuse on any problem somebody brings you.

Prompting is the cheapest and fastest lever you have. You put your instructions and examples directly into the request. No training, no infrastructure, and you can rewrite it and try again in ten seconds. Because the model adapts purely from what you put in front of it, this is called in-context learning. Nothing is being learned permanently. The model is adjusting to what you gave it, for this one request, and then forgetting it entirely.

Four techniques carry most of the value.

• Separate the system prompt from the user's question. The system prompt holds the durable rules, the role, and the boundaries: "You are a maths tutor. Never give the final answer. Give one hint at a time." The user prompt is whatever they typed. Keeping them apart means your rules stay stable no matter how the conversation wanders.
• Show examples rather than describing what you want. Asking with no examples is called zero-shot. Including a handful of worked input-and-ideal-output pairs is called few-shot, and it is the fastest way to pin down a format or a style without retraining anything. If you are struggling to describe the output you want, stop describing and show three of them.
• Ask it to work step by step. This is chain-of-thought prompting, and it noticeably improves accuracy on anything with multiple stages, because it gives the model room to work rather than committing to a first guess. Reasoning models do this internally without being asked.
• Force the answer into a fixed shape. If the output feeds into other software rather than a human, demand JSON or a fixed set of fields, using the provider's structured output or function-calling features. This is what turns a chatty model into a component you can build on.

RAG stands for Retrieval-Augmented Generation, which is a heavy name for a simple idea. Instead of hoping the model memorised your facts, you look up the relevant passages in your own documents at the moment the question is asked, and hand them over with the question. The model then answers from your material and points at where each claim came from. This is the default design for almost every serious organisational AI application, and once you see why, you will spot it everywhere.

• It stays current. Change a regulation and you update one document. Nothing gets retrained.
• It can be checked. Answers point back to the passage they came from, so a human can verify them and an auditor can follow them.
• It respects permissions. You can enforce who is allowed to see which documents at the moment of lookup, which you simply cannot do with facts baked into a model.
• It is cheaper than retraining for knowledge, and it cuts down invented answers substantially, because the model has the actual text in front of it rather than a hazy memory of something like it.

How it works, in three stages

RAG has a known set of failure modes, and being able to run through them is what separates somebody who has built one of these from somebody who has read about them. When an answer comes back wrong, the useful question is never "is the model bad", it is "which link in this chain broke". Here is the table I keep in my head.

Fine-tuning means continuing to train a model on examples you provide, so that a new behaviour, style, or skill is built into the model itself rather than supplied fresh in every request. Reach for it when you want to change how the model acts. Almost never reach for it to teach the model new facts.

Good reasons to fine-tune

• A format, style, or tone you cannot get reliably from prompting. Always producing one exact extraction format, for instance, or always writing in a specific professional register.
• A domain language or dialect the base model handles badly. A regional Arabic dialect rather than Modern Standard Arabic is the example I have seen make the biggest difference in practice.
• A narrow, high-volume task where a small fine-tuned model matches a big general one at a fraction of the cost and waiting time. This is what makes running on your own hardware realistic.
• Cost and speed. Baking the behaviour in lets you drop a very long prompt from every request, or drop down to a smaller model entirely.

Modern fine-tuning is usually parameter-efficient, which means you do not touch most of the model. Rather than adjusting billions of numbers, you freeze nearly all of them and train a small set of new ones. The common method is LoRA, short for Low-Rank Adaptation, which slots small adapter layers into the model and trains only those. The intuition is that you are not repainting the building, you are hanging a few things on the walls. It is roughly ninety-nine percent fewer numbers to train, dramatically cheaper, and it means you can keep a library of task-specific adapters and swap them in and out over the same base model.

You will be asked this question in a hundred different disguises, so it pays to have one answer you can walk through out loud.

• 01Does this need AI at all? If the task is fixed, structured, and there is exactly one correct answer, like validating a reference number or calculating a fee, use ordinary software or a rules engine. It is cheaper, faster, fully auditable, and cannot invent anything. Being willing to say "this should not use AI" is a senior signal, not a lack of ambition.
• 02Can better instructions do it? Try that first, properly. It is the cheapest thing to try and the fastest to change your mind about.
• 03Does it need knowledge the model does not have? Add retrieval. Proprietary documents, facts that change, a requirement to cite sources, per-user permissions. This is the default answer for most real knowledge work.
• 04Does it need a behaviour, style, or dialect that prompting will not hold, or does it need to run cheaply on a small model? Fine-tune, usually with LoRA, and usually on top of retrieval rather than instead of it.
• 05Does it need to take actions or work through several steps on its own? Now you need an agent with tools, which is lesson three.
• 06Throughout all of it: decide how you will measure success before you choose the architecture, and pick the smallest thing that clears the bar. Add complexity only where your own measurements prove it is needed.

A policy team asks whether Article 14 of a new digital services law conflicts with existing data protection rules, or with how other countries have handled the same thing. Here is how the levers combine.

• 01Start with the instructions. The system prompt sets the role and the boundaries: act as a legal research assistant, cite every claim, keep source-backed findings separate from your own suggestions, and never make the final policy call.
• 02Use examples to fix the format. Show three examples of the output you want: the issue, the source clause, the comparable jurisdiction, the risk, suggested wording, and a confidence level. The model picks up the format for this request without anything being retrained.
• 03Build the document library. Parse the draft law, the official gazettes, the policy memos, and the comparable laws from elsewhere. Split them by article and clause rather than by arbitrary length, so a legal provision is never cut in half.
• 04Index and search properly. Each clause is converted into numbers and stored with its metadata: jurisdiction, date, version, source, and permission level. Questions are searched against it using both meaning and exact keywords, so "Article 14" matches as a literal string and "cross-border transfer" matches by meaning.
• 05Answer from the sources. The final request carries the system rules, the retrieved clauses, and the question. Attention links the phrase "cross-border transfer" to the data export rule that was fetched. The answer cites the exact clause, and declines when the evidence is thin.
• 06Fine-tune for behaviour only. If the team needs a consistent legal register or their own risk taxonomy applied every time, that is a fine-tuning job. The text of the law is not. Laws change, and changing things belong in retrieval.
• 07Add an agent only when the work involves steps. A tightly constrained agent can search the legislation, pull comparisons from other jurisdictions, draft a memo, and route it to a lawyer. Ordinary software still handles the exact dates, article numbers, permissions, and version control, because those need to be right rather than probable.