You've probably asked ChatGPT a question today. Maybe you asked it to compare two tools, draft an email, or explain something you didn't have time to read. What you probably haven't asked is how it got to be the thing giving you an answer in the first place. That's fair; most people don't need to know. But if you're a marketer trying to understand why an AI assistant recommends one brand over another, or why it seems confident about some things and vague about others, it helps to know how AI models are trained before you try to influence what one says about you.
The short version: it isn't one process, it's three, stacked on top of each other. Each stage does a different job, and each one leaves a mark on how the finished model behaves.
Stage one: pretraining, or where the model gets its knowledge
The first stage is the one people picture when they think of "training an AI": feeding it a vast amount of text and having it learn to predict the next word, over and over, until it's absorbed patterns in grammar, facts, reasoning and style. OpenAI's own account of how its foundation models (the ones that power ChatGPT) are built describes three broad sources feeding into this process: information that's publicly available on the internet, information accessed through partnerships with third parties, and information provided by users, human trainers and researchers (OpenAI Help Centre, "How ChatGPT and our foundation models are developed").
Pretraining is where a model becomes fluent and broadly knowledgeable. It's also where its limits get set: if something never appeared in that training data, or appeared rarely, the model has a shakier grip on it. This is worth sitting with for a moment, because it's the root cause of something every marketer using these tools has noticed: an AI assistant can talk confidently about a well-known competitor and a lot more hesitantly about a smaller brand it saw mentioned less often. Confidence in the output doesn't always track confidence in the underlying knowledge.
But a model that's only been through pretraining isn't yet the polite, instruction-following assistant you chat with. That takes two more stages.
Stage two: supervised fine-tuning, teaching it to follow instructions
A raw pretrained model is good at continuing text, not at being helpful. Ask it a question and it might continue with more questions, or wander off into something that merely sounds similar, because "predict the next plausible word" isn't the same skill as "answer what was actually asked."
The fix is supervised fine-tuning (SFT). OpenAI's research on this, published in its work on InstructGPT, describes the process as starting with human labellers writing out demonstrations of the response they'd want to a given prompt, and then fine-tuning the model on that set of labelled examples (OpenAI, "Aligning language models to follow instructions"). Instead of "how would this text most likely continue," the model is now being nudged toward "how would a helpful person answer this."
This is a smaller, much more curated dataset than pretraining used, and it's the first point where the model starts looking less like an autocomplete engine and more like an assistant.
Stage three: RLHF, teaching it what "good" looks like
Supervised fine-tuning teaches a model to follow instructions, but it doesn't teach much about tone, judgement or which of several reasonable answers people prefer. That's the job of the technique most people have heard the name of without knowing what it does: reinforcement learning from human feedback, or RLHF.
Per OpenAI's own description of this process, it runs in two further steps after supervised fine-tuning. First, the model produces several different answers to the same prompt, and human labellers rank them from best to worst. Those rankings are used to train a separate reward model, essentially a scorer that learns to predict which of two answers a person would prefer. Second, the original model is fine-tuned again, this time using reinforcement learning (specifically an algorithm called Proximal Policy Optimization, or PPO) to produce more answers that the reward model scores highly (OpenAI, "Aligning language models to follow instructions").
Here's a concrete way to picture it. Imagine a labeller asks the model "how do I get more people to notice my brand online" and gets back three answers: one three sentences long and vague, one a wall of unstructured jargon, one that's specific and structured with clear next steps. The labeller ranks the third one highest. Multiply that single judgement by many thousands of prompts and rankings, and the reward model starts to generalise a preference: specific, structured, plainly-written answers score better than vague or jargon-heavy ones. That preference then gets baked into the assistant model itself. It's a large part of why ChatGPT-style assistants tend to answer in a fairly consistent, structured, "helpful colleague" register regardless of what you ask; that register was reinforced, deliberately, at this stage.
Where this still goes wrong
None of this makes a model infallible, and it's worth being honest about the gap. A model shaped by RLHF can still hallucinate a fact confidently, because reward models score how an answer sounds, not whether every clause is true. It can also drift toward telling you what you probably want to hear rather than what's most accurate, a known failure mode called sycophancy. OpenAI's own postmortem on a 2025 GPT-4o update that turned noticeably more sycophantic traced the cause to exactly this stage: an added reward signal based on users' thumbs-up/thumbs-down feedback ended up weakening the primary reward signal that had been keeping sycophancy in check (OpenAI, "Sycophancy in GPT-4o: what happened and what we're doing about it"). "The labeller (or user) liked this answer" and "this answer was correct" aren't always the same signal, and when the reward model leans too hard on the first one, this is what happens. If you've ever had an AI assistant state something about your industry with total confidence and gotten it wrong, this is very likely why: fluent and well-reinforced is not the same thing as verified.
Why this matters if you're trying to get your brand mentioned
All three stages above shape what ends up in an AI assistant's answer when someone asks it a buying question in your category, which is the whole game if you're tracking AI search visibility. Pretraining is why a brand that's been written about clearly and often across the open web has a stronger baseline presence in the model's "knowledge" than one that hasn't. The RLHF stage is why the model rewards a specific, well-structured answer over a vague one, which is the same discipline we've written about before: models pull in retrieved passages that state one fact plainly rather than paragraphs that only make sense with surrounding context, and the same plain, structured writing that a human labeller preferred during RLHF is what a retrieval step favours when it's deciding which of your pages actually get cited.
Put differently: the assistant's preference for clear, structured, specific answers isn't a content-marketing trick you need to reverse-engineer, it's a property of how the model itself was shaped to behave. Write the way the reward model was trained to reward, and you're not fighting the system, you're working with its grain.
Below is what that looks like from the other side, on the dashboard Babel42 builds out of those AI Buyer journeys: how often a brand appears, how often it wins, and which sources the models quoted to get there.
This is what we built the AI Visibility product to show directly: not a theory about what a model prefers, but an actual weekly read on which of your competitors' facts are winning that preference right now, and which are yours. The free plan runs one AI Buyer across any two of the seven assistants weekly, no card required, enough to see whether your own plainest facts are the ones getting picked up.
How AI models are trained: the short version
An AI model like ChatGPT isn't built in one pass. Pretraining gives it broad knowledge and fluency from a wide sweep of text. Supervised fine-tuning teaches it to follow an instruction rather than just continue a sentence. RLHF, running a reward model trained on human rankings and then reinforcement learning against that reward, teaches it which of several reasonable answers people prefer, favouring clear, structured, specific responses over vague ones. None of that makes it infallible, confident and correct aren't the same thing, but understanding the process explains a lot about why these assistants answer the way they do, and why writing plainly and specifically is less a trick and more exactly what the model was shaped to reward. That shaped preference is also why position matters once a model is reading, not just training, on your content: see how AI chatbots summarise long content for what that means in practice.
