RLHF Explained: Why AI Answers Instead of Asking
RLHF (Reinforcement Learning from Human Feedback) is the training step that turns a raw language model into a helpful assistant. It's also a big reason an AI will confidently answer an ambiguous data question instead of stopping to ask what you meant.
Quick answer: In RLHF, people compare pairs of model responses and pick the better one. The model is then trained to produce responses that would win those comparisons. A direct, complete answer usually wins over a clarifying question, so the model learns to answer. In data work, that means undefined terms like "active customer" get silently defined by the model instead of flagged.
How does RLHF work?
The common recipe, popularized by OpenAI's InstructGPT work in 2022, has three stages:
- Supervised fine-tuning. A pretrained model is trained on example conversations written by people, so it learns the assistant format.
- Preference data and a reward model. The model writes several responses to the same prompt. Human raters rank them. A separate reward model is trained to predict which response a rater would prefer.
- Reinforcement learning. The assistant is optimized to produce responses the reward model scores highly, with a penalty for drifting too far from its starting behavior.
Newer variants such as DPO (Direct Preference Optimization) skip the separate reward model and train directly on the preference pairs, and some labs use AI feedback instead of, or alongside, human raters. The core idea is the same: the model learns to produce what raters prefer.
Why does RLHF push models to answer instead of ask?
Because of what raters tend to reward. Shown a full answer next to "Can you clarify what you mean?", most people pick the answer, especially when the answer looks reasonable. Over millions of comparisons, the model learns that being direct, complete and frictionless wins.
Related research backs up the broader pattern. Anthropic's 2023 study on sycophancy found that human preference judgments, and the preference models trained on them, sometimes favor agreeable, convincing responses over correct ones. The same pressure that rewards agreeableness rewards answering over questioning.
None of this means models can't ask questions. Labs increasingly train for clarification on purpose. But by default, the incentive points toward giving you something rather than asking you something.
What does this look like in data analysis?
Ask an AI "how many active customers do we have?" against a database where "active" was never defined. A human analyst would probably ask: active in the last 30 days or 90? Do trial accounts count? Paying only? The model usually picks an interpretation, writes the SQL, and hands back a number with total confidence.
Ask again later and it may pick a different interpretation. That's how the same question returns different numbers. See Metric Consistency in AI Analytics for what that costs and how to decide how much of it to tolerate.
How do I get an AI to ask before it answers?
- Make it state assumptions. Instruct it to list every assumption (time window, filters, exclusions) before running a query. Hidden choices become reviewable ones.
- Give it permission to stop. Tell it explicitly that asking a clarifying question is the right move when a business term is undefined. Models follow this much better when it's said out loud.
- Remove the ambiguity at the source. A semantic layer with defined metrics means the model doesn't have to guess what "active" means.
- Test it. Keep a small set of deliberately ambiguous questions and check whether the model asks or guesses.
Common pitfalls
- Reading a confident tone as correctness. Fluency is what RLHF optimizes. Accuracy isn't guaranteed by it.
- Judging an AI analyst only on speed. The clarifying step that looks slow is often the step that prevents a wrong number.
- Assuming a newer model fixed this. Check its behavior on your own ambiguous questions.
Pro Tip: Add one line to any data agent's instructions: "If a business term in the question isn't defined in the semantic layer, ask what it means before writing SQL." It's the cheapest consistency fix there is.
← Back to AI & ML Tips