AI and the Confidence Gap

The more I use generative AI, the more I notice an uncomfortable paradox. When I ask a large language model about a subject I know well, I can usually see the weaknesses quite quickly: an assumption that is too broad, terminology that is almost but not quite right, a conclusion stronger than the evidence permits, or a recommendation that would struggle in the real world.

The answer may still look excellent. It is articulate, structured, confident and often comprehensive. But because I know the subject, I also know where to challenge it. When I ask the same system about a subject I know much less about, the answer can look equally impressive. The difference is that I am now much less able to tell which parts are correct, which are merely plausible, which are incomplete and which have been invented.

The asymmetry of expertise

This creates an important asymmetry. Expertise does not necessarily make an AI answer look better; it makes its weaknesses easier to see. A domain expert notices the missing constraint, the unusual edge case, the regulatory nuance, the impractical recommendation or the citation that does not actually support the claim. Someone with less domain knowledge may see none of these things. They see a polished answer that appears to have done the hard work for them.

This is why generative AI can amplify a confidence problem often associated with the Dunning–Kruger effect. I do not mean that AI creates the effect itself. The deeper issue is calibration: the people least equipped to validate an answer may also be the people least able to recognise how much validation it requires. Confidence can therefore rise faster than competence.

Plausible correctness may be more dangerous than hallucination

Much of the discussion about AI risk focuses on hallucination: the model states something that is simply false. Obvious hallucinations are a problem, but they are also comparatively easy to understand. The more difficult failure mode is what I think of as plausible correctness.

Plausible correctness is an answer that looks right, sounds expert and may contain a substantial amount of truth, but is wrong in one or two ways that materially change the conclusion. A laboratory protocol can be scientifically literate yet experimentally unusable. A regulatory analysis can cite the correct framework while misclassifying the product. A commercial recommendation can be supported by accurate market data while overlooking the constraint that determines whether it can actually be executed.

These are not absurd outputs. They are convincing outputs. That is precisely why they are dangerous.

Human in the loop is not enough

The usual response is to say that AI should have a human in the loop. For serious applications, that is not sufficient. You need the right human in the loop: someone who understands the subject well enough to challenge the model rather than simply approve what it has produced.

A non-expert reviewer may improve tone, formatting and apparent completeness while leaving the substantive error untouched. Domain expertise changes the nature of the review. It asks whether the evidence really supports the claim, whether the terminology is being used correctly, whether the recommendation is workable, whether the regulatory interpretation is defensible and what important question has not been asked at all.

Grounding helps, but it does not remove judgement

Grounding AI in authoritative evidence is therefore essential, particularly in scientific, medical, legal, financial and regulated environments. It reduces the freedom of the model to invent, and it makes provenance visible. But grounding does not eliminate the need for expertise. A model can retrieve the right source and still interpret it badly, apply it to the wrong context or draw a conclusion that the source does not justify.

A more credible operating model is not simply question, model, answer. It is model, evidence, explicit claim, expert review and accountable action. The point of the human is not ceremonial approval. It is judgement.

AI makes expertise more valuable, not less

There is an irony here. Generative AI is often described as democratising expertise. It certainly democratises access to expert-sounding output. Those are not the same thing. If anything, AI may increase the value of genuine domain expertise because the volume of material requiring intelligent validation is growing so quickly.

The people who know a field deeply are often the least dazzled by the answer because they can see how much work remains between something that reads well and something that is factual, actionable and right. That scepticism is not resistance to AI. It is what responsible use of AI looks like.

The skill that matters

Perhaps one of the most important skills of the AI era will not be prompt engineering or knowing how to make a model produce a more polished response. It will be knowing when the answer is good enough to use, when it requires evidence, when it requires an expert and when it should not be trusted at all.

AI can accelerate research, drafting, analysis and synthesis extraordinarily well. But fluency is not authority, and confidence is not validation. The greatest risk from generative AI may not be that it occasionally produces obvious fiction. It may be that it produces answers sufficiently convincing that the person least able to validate them is the person most likely to trust them.