What a Model Actually Does

The word "model" does a lot of quiet work, and it is worth unpacking, because most arguments about technology are really arguments about what a model is.

A model is a compressed description of a pattern. You give it a large number of examples, it finds regularities in them, and what it keeps is not the examples but a summary compact enough to be useful and general enough to apply to something it has not seen.

That last property is the whole point, and it is also the whole danger. A system that could only repeat its examples would be a filing cabinet. A system that generalises can answer new questions — and can generalise from a pattern that was true of the examples but is not true of the world.

This is why the question "is it accurate?" is usually the wrong one. Accurate on what? A model is accurate on the distribution it learned from, and the interesting failures all happen at the edges of that distribution: the unusual case, the new situation, the population the examples underrepresented.

So the honest way to describe any of these systems is conditional. It is reliable here, under these conditions, for this kind of question, and its confidence tells you how familiar the input looked, not how right the answer is. Those two things are easy to confuse, and almost every serious mistake made with a model comes from confusing them.