The Threshold Problem

AI safety frameworks increasingly talk about thresholds. If a model reaches capability X or risk level Y, additional safeguards are required or development should stop. Thresholds are attractive because they make messy questions look technical. Put enough numbers around uncertainty and eventually it starts to resemble an engineering problem with a correct answer.

But the most important part of a risk threshold is not mathematical. Somebody has to decide how much risk is acceptable. A one percent chance of something going catastrophically wrong may sound tiny until you are one of the people expected to live with the consequences. And the person making that calculation may have billions invested in continuing. Risk is never just a probability. It also has a distribution: who gets the benefit if we proceed, who carries the damage if we are wrong, and who gets to decide that the trade-off was worth it?

Numbers can help us understand risk, but they cannot make the political decision disappear. Somewhere underneath every supposedly objective threshold sits a judgment about what we are willing to endanger, for whose benefit, and on whose behalf. Risk thresholds look mathematical. Deciding what risk other people must live with is political.