First, without referees (judges, umpires, arbiters, monitors), much of life would be a chaotic ordeal dominated by cheats, bullies, and dirty players. That’s why dictators refuse to allow election monitors, corrupt politicians defame the media who report their corruption, and bankers lobby incessantly against regulators.
Second, even with referees, action moves too fast for certainty. In baseball, even the most skilled umpires miss calls at first and enforce idiosyncratic strike zones.
Third, even-handed justice requires trust in the referee and a belief that missed calls tend to even out over time. If the referee is a cheat — as happens frequently when regulators get paid off by the regulated — trust in the game vanishes, and no one feels safe.