Scientific Rigor

New Hampshire, where seat belts are not mandatory, has 9 road deaths per billion miles driven. South Carolina, where they are mandatory, has 21.*

Do we conclude seat belts don’t work?

No, that would be idiotic. Anyone who knows basic physics understands it’s harmful for the body to hit the car’s cockpit, and that a restraining cord around the torso can prevent that. Clearly, something else must explain the difference. Could it be that New Hampshire drivers use seat belts at a similar rate to South Carolina’s? (Yes: 78% vs 93%.) Could it be that New Hampshire roads and vehicles are safer, or that their drivers are slower and more careful? (Probably.)

Basic scientific rigor requires examining all relevant variables. Similarly, when human behavior is a key factor, it requires that we do not assume a 100% compliance rate in the study group and 0% in the control group.

All of this should be obvious.

And yet.

When it comes to Sweden, masking, and lockdowns, many conclude these interventions did not work because countries that mandated them had results not too different from Sweden’s, which did not.

But rigor demands two questions. First, what was the actual usage rate of these interventions in Sweden compared to others? If it was similar, then what had limited effect was mandating the measure, not the measure itself. (Even then, mandates differ: how a country announces and enforces them matters. We should not conclude mandates do not work, but that mandates implemented as they were did not.)

Second, were environmental or demographic factors different in Sweden? Did it have fewer initial cases, better healthcare, a healthier population, or social habits that slowed spread? These and other factors matter.

The point is, there is nothing scientific in comparing large datasets over a single variable without controlling for others. Or in reducing non-binary variables to binary ones. Likewise, there is nothing scientific in analyzing policies without considering compliance rates. And there is nothing scientific in running behavioral experiments as if the skills and influence of the person giving instructions did not affect participants’ behavior.

Perhaps above all, there is nothing scientific in accepting results that contradict the basic laws of physics. (For lockdowns not to work, either germ theory would have to be wrong or lockdowns would have to fail to reduce social contact. Are these reasonable claims? A similar logic applies to masks.)

Performing complex statistical analyses, drawing professional-looking charts, and packaging them into a LaTeX-style document does not compensate for a lack of basic rigor.

(*Source: death data from NHTSA; mileage data from the U.S. Bureau of Transportation Statistics, both for 2021.)