Abt Thought Leadership Paper Series

When deciding how to allocate limited taxpayer or donor funds for social programs, policymakers and program managers increasingly ask for evidence of effectiveness based on studies that do not raise quibbles over methodology. They want to assess the extent to which their interventions have their intended effects based on research that all sides of the policy debate can agree provides credible scientific evidence. The basic claim for the “social experiment”—that the “coin flip” of randomization creates two statistically equivalent groups that cannot subsequently diverge except through the effects of a successful intervention—makes resulting estimates unbiased measures of the intervention’s impact. Despite the transparency and conceptual strength of the experimental strategy for revealing the causal connection between an intervention and the outcomes of its participants, experiments are often criticized on a variety of factors. We address 15 of these concerns in this paper and find each of them less objectionable than is widely believed. In our judgment an assortment of issues concerning ethics, scientific integrity, and practical feasibility need not stand in the way of expanded use of social experiments by governments and foundation funders seeking to convincingly and accurately measure the impacts of their social policies and programs.