Conversations around data science typically contain a lot of buzzwords and broad generalizations that make it difficult to understand its pertinence to governance and policy. Even when well-articulated, the private sector applications of data science can sound quite alien to public servants. This is understandable, as the problems that Netflix and Google strive to solve are very different than those government agencies, think tanks, and nonprofit service providers are focused on. This does not mean, however, that there is no public sector value in the modern field of data science. With qualifications, data science offers a powerful framework to expand our evidence-based understanding of policy choices, as well as directly improve service delivery.
To better understand its importance to public policy, it’s useful to distinguish between two broad (though highly interdependent) trends that define data science. The first is a gradual expansion of the types of data and statistical methods that can be used to glean insights into policy studies, such as predictive analytics, clustering, big data methods, and the analysis of networks, text, and images. The second trend is the emergence of a set of tools and the formalization of standards in the data analysis process. These tools include open-source programming languages, data visualization, cloud computing, reproducible research, as well as data collection and storage infrastructure.
Perhaps not coincidentally, these two trends align reasonably well with the commonly cited data science Venn diagram. In this diagram, data science is defined as the overlap of computer science (the new tools), statistics (the new data and methods), and critically, the pertinent domain knowledge (in our case, economics and public policy). While it is a simplification, it is still a useful and meaningful starting point. Moving beyond this high-level understanding, the goal of this paper is to explain in depth the first trend, illuminating why an expanded view of data and statistics has meaningful repercussions for both policy analysts and consumers of that analysis.
Using data to learn about public policy is not at all new. The origins of the social sciences using statistical analysis of observational data goes back at least to the 1950s, and experiments started even further back. Microsimulation models, less common but outsized in their influence, emerged as the third pillar of data-driven policy analysis in the 1970s. Beyond descriptive statistics, this trifecta—experiments, observational statistical analysis, and microsimulation—dominated the quantitative analysis of policy for around 40 years. To this day, they constitute the overwhelming majority of empirical knowledge about policy efficacy. While recent years have seen a substantial expansion in the set of pertinent methods (more on that below), it is still critical to have a strong grasp of experiments, observational causal inference, and microsimulation.
Since public policy can’t be conducted in a laboratory, experiments are rare in policy studies. Experiments require random assignment, which for policy means a benefit or program is made available randomly to some people and not to others—hardly a politically popular strategy. Many would also say it is ethically questionable to do this, though randomized experiments have taken firm root in medicine, sacrificing fairness in the short term for progress in the long term. Regardless of the political and ethical barriers, they do happen. Experiments are most often supported by nonprofits or created by an accident of governance, and can produce relatively rigorous results, compared to the other methods discussed here.
Perhaps the most famous experiment in modern public policy is that of the Oregon Medicaid expansion. When Oregon moved to expand access to Medicaid in 2008 (before the Affordable Care Act), the state quickly realized that it could not afford to cover all the individuals eligible under the loosened criteria. Opting to randomly select which residents would be able to receive benefits, Oregon officials created the perfect circumstances for researchers to compare recipients of Medicaid with non-recipients who were otherwise very similar. Professors Katherine Baicker and Amy Finkelstein led the research efforts, resulting in extensive evidence that Medicaid improved some health outcomes and prevented catastrophic medical expenses, while also increasing health-care utilization and costs. Signaling a growing interest in this approach, the recent Nobel Prize in Economics recognized three scholars who have taken experiments (sometimes call randomized control trials, or RCTs) into the developing world to examine how to best tackle global poverty.
Due to the financial and political difficulties that experiments present, they remain rare, and much more research is based on the statistical analysis of observational data. Observational data refers to information collected without the presence of an explicit experiment—it comes from surveys, government administrative data, nonprofit service delivery, and other sources. Usually by obtaining and combining several datasets, researchers look for various opportunities to examine the causal effects of policy changes with statistical methods. These statistical methods, broadly called causal inference statistics (or quasi-experiments), take advantage of differences within populations, or policy changes over time and geography to estimate how effective a service or intervention is.
Individually, the strength of the evidence from a single study is limited. (This is true in any field, and it suggests prudence when changing your beliefs based on results from one study.) However, since observational data is far easier to gather and analyze than experimental data, it is possible to find many opportunities to re-examine the same policy questions. Eventually, it’s possible to examine many papers on the same subject, called a meta-analysis. Meta-analysis of observational studies have convincingly argued that increased school spending improves student outcomes, gun access leads to higher risk of suicide and homicide, and that taxes on sugary beverages are associated with lower demand for those beverages.
Although at times difficult to interpret, this slow accumulation of many observational analyses by different research groups often becomes the most informative and trustworthy source of information about potential policy changes.
Although microsimulation is a lesser-known type of modeling, it remains a critical one. The news is frequently covered in estimates from microsimulation methods, such as how effective taxes would change under the Tax Cuts and Jobs Act and how many people would lose health insurance under the curtailing of the Affordable Care Act. Even a substantial part of the (in)famous Congressional Budget Office scoring of the cost of federal legislation employs microsimulation.
The Urban Institute-Brookings Institution Tax Policy Center model is perhaps the easiest to understand intuitively. The model starts with a sample of anonymized administrative data from the Internal Revenue Service, which contains lots of information about taxpayers that is specific to each person. (This puts the “micro” in microsimulation.) The model itself then does the same thing as online tax preparation software: It runs through the rules of the tax code and calculates how much this person should be paying in taxes. However, the model contains many different knobs that can be turned and switches that can be flicked, each one changing something about the way the tax code works. By altering some of these inputs, the model creates a simulation, that is, an alternative possible outcome from the real world of tax policy.
These models are highly complex, and usually take years to build. They also require a lot of information about how a set of public policies are currently affecting a population, so the data typically comes from government administration records. However, once they are built, they offer a quick and flexible lens into potential policy changes. In reality, the behavioral consequences—how people and firms react to new policy—are large enough that few experts are ever really convinced that estimates from these models are precisely correct. That said, microsimulation methods can ground policy discussions to reasonable predictions, make assumptions explicit, and give a reasonable sense of what complex and interacting policy changes might do. Compared to letting pundits invent numbers out of thin air, microsimulation offer a dramatically more rigorous approach to estimating policy outcomes.
