Richard Wolcott Carson Architect Of Statistical Rigor In Modern Policy Analysis

Statistical Rigor Isn't Something You Learn From a Textbook. It's Something You Bleed Over.

I remember sitting in a conference room at the RAND Corporation back in the late '90s, watching a junior analyst try to defend a cost-benefit model for a public housing initiative. The numbers looked fine on the surface. The regression held. The confidence intervals were tight. But when someone asked why the control group had been stratified by zip code instead of household income level, the room went quiet. That was the moment I understood what Richard Wolcott Carson Architect Of Statistical Rigor In Modern Policy Analysis really meant in practice. It wasn't about fancy software. It was about catching the structural lie before it became policy.

The core idea Carson brought to the field — and I say brought because he didn't publish a single paper on it — is that rigorous policy analysis requires an architect to be uncomfortable with their own assumptions. Most statisticians want to prove something. Carson wanted to disprove himself. This distinction sounds minor until you are the one explaining to a senator why her flagship healthcare provision would actually worsen outcomes for the exact demographic it was designed to help. Here is how you actually do it. Not the textbook version. The version that survives a GAO audit. Start with the counterfactual. Before you write a single line of code, before you touch any dataset, write down what the world would look like if the policy did not exist. This sounds obvious. It is not. Most analysts skip this because the counterfactual is usually ugly and embarrassingly uncertain. Write it anyway. If you cannot articulate a credible baseline, your entire analysis is fiction wrapped in R code.

Then identify your identification strategy. What causal mechanism actually links the intervention to the outcome? Is it a difference-in-differences setup where you compare pre- and post-intervention across treated and untreated groups? Is it a regression discontinuity design where policy eligibility cuts off at a specific threshold? Is it an instrumental variable approach where something exogenous affects treatment but only affects the outcome through that treatment? Pick one. Justify it. Then pick another and show why it fails on the same data. I spent three weeks once trying to use IV estimation on a workforce training program evaluation. The instrument — distance to the nearest training center — was elegantly exogenous. It violated every relevance test. The first-stage F-statistic was 3.4. A textbook would have accepted it. The truth is a weak instrument produces biased estimates that point in the same direction as omitted variable bias but with false precision. I threw out the entire approach and went back to matching. Took twice as long. Produced a result the clients actually trusted.

Why Most Policy Analysis Fails Its Own Standards

The problem is not incompetence. The problem is incentives. Policy analysis is commissioned by people who already believe the policy works. When you hand them a null result, they call it a flawed analysis. When you hand them a statistically significant positive result, they call it rigorous. This pressure exists at every level, from agency staff to congressional research offices. Carson understood this dynamic intimately. He once told me during a break at a PBNA meeting that the most dangerous word in policy analysis is "significant." Not "not significant." Significant. Because a significant result with a weak identification strategy is still a significant result, and it will get printed in a white paper regardless. What kills policy analysis is not a flat finding. It is a confident finding built on shaky ground. One practical technique I learned from Carson that has saved me more times than I can count: always report the balance table. Before you run your main specification, show that your treated and control groups are comparable on every observable characteristic. If the standardized mean differences are above 0.25 for any variable, stop. You do not have a policy evaluation. You have a comparison of two different populations with a treatment correlated with everything that matters.

Another technique: run your sensitivity analysis before you announce results. Rosenbaum bounds for unmatched observational studies. Oster's delta for omitted variable bias. The placebo test where you apply your methodology to an outcome that the policy cannot possibly affect. If your main result disappears under reasonable sensitivity parameters, nobody needs to see the original number. Put it in an appendix labeled "sensitivity check." It tells the truth about your method.

The Carson Standard in Action: A Worked Example

Let me walk through a real case. Not a hypothetical. Something I worked on, something Carson reviewed before we published. We were evaluating a state-level expansion of Medicaid under the ACA. The policy assigned eligibility based on a federal poverty level cutoff, which gave us a natural regression discontinuity design. The running variable was normalized household income divided by the FPL threshold. The treatment was binary. The outcome was medical expenditure in the following calendar year. The first thing I checked was the bandwidth. Imbens and Kallals recommendation is around 1.5 times the optimal bandwidth calculated by Cattaneo, Ididia, and Titiunik. But I also tested the local linear specification versus local quadratic, because the density of observations near the cutoff was asymmetric. The quadratic fit pulled the treatment effect down by 12 percent compared to the linear specification. That 12 percent gap is your honest uncertainty range.

The second thing I checked was the covariate balance at the cutoff. Age, race, employment status, prior health conditions. Everything passed except prior emergency room utilization, which jumped by 0.8 standard deviations at the eligibility threshold. This was a behavioral response, not a confounder. People who discovered they were now insured began using emergency rooms more intensively. I controlled for it as a leading indicator. Carson insisted on keeping it in the main specification rather than dropping it to make the balance table look prettier. He was right. Dropping it would have introduced post-treatment bias. The final specification produced an average treatment effect of negative four hundred dollars per enrollee in annual medical expenditure, which sounds counterintuitive until you remember that the control group was spending money on uncovered services out of pocket while the treatment group shifted to preventive care covered by Medicaid. The net effect on out-of-pocket spending was positive nine hundred dollars. The policy succeeded by the metric it was designed to improve. I put the full Do-files in a GitHub repository. The sensitivity analysis comes first in the read me. The main result is in Appendix B. This ordering is non-negotiable in rigorous work. Reviewers should encounter your limitations before your headline number.

Software, Tools, and the False Comfort of Automation

There is a growing ecosystem of packages that claim to automate causal inference. Causalml from Uber. DoubleML from the Vienna team. Even R has propensityscorematch and fixest doing heavy lifting now. These tools are useful. They are not substitutes for thinking. When I first used automated matching in a large-scale evaluation for DOL, the software produced 47 covariate matches with near-perfect balance. I checked the weight distribution and realized the algorithm was matching a 62-year-old man with a GED to a 23-year-old woman with a associates degree because they shared the same employment trajectory in the previous quarter. The algorithm optimized for balance. It ignored common sense. I manually excluded six variables and re-ran. The balance deteriorated slightly. The matches made sense. Automation is dangerous when it produces results faster than you can verify them. Use the tools. Trust the output only after you have personally traced one complete calculation from raw data to final coefficient by hand. When you know where the body is buried, you can audit the graveyard.

Richard Wolcott Carson Architect Of Statistical Rigor In Modern Policy Analysis

The phrase carries weight in certain circles and means nothing in others. Carson himself would likely find both reactions annoying. He never sought recognition. He sought correct answers. The distinction matters because anyone calling themselves a policy analyst should prioritize the latter over the former. If you want to read more about the methods that shaped Carson approach, start with Angrist and Pischts Ke Mostly Harmless Econometrics. Then read Imbens and Rubin Causal Inference. Neither book mentions Carson. Both books describe the principles he lived by. The gap between the literature and the practice is where the real education happens.

Common Failures I See Repeatedly

Let me be blunt about what goes wrong. I have reviewed dozens of policy evaluations from state agencies, federal offices, and private consultants. The failures cluster around six patterns. First, p-hacking through specification searching. Running ten different models, reporting only the one that hits significance. This is not subtle. It is the default behavior of most analysts under deadline pressure. The fix is pre-registration. Write your specification before you look at the data. If you change it, document why and show the sensitivity of results to those changes. Second, ignoring clustering. Standard errors computed under the assumption of independent observations are nearly always too small when your data has any hierarchical structure. School districts. States. Counties. Hospitals. Cluster at the level where treatment varies. If you cannot identify the appropriate cluster, report unclustered and clustered SEs side by side. Let the reader decide.

Third, using OLS for binary outcomes without acknowledging the probability problem. Linear probability models produce predicted values outside the 0 to 1 range. They are fine for quick interpretation. They are terrible for anything requiring accurate marginal effects at the extremes. Use GLM with binomial family and logit link. The coefficients are less intuitive. The predictions are honest. Fourth, treating missing data as a technical problem rather than a substantive one. Missingness is never random in policy datasets. It is informative. If a survey respondent skipped the income question, it is probably because their income fell outside the expected range. Run little test. Compare respondents with complete data to those with missing data on every observable variable. If the groups differ significantly, your complete-case analysis is biased. Use multiple imputation. Report the comparison tables. Acknowledge the assumption. Fifth, confusing correlation with policy relevance. A statistically significant coefficient tells you nothing about whether the policy is worth implementing. You need the effect size relative to the cost. A 0.5 percentage point increase in employment among a targeted population may be significant at the 1 percent level. It may also be utterly insufficient to justify a billion-dollar program. Report both numbers. Let policymakers weigh them.

Sixth, and this is the one Carson cared about most, failing to communicate uncertainty. Point estimates are lies told with authority. Confidence intervals are the truth, usually boring and wide. Never present a finding without its uncertainty range. Never present uncertainty without explaining what it means in plain language. Your audience is not your peers. They are people who will make decisions based on your numbers.

What Carson Actually Taught Me

I worked with Carson for approximately fourteen months in the early 2000s before he moved to a private consultancy in Arlington. He never gave me a formal syllabus. He gave me bad analyses and asked me to find the errors. One exercise he gave me stands out. He handed me a published study claiming that a job training program increased earnings by 15 percent. The methodology was difference-in-differences with state fixed effects. The data came from administrative records. Everything looked technically sound. Carson asked me one question: what happened to the participants who dropped out of the program? The original authors had analyzed only completers. Dropouts were excluded from both the treatment and control groups, effectively creating a self-selected treated sample. The 15 percent estimate was almost certainly an upper bound inflated by positive selection. When I re-ran the analysis including all assigned participants using intent-to-treat estimation, the effect dropped to 3.2 percent and lost statistical significance at conventional levels. The program probably worked. The claim was wildly overstated.

This lesson repeated itself throughout our time together. Every published result is a maximum plausible claim until someone applies the most conservative credible specification. That someone should be you before anyone else sees the work.

A Practical Checklist for Rigorous Policy Analysis

Use this before you submit anything. It is shorter than you think. It catches more errors than you expect. Is your identification strategy stated in one sentence before the methods section? If not, rewrite it. Have you tested at least one alternative specification that produces a meaningfully different result? If not, you have not found the bounds of your uncertainty.

Is your balance table reported for every covariate you include? If not, your comparability assumption is unsupported. Have you checked for manipulation of the running variable in any regression discontinuity design? If not, the cutoff may be endogenous. Are your standard errors clustered at the appropriate level? If you are unsure which level is appropriate, cluster at the highest level and note the conservative bias this introduces.

Have you run a placebo test? If not, your result may reflect noise rather than signal. Does your sensitivity analysis change the qualitative conclusion? If yes, the qualitative conclusion is not supported by your data. Is your code publicly available? If not, your analysis cannot be replicated, and replication is the only peer review that matters in empirical work.

That checklist is not exhaustive. It is the minimum. Anything below it is guesswork dressed as analysis.

The Hard Truth About Policy Analysis

Most policy decisions are made for reasons that have nothing to do with statistical evidence. Funding flows to programs that look good in press releases. Agencies chase metrics that are easy to report rather than metrics that matter. Politicians prefer simple stories to complex causal chains. None of this will change. The role of the rigorous analyst is not to control the process. It is to ensure the evidence survives scrutiny. When you produce work that can withstand a hostile review, you give decision-makers something rare and valuable: confidence that their choice is not entirely baseless. That is a modest goal. It is also the only honest one. Carson retired from active consulting around 2018. He publishes nothing. He consults privately for state agencies that specifically request his reputation for being difficult to fool. The people who work with him know that every claim will be challenged, every assumption will be forced into the open, and every result will come with its full uncertainty range attached like a warning label.

That is the standard. It is not comfortable. It is not fast. It produces fewer headline numbers and more careful caveats. But the policies built on careful caveats tend to outlast the policies built on headline numbers.