A Working Tour Of The Files And What They Actually Contain
The main portal sits behind a simple search interface. You land on a dashboard that lists facilities by state, then you drill down into daily logs. Each facility has a different format for how they release their data. Some post clean CSV files. Others are PDF scans that you have to copy out by hand. That difference alone determines whether a project finishes in a week or drags out for months. I spent six months tracking solitary population numbers across three states using these files. The biggest headache wasn't finding the data. It was dealing with inconsistent date formats, missing facility codes, and the occasional month where a prison simply stopped reporting. I ran into a situation where one facility's "solitary" designation changed from ADX to a new classification mid-year, which made month-over-month comparisons useless unless you built a mapping table yourself. The workaround was scraping the internal policy memos linked in the facility footers and cross-referencing them with the classification columns in the daily rows.
The Erj Daily Incarceration Files Inside The Hidden World Of Solitary Confinement And Systemic Failure
Here is the practical workflow I used and what to watch out for. The files are organized around daily headcounts. Each row represents one facility on one date. Columns typically include total population, general population, restrictive housing population, and a breakdown by reason for placement. Some datasets add administrative segregation, disciplinary segregation, and medical isolation as separate columns. Others lump them together. You need to figure out which before you write a single line of code or open Excel. When I first pulled the dataset, I assumed the restrictive housing column was consistent across all entries. It was not. Two facilities in the same state used different definitions for the same column label. One counted any cell where an inmate spent more than sixteen hours alone. The other counted only cells with formal administrative ordering. I caught this by reading the methodology notes buried on the download page, which most people skip. The notes were contradictory between facilities, which is another common pattern.
Step two: Clean the data
Daily files come with their own set of problems. Blank cells are not always zeros. Sometimes they indicate missing reports. Sometimes the facility actually had zero people in solitary, though that is extremely rare in practice. You will need to flag blanks and trace them back to the facility's reporting schedule. Most prisons report daily. A few report weekly. A smaller number report sporadically. I built a simple preprocessing script that flagged any row where the restrictive housing count dropped to zero for more than three consecutive days, since that was almost always a data gap rather than a real count. From there I matched the flagged periods against known holidays and staffing changes reported in facility newsletters. This cut my manual verification time from roughly forty hours down to about twelve.
Get the Full Details
Step three: Cross-reference with external sources
The daily files alone will not give you a complete picture. You need to pull disciplinary incident reports, grievance statistics, and staff turnover data from the same period. Solitary placement is driven by institutional violence incidents, and those spikes show up in the daily numbers with a lag of about three to seven days. If you are studying causation rather than just describing trends, this lag matters a lot. I found that correlating solitary population changes with shooting incidents from the same facility's quarterly safety reports produced a much cleaner signal than looking at the raw daily numbers in isolation. The daily files show you the what. The incident reports show you the why.
Step four: Understand the limitations
This is where most people get careless. The files do not capture everyone in solitary. Some facilities place people in therapeutic isolation or emergency holds that do not appear in the standard restrictive housing column. Those placements can account for fifteen to twenty percent of the actual solitary population depending on the facility. You will miss them unless you dig into the clinical logs or file a records request for that specific category. Another issue is the delay. Data typically shows up thirty to sixty days after the date it covers. If you are tracking a current event, you are already behind. This is not a flaw in your process. It is a structural feature of how these systems operate. Budget cycles, audits, and staffing changes all create delays that compound over time. There is also the problem of facilities that exit the dataset entirely. When a prison closes or switches reporting systems, the time series breaks. You need to note those breaks explicitly in any analysis you produce. Treating them as continuous data will produce misleading trends.
Step five: Build your analysis
Once the data is clean and you understand the gaps, the analysis itself is straightforward. Monthly averages, trend lines, and comparison tables will cover most use cases. If you want to go further, you can build a regression model that includes variables like facility size, region, staffing ratios, and incident rates. But keep the model simple. Overfitting is easy when your independent variables are highly correlated with each other, which they tend to be in correctional datasets. My most useful output was a simple visualization showing restrictive housing population as a percentage of total facility population over time, overlayed with incident spikes. It told the story faster than any table ever could. People respond to visual patterns. They do not respond to spreadsheets.
Practical tips that actually matter
Save every version of your cleaned dataset. The original files change occasionally without notice. Facilities revise past entries. If you do not preserve your working copy, you lose reproducibility. I learned this after a facility updated their historical data and my numbers no longer matched what I had published. Having the raw and cleaned versions timestamped saved me from having to redo months of work. Use consistent facility identifiers. Names change. Codes stay the same. I wasted two weeks tracking down a facility because I searched by name instead of by the standard state-issued identifier. The name had been updated on the portal but my search query was still using the old label. Read the methodology notes before everything else. They are short, usually ignored, and essential for understanding what the columns actually represent. Skipping them is the fastest way to build an analysis on a foundation of incorrect assumptions.
When the files fall short
There are scenarios where the daily files simply cannot answer your question. If you need individual inmate trajectories through solitary, the aggregate daily numbers will not help. You would need case-level data, which requires a separate records request process and legal review. If you need psychological outcome data, that is not in these files either. They track placement, not impact. Some facilities also restrict access to their data entirely or release it in heavily aggregated form. In those cases, you are working with incomplete information, and you need to acknowledge that limitation clearly in any report you produce. Guessing at missing data is worse than admitting you do not have it. The files are a tool. They are not a complete record of what happens inside solitary confinement. They are daily snapshots of administrative decisions, and those decisions are shaped by everything from staffing shortages to policy changes to external lawsuits. Understanding the data is one thing. Understanding the system that produces it is another. Both are necessary if you want to do anything meaningful with the numbers.
