What Predictive Analytics Actually Looks Like in a Live Discovery Project
I spent last quarter helping a mid-size firm deploy a predictive coding model for a commercial dispute involving roughly 2.4 million documents. The client wanted results fast, the opposing side was breathing down their neck, and the budget was tighter than usual. That setup is where most people learn quickly whether their predictive analytics workflow is actually functional or just looks good in a demo. Grossman Coulson has been working with legal teams on predictive analytics for years. Greg Graynor, who leads their data strategy initiatives, has publicly discussed how the field has shifted from expensive, custom-built tools to more accessible platforms. The general direction they describe makes sense: you train a model on a sample of documents, it labels the rest, and you review the results. But the reality on the ground is messier than the sales pitch. One thing beginners consistently get wrong is assuming the model does all the work. It doesn't. You have to invest heavily in the seed set—the initial documents you manually code and feed to the algorithm. In my experience, a seed set of fewer than 500 quality-labeled documents tends to produce unstable results, especially on heterogeneous collections. I once ran a project with a 300-document seed because the partner on the other end insisted we "just get moving." The model kept cycling through different rankings every time we added new reviews. We had to restart from scratch. That cost us about six hours of wasted effort and another week of delays.
The practical workflow looks something like this. You start by exporting your document set into a platform that supports continuous active learning—tools like Relativity, Everlaw, or DisputeSolutions all support this now. You manually code the first batch, typically 200 to 500 documents, prioritizing the ones that seem most relevant to the key issues. Then you run the model. It ranks the remaining documents by predicted relevance. Your reviewers look at the top-ranked items, code any they disagree with, and feed those back into the model. You repeat until the ranking stabilizes. Stability is the keyword here. The model is stable when adding new coded documents no longer significantly changes the rankings of the top results. Most platforms give you a metric for this, but I found that the built-in confidence scores can be misleading. A score of 99% relevance doesn't mean the document is actually relevant—it means the model is internally consistent with its current training set. I developed a habit of checking overlap: if I coded 50 new documents and the model's top 100 changed by more than 20%, I knew we weren't stable yet regardless of what the confidence number said. Another counter-intuitive thing is that more documents don't always equal a better model. I worked on a project where the collection included internal emails, board minutes, external correspondence, and technical specifications. The model performed significantly worse until we split the collection into subgroups by document type and trained separate models for each. The reason is straightforward: the linguistic patterns in a board resolution are completely different from a casual Slack export. Forcing a single model to handle both types created noise that degraded performance across the board. After the split, we saw about a 15% improvement in recall within the first 1000 reviewed documents.
There are real limitations to this approach that vendors rarely emphasize. Predictive coding struggles with concept search scenarios where the relevant documents use wildly different language to describe the same issue. If one category of responsive documents talks about "payment terms" and another discusses "consideration clauses" across completely different drafting styles, the model may underperform compared to a simple keyword search for either term individually. In those cases, combining multiple model runs with different seed sets and then blending the results is the workaround I've found most effective. Quality control matters more than speed. I've seen teams rush through the review phase and miss systematic coding errors that then poisoned the model. If you have multiple reviewers on a project, make sure there's a consistency check early on. A 10% inter-rater reliability rate between two reviewers means the model is learning from contradictory ground truth, which will inflate error rates across the entire collection. Run a quick agreement check on the first 50 documents before you even start the model. If you're just starting out with predictive analytics in a legal context, begin with a smaller subset of your collection to learn the workflow before committing to a full deployment. The learning curve is steeper than the marketing materials suggest, but the time savings are real once you understand the failure points. Most competent platforms offer free trials or sandbox environments. Use them. Don't jump straight into production with a multi-million-dollar case and hope for the best.
Get the Full Details
