Uncovering The Hidden World Of Jsonline Obits A Treasure Trove Of Family Histories And Community Connections

Working With Obituary Data At Scale

I spent about three years building pipelines that scrape, normalize, and index newspaper obituaries from hundreds of regional publications. Most people approaching this problem start by thinking they need a website or a search portal. They don't. What actually works is a clean JSON Lines file and a willingness to deal with messy source data. That is the foundation. JSON Lines, sometimes called newline-delimited JSON, is simply one valid JSON object per line in a text file. No commas between records. No outer array brackets. When you are working with tens of thousands of obituary records, the difference between a standard JSON file and JSON Lines is not aesthetic. A 50-megabyte JSON file with nested arrays will choke most tools. A JSON Lines file of the same data streams through grep, jq, Python, and awk without breaking a sweat. You can process it record by record with minimal memory overhead.

Uncovering The Hidden World Of Jsonline Obits A Treasure Trove Of Family Histories And Community Connections

Here is how the format typically looks when you extract a single obituary record: { "name": "Martha Eleanor Voss", "date_of_birth": "1928-03-14", "date_of_death": "2023-11-02", "location": "Grundy County, IL", "survivors": ["Robert Voss (husband)", "Linda Voss-Martin (daughter)", "Thomas Voss (son)"], "service_details": "St. Mary Catholic Church, Morris, IL, 11/09/2023 at 10:00 AM", "newspaper": "Greenville Gazette", "url": "https://example.com/obits/voss-martha" } Each line is independent. You can append new records without rewriting the whole file. You can split the file across multiple workers for parallel processing. You can read it back in any order. This matters when you are combining data from twenty different newspaper archives, some of which only provide CSV exports with no consistent schema.

The real work starts with normalization. Obituary sources are wildly inconsistent. One paper lists the deceased as "Martha E. Voss." Another lists "M. Eleanor Voss." A third lists "Martha Voss, age 95." If you dump raw data into JSON Lines without a cleaning step, your downstream search will return three separate records for the same person. That breaks everything that comes after it. My standard approach is a two-pass normalization pipeline. The first pass extracts structured fields from the raw HTML or PDF using regex and heuristic rules. The second pass resolves identities across sources. For identity resolution, I use a combination of fuzzy matching on names plus exact matching on dates of birth and death. A tool like Dedupe.io handles this well for datasets under about fifty thousand records. Beyond that, you are better off writing a custom matcher that uses phonetic encoding like Soundex or Metaphone alongside date constraints. I ran into a specific edge case that took me two weeks to resolve. A set of obituaries from rural Kentucky newspapers had a habit of listing married women under their maiden names in the headline but under their married names in the body text. So "Betty Jean Harlow" would appear in one database as both "Betty Jean Harlow" and "Betty Jean Harlow-Cummings." Standard fuzzy matching rated those as a partial match, but the deduplication logic kept them separate because the surname field did not align closely enough. The workaround was to add a middle-name and maiden-name expansion step before the matching phase. I wrote a small function that takes any hyphenated surname, splits it, and creates alternate name variants. "Harlow-Cummings" becomes both "Harlow" and "Cummings" as potential surnames. After that change, my match rate jumped from about 62 percent to 94 percent on that particular dataset.

Once your data is normalized, the JSON Lines file becomes your single source of truth. You can query it directly with jq for quick lookups, feed it into Elasticsearch or Meilisearch for full-text search, or import it into a SQLite database if you need relational queries. I usually keep all three: the raw JSON Lines file for archival purposes, an Elasticsearch index for search, and a SQLite database for complex joins between obituary data and census records or family tree exports. There are a few things beginners consistently get wrong. The first is assuming that date fields are clean. They are not. You will find "March 2, 1928," "03/02/1928," "2 Mar 1928," and plain text fragments like "passed away peacefully on a Tuesday in early November." Parse what you can. Leave the unparseable dates as raw strings in a separate field rather than forcing a conversion that will produce garbage. An invalid date is worse than no date because it silently corrupts your timeline queries. The second mistake is flattening survivor information into a single comma-separated string. Survivor lists are structured data. Each survivor has a name and a relationship. If you dump them into one field, you cannot query for "all people who survived by a daughter named Linda" or "all obituaries where the spouse predeceased the subject." Store survivors as an array of objects with name and relationship keys. It makes the file slightly larger and the initial parsing slightly more involved, but the querying payoff is massive.

Another counter-intuitive point: JSON Lines is not always the best format for the final product. It is excellent for intermediate processing and pipeline work. But if you are sharing data with other researchers or handing it to a family history society, they will likely want CSV or a proper database export. JSON Lines requires tooling knowledge to read comfortably. I recommend keeping your JSON Lines file as your working master and generating CSV snapshots on demand for distribution. A simple Python script with the csv module can do this in under thirty lines. The downsides of this approach are real. JSON Lines gives you no schema enforcement. A typo in a field name on one line will not break the file, but it will break every script that expects that field to exist. I solve this by running a validation step after normalization using a library like JSONSchema or even a lightweight pydantic model. The validation catches malformed records before they enter the main file. It adds maybe ten minutes to a five-hour processing run, which is a reasonable tradeoff. You also need to consider data provenance. Obituaries are frequently copied across newspapers. The same text may appear in five different local papers with minor variations. If you do not track the original source URL and publication date, you will double-count events and inflate survivor networks. Every record in your JSON Lines file should have an explicit source field. I include source_url, source_publication_date, and a scrape_timestamp. This makes it easy to filter out duplicates and verify information against the original publication.

For people who want to start without building a pipeline from scratch, there are a few existing resources. The US Newspaper Directory provides metadata but not full text. Chronicling America from the Library of Congress has obituary content but no structured API. Some genealogy platforms like Ancestry and Find a Grave have obituary collections, but their data is locked behind subscriptions and their export options are limited. The most practical path is to scrape or request data directly from local newspaper archives, normalize it yourself, and store it in JSON Lines format. It is more work upfront, but you own the data and you can query it however you need. If your dataset grows beyond roughly one hundred thousand records, you will want to move to a proper search engine. Elasticsearch handles that scale comfortably. Meilisearch is lighter and faster for smaller deployments. For under ten thousand records, jq and SQLite are sufficient and require zero infrastructure. I usually start projects with SQLite and migrate to Elasticsearch only when query latency becomes a problem, which typically happens around the fifty-thousand-record mark depending on your indexing strategy. The format itself is trivial to generate. Any language with a JSON library can write JSON Lines. A Python example is straightforward: open a file for writing, iterate over your records, and call json.dumps on each one followed by a newline. No special libraries beyond the standard json module. If you are pulling from a web scrape, write each record to the file as you parse it rather than accumulating everything in memory. This keeps your RAM footprint flat regardless of dataset size.

What makes this approach useful for family history is that obituaries contain information that official records often omit. Maiden names of spouses. Migration patterns revealed through service locations. Church affiliations. Employer names. Military service details. Sibling relationships listed through survivor names. These fragments are scattered across thousands of local newspapers and rarely indexed together anywhere. A well-structured JSON Lines collection lets you assemble them into something coherent. You can run queries like "find all people from Grundy County, Illinois who died between 1920 and 1940 and were survived by a sibling named James." That kind of query is awkward or impossible in most genealogy platforms but trivial against a local JSON Lines file. The community angle is worth mentioning separately. These datasets tend to attract people who share them. I have seen small towns where a single researcher built a local obituary index and then posted it online. Within a year, descendants of the people in those obituaries were contacting each other, filling in gaps, and correcting errors. The data became a living resource rather than a static archive. If you build something like this, plan for community contributions. A simple pull-request workflow on GitHub works better than you might expect for this purpose. I do not recommend this approach if you are looking for a quick solution. Building a clean, normalized obituary dataset from raw newspaper sources typically takes two to four weeks for a moderate-sized county archive, depending on the quality of the source material. If your goal is simply to find a specific obituary, searching the original newspaper website or using a commercial genealogy site will be faster. JSON Lines pipelines are for people who need to analyze patterns across many records, not for people who need one record quickly.