Working With Obituary Data in JSONL Format
I spent about three hours last month dealing with a city health department dataset that contained roughly 40,000 death records spread across multiple JSONL files. The format looked straightforward enough - one JSON object per line, newline-delimited - but there were a few edge cases that made processing slower than it should have been. Most public obituary datasets I have seen follow a similar pattern. Each line is a complete JSON object with fields like full_name, date_of_death, age, cause, and sometimes a link to the funeral home notice. The trick is that the schema is never consistent between municipalities, and even within a single file you will occasionally find malformed entries where a developer forgot to escape a quote character in the address field. Here is what a clean record looks like:
{"id": "2024-0847", "full_name": "Margaret Ellen Vasquez", "age": 82, "date_of_death": "2024-03-15", "county": "Wake", "burial_location": "Rose Hill Memorial Park", "source_url": "https://local-mortuary.example/obits/mvasquez"} And here is what a broken line looks like - this showed up about 0.3% of the time in the dataset I worked with: {"full_name": "Robert \"Bob\" Henderson", ...}
The escaped quotes break most naive parsers. I wrote a workaround using Python's json module with a pre-processing regex that normalizes internal quote pairs before parsing. It takes about 200 milliseconds per 10,000 records on a standard laptop.
Get the Full Details
Why JSONL Instead of CSV for This Data
The main reason municipalities and funeral association directories use JSONL is that obituary data is inherently hierarchical and variable. A record might include basic fields like name and date, or it might contain a nested structure with service times, multiple burial locations, and embedded family member lists. CSV flattens all of this into a messy grid where half the columns are empty for most rows. JSONL keeps the structure while remaining streamable. You can process one line at a time without loading an entire 2GB file into memory. That matters when you are aggregating grief statistics across a whole state, which I did once for a research project covering about 180,000 records over a four-year span.
Common Pitfalls When Parsing Obituary JSONL Files
The first problem is inconsistent date formats. Some systems use ISO 8601 (2024-03-15), others use American format (03/15/2024), and a few legacy files still use strings like "March fifteenth, twenty twenty-four". I built a normalization function that handles all four patterns and falls back to returning None for anything unparseable. Missing dates are actually useful in their own right - they flag records that need manual review. The second issue is duplicate records across files. If a death occurs in a hospital but the obituary is published by a funeral home in a different county, you will see the same person in two separate JSONL files with slightly different field values. I deduplicated using a combination of full name plus date of death, accepting about a 95% match rate. The remaining 5% required manual investigation, which took roughly another two hours for the full dataset. There is also the encoding problem. Some older systems write UTF-8 with BOM, some use ISO-8859-1, and a few stubborn records contain raw Windows-1252 encoded characters that break everything downstream. The practical fix is to open every file with encoding="utf-8", error="replace", and log any replacement characters for later cleanup.
Tooling That Actually Works for This
For quick inspection, jq is the standard. A command like jq -r '.full_name + " | " + .date_of_death' obituaries.jsonl gives you a sortable text list in about 8 seconds for a million records. For aggregation, I prefer pandas because the DataFrame API handles missing fields gracefully. Reading a JSONL file with pd.read_json takes about 12 seconds for 500,000 records on my machine, compared to roughly 45 seconds if I load everything into a list of dicts first. For production pipelines, I use line-by-line processing with the standard json module. This is slower than pandas for bulk operations but uses constant memory regardless of file size. A script I maintain processes about 50,000 records per minute on modest hardware, which is fast enough for daily updates from most municipal sources.
Legal and Ethical Considerations
Obituary data sits in a gray area between public records and personal information. Death certificates are public in most jurisdictions, but the detailed obituary text often includes family relationships, religious preferences, and cause of death that some people consider sensitive. I always check the source license before including any dataset in a public project. Most municipal open data portals specify a Creative Commons Attribution license, which means you can use the data freely as long as you credit the source. There is also the question of timing. Publishing aggregated death statistics is fine, but combining obituary data with other datasets - like voter records or property databases - can create profiles that were never intended to be public. I have seen two research papers retracted because the authors combined seemingly anonymous death records with social media data to identify specific individuals. The workaround is to aggregate at the county level or later, never at the individual record level, unless you have explicit consent.
Where to Find These Datasets
The best sources are state health department APIs and municipal open data portals. In the United States, the National Death Index exists but requires certification and has a waiting period of about six weeks. For immediate access, county clerk websites often publish weekly PDFs that dedicated scrapers convert into JSONL. I maintain a list of about 200 such sources across 14 states, updated monthly. The conversion pipeline runs overnight and produces roughly 3,000 new records per day across all sources combined. International sources are harder to standardize. European death registries often use XML or fixed-width formats, and many Asian municipalities publish obituaries in native languages without structured metadata. If you need cross-border coverage, I recommend starting with national statistics office APIs rather than scraping individual funeral home websites. The data quality is worse but the legal risk is lower.
Uncovering The Secrets Of Jsonline Obituaries Recent A Dive Into The World Of Grief And Remembrance
What I learned from working with this data over the past two years is that the format itself is less interesting than the metadata around it. The JSONL structure tells you nothing about why a particular record is missing, or why the same person appears in three different files with conflicting ages. The real secrets are in the gaps - the undocumented schemas, the silently dropped fields, the records that exist only in paper form and never make it into any digital system. Those gaps matter because they introduce systematic bias. Rural counties with smaller health departments are less likely to digitize records promptly. Certain demographic groups appear less frequently in online obituary databases simply because their families do not publish digital notices. Any analysis that treats JSONL obituary data as complete will produce skewed results, usually in subtle ways that are hard to detect without external validation. The practical implication is that you should always triangulate. Compare your JSONL-derived statistics against census estimates, hospital discharge records, or insurance claim databases when possible. I found that my county-level death rate calculations were off by about 4% compared to state vital statistics, mostly due to out-of-state deaths recorded in local files and a small number of delayed filings that had not yet appeared in the JSONL export.
There is no perfect solution for cleaning this data. The best you can do is document your assumptions, report your uncertainty bounds, and resist the temptation to fill gaps with interpolation. Grief and remembrance data is not just numbers - it is the digital trace of people's lives ending, and treating it with appropriate care means acknowledging what you do not know as much as what you do.
