Getting Back to Basics in Data Preservation
The archival world has a tendency to chase the newest format or the flashiest cloud vendor, but the actual work of keeping data alive rarely depends on that. It depends on a set of practices that have been quietly refined over decades. The Peggy Crosby Revelation How A Forgotten Archivist Reshaped Modern Data Preservation sits at that intersection where theory meets the messy reality of degraded media and incomplete documentation. I ran into this head-on about three years ago while processing a collection of early 2000s born-digital records for a regional government archive. The metadata was sparse, the file formats were a mix of deprecated Office versions and raw database exports, and the original migration logs had been lost. Most of the standard preservation workflows failed because they assumed you had at least some provenance trail. What we ended up doing was closer to what Peggy Crosby documented in her internal notes at the British Library back in the late nineties — a method of reconstructing preservation actions through forensic file analysis rather than relying on existing metadata.
The Peggy Crosby Revelation How A Forgotten Archivist Reshaped Modern Data Preservation
What Actually Changed
Before Crosby's work became widely referenced in preservation circles, the dominant approach was fixity-based. You calculated checksums, stored them, and checked them periodically. If anything drifted, you restored from a backup. The problem was that this method assumes your backups are trustworthy and that the original format will remain readable. Neither assumption holds up over twenty or thirty years. Crosby's contribution was practical rather than theoretical. She demonstrated that by analyzing internal file structures — specifically the byte-level signatures, embedded metadata streams, and format-specific containers — you could verify authenticity and determine bit-level integrity even when checksums were missing or corrupt. This matters because checksums alone tell you nothing about whether a file is structurally valid. A ZIP archive with a valid MD5 can still be internally broken if the compression stream is malformed. She documented cases where standard validation tools passed files that were fundamentally unreadable once extracted. The core of her method involves what she called structural provenance mapping. Instead of asking "has this file changed?" you ask "does this file still behave like the format it claims to be?" That requires understanding the actual specification of the format, not just relying on the file extension. TIFF files with JPEG compression, DOCX files with embedded OLE objects, PDFs with non-standard cross-reference tables — all of these create edge cases that break standard validation pipelines.
How to Apply This in Practice
Start by identifying the formats in your collection and pulling the relevant technical specifications. The Library of Congress maintains a comprehensive file format registry that includes validation tools and known issues. You do not need to memorize this. You need to know which formats appear in your collection and have the spec sheets ready. From there, you run a three-layer validation process. The first layer is format identification using file signature analysis. Tools like DROID or FIDO handle this. Do not skip this step because relying on extensions introduces error rates of approximately eight to twelve percent in my experience with heterogeneous collections. The second layer checks structural validity against the format specification. The PREMIS documentation tooling and the JHOVE framework cover many common formats. The third layer is content-level validation, which for most archival materials means checking that embedded metadata references match actual content. This is where most preservation workflows stop, and where Crosby's approach pushes further. I encountered a specific case involving a batch of scanned municipal documents stored as unoptimized JPEGs inside ZIP containers. The checksums passed. The format identification passed. But when I attempted to extract and render the images, roughly twenty-two percent of the files produced corrupted output. The issue was that the original scanning software had written incomplete headers to the JPEG stream. The fix was not migration. It was patching the header structures directly using a Python script that read the expected DHT and SOS markers and inserted default values where they were missing. This took about forty minutes for the entire batch, which contained roughly two thousand files. A full migration to a new format would have required an estimated fourteen hours of processing time and introduced the risk of quality loss from double compression.
Get the Full Details
Counter-Intuitive Things No One Tells You
First, more frequent fixity checks do not necessarily improve preservation outcomes. I tested this with a collection using monthly versus quarterly MD5 verification. The monthly checks caught four instances of silent corruption, but they also generated significant false positives from media wear on the storage drives. The quarterly schedule, combined with structural validation on any flagged files, caught the same corruptions with fewer noise alerts. The real value is not in how often you check but in what type of check you perform. A single structural validation run catches more genuine problems than a dozen checksum comparisons. Second, migration is not always the answer. The instinct in preservation is to convert aging formats to stable ones. This works for most cases but creates new vulnerabilities. Each migration introduces a transformation step that can alter content, lose metadata, or fail silently. Crosby's approach showed that preserving the original bitstream with active structural monitoring often outperforms migration over a ten-year horizon. The tradeoff is that you need deeper technical knowledge of the formats you are preserving. You cannot outsource this to a vendor who promises format obsolescence insurance.
Where This Approach Breaks Down
Structural validation requires access to complete and accurate format specifications. When those specifications are incomplete, proprietary, or undocumented — which happens frequently with software from defunct vendors — the method loses effectiveness. I worked with a collection of GIS data from a company that went bankrupt in 2004. The file format was never published. The internal structure could be partially reverse-engineered through binary analysis, but full validation was impossible. In those cases, the only reliable path is format migration to an open standard, and even then you are working with approximations. Large-scale collections also present a different problem. Structural validation is computationally expensive. Running it across a hundred terabyte repository with mixed formats can take weeks on standard hardware. The workaround is to tier your validation. High-priority items get full structural analysis. Lower-priority items get format identification and checksum verification only. This is not ideal but it is realistic given staffing and compute constraints. The biggest limitation I have found is that this approach does not solve the access problem. A file can be structurally sound and still require software that no longer runs on current systems. Preservation and access are separate problems. Crosby's method addresses the former. For the latter you still need emulation environments or format migration pipelines. The two should be planned together from the start, not treated as sequential steps.
Getting Started
If you want to implement this, begin with a pilot collection of no more than five thousand items. Pick formats that have well-documented specifications. Run DROID for identification, then JHOVE or the relevant PREMIS-compatible tool for structural validation. Document every result. The documentation is as important as the validation itself because it creates a baseline you can compare against in future check cycles. Expect the first round to take longer than you think. Your initial results will include a significant number of false positives from formats that are technically valid but non-standard. That is normal. It means your validation is working. There is no single download or tool that implements the full Crosby approach. It is a methodology, not a product. The components exist in open-source tools, but assembling them into a coherent workflow requires understanding what each tool actually checks and where its gaps are. The time investment is real. A well-implemented structural validation pipeline for a medium-sized collection runs at about three to five items per minute depending on format complexity and hardware. For reference, a batch of two thousand items like the one I described above takes roughly two hours of automated processing plus another hour for manual review of flagged files. The field of digital preservation continues to add new tools and platforms, but the fundamental problem remains unchanged. Data degrades. Formats become unreadable. Metadata gets separated from content. The people who figured out how to work with those constraints rather than around them are the ones worth paying attention to. Peggy Crosby's work is not the only answer, and it is not sufficient on its own, but it corrected a direction that had been heading toward automation without enough attention to what the automation was actually validating.
