How I Actually Get Government Procurement Data Without Losing My Mind
I spent three weeks last year trying to pull contract data for a mid-sized defense subcontractor looking at agency spending. The official path went through SAM.gov, FPDS-NG, and apparently a half-dozen API endpoints that changed their schema twice in six months. What I ended up doing was build a simple script that pulls from the USAspending.gov bulk downloads and cross-references against the Federal Procurement Data System JSON files. It took four hours to set up and now runs automatically every morning. Most people don't know you can just download the entire dataset locally instead of making individual API calls. The raw JSON files for FPDS-NG are available at fpds.gov/developers/downloads. Each monthly release is roughly 2 to 4 gigabytes depending on the fiscal quarter. That seems excessive until you realize a single API call for a specific grant or contract returns maybe 3KB of data with all the context stripped out. The core platform for public procurement transparency in the United States is USAspending.gov. It pulls from two primary source systems: the Federal Procurement Data System (FPDS), which tracks procurement contracts and solicitation data, and the Federal Award Information System (FAIMS), which handles grants, direct payments, and other non-procurement awards. They feed into a single searchable interface, but they are not the same thing. When I was tracking down a particular IDIQ vehicle for an environmental remediation contract, I kept getting results from FPDS that overlapped with FAIMS entries for the same awardee because the crosswalk between the two systems is loose at best. The award identification number format differs. FPDS uses a 16-character string while FAIMS uses something closer to a 20-character alphanumeric key. If you are doing any serious analysis, you need to keep the two systems separate in your head even though the website tries to merge them. Here is something most beginners miss: the solicitation data on USAspending.gov is complete for awards over $25,000 under thekle Act requirements, but the award level data has known gaps. Specifically, task orders under blanket purchase agreements and micro-purchases below the Simplified Acquisition Threshold are often missing or incomplete. I encountered this directly when a client was trying to reconstruct the full spend history of a specific agency program. The published data showed around $2 million in contract actions. The actual obligation was closer to $4.7 million once we pulled the task order level detail from the agency's own FOIA responses and matched them against the individual vendor invoices they had on file. The gap existed because the task orders were individually below the reporting threshold even though the parent contract was well above it. This is not an anomaly. It is a structural limitation of how the reporting thresholds are defined.
If you want to actually work with this data rather than just search it, your best path is to use the USAspending.gov Advanced Search API or the bulk download FTP. The API has a rate limit of roughly 100 requests per minute for anonymous access, which is fine for occasional lookups. It becomes a problem when you are building a dataset spanning multiple fiscal years with thousands of records. I recommend setting up an API token through the developer portal at USAspending.gov/developers and then moving to the monthly bulk downloads. The data files are compressed CSV and JSON, and they update on the first business day of each month for the prior month's activity. There is a lag of about 45 days between the date of obligation and the date the data appears publicly, so do not expect real-time visibility into current contracting activity. Another thing nobody warns you about is the entity identifier problem. The data uses DUNS numbers as the primary business identifier, and Transition Notice 24-03 requires a move to Unique Entity IDs under the System for Award Management. As of October 2025, the transition is incomplete and both formats coexist in the datasets. If you are merging data across multiple fiscal years, you will need a mapping table that correlates legacy DUNS numbers to the new UEI strings. I built one by pulling the SAM.gov registrant exports and cross-referencing them against the historical FPDS records. It took about 12 hours of data cleaning because roughly 8 percent of the records had mismatched or missing identifiers due to vendor registration lapses during the transition period. For anyone actually relying on this data for business decisions, here is the blunt assessment: USAspending.gov is useful for identifying opportunity patterns and researching competitors, but it should never be your sole source for award value or obligation data. The numbers can be off by 10 to 20 percent at the agency level depending on how frequently reprofiling occurs. Contract modifications that increase or decrease obligations are reported back to FPDS, but the timing is inconsistent across agencies. Some upload corrections within days. Others batch them quarterly. The data on the website reflects the most recent published snapshot, which means you are always looking at slightly stale information. If you need precision, you go to the individual agency procurement portals or file a targeted FOIA request for the specific contract action you are investigating.
The most practical workflow I have found is to run an initial scan through USAspending.gov to identify potential opportunities, then validate the key details against FPDS-NG direct downloads and cross-check against agency procurement announcements when they exist. For DoD contracts specifically, the Defense Contract Management Agency publishes daily summaries that fill in many of the gaps. For civilian agencies, the results are mixed. GSA has a reasonably complete portal. The Department of Health and Human Services procurement section is harder to navigate and their data publication schedule is irregular. The Department of Energy sometimes publishes award data with a delay of up to 90 days. I do not recommend paying for third-party procurement intelligence platforms unless you need daily automated alerts and historical trend analysis going back beyond the current decade. The data is available for free. It is just tedious to work with directly. A well-structured SQLite database with the monthly FPDS JSON files and a clean entity mapping table will serve most users far better than a subscription service that is ultimately pulling from the same public sources. The setup is not trivial. I have spent weekends building and maintaining mine, but once it is running, the queries are fast and the data is yours. You are not subject to someone else's rate limits or data quality standards.
Get the Full Details
