Getting a List Crawler to Work in Detroit Actually Isn't That Hard Once You Figure Out the Local Data Landscape
Detroit's business and public records are scattered across multiple county databases, city portals, and third-party aggregators, which makes crawling it a bit of a chore. The straightforward part is setting up the crawler itself. The part most people gloss over is the data source mapping, and that's where projects usually fall apart. I spent about three months troubleshooting a project that pulled from Wayne County property records, Michigan business filings, and Detroit city permits. The data sources overlap significantly but use different schemas and rate limits, so I ended up building a three-layer extraction pipeline instead of trying to hit one endpoint. I'll walk through what actually worked. First, pick your data source tier. In Detroit, the biggest buckets are county assessor records (Wayne County), state-level business registrations (Michigan SOS), municipal licensing data, and then aggregated commercial lists from providers like ZoomInfo or Apollo that already cover the metro area. If you're building this yourself, start with the county assessor data because it's public and relatively well-structured. The URL is the Wayne County Recorder and Treasurer site, but it's behind a search interface that requires form submissions, not direct API calls. You'll need to simulate POST requests with the right payload parameters. I found that using requests with session objects and persistent cookies worked better than trying to parse the page renders with Selenium, which slowed everything down by an order of magnitude.
The second layer is the Michigan Business Registry. This one has an actual search endpoint, but it's rate-limited to something like 10 requests per minute unless you have an account with higher limits. The workaround I used was to batch queries in groups of 8 with a randomized delay between 6 and 12 seconds between batches. It added time but kept the crawler from getting blocked. I also noticed that searching by county name returns results faster than searching by address, which is useful when you're pulling entire ZIP code lists for Detroit proper (48201 through 48239 roughly). The third layer is where things get messy. Detroit municipal data lives on the city's own portals and they change their interfaces without notice. I ran into this specific problem last year when the city switched their business license lookup from a standard web form to a JavaScript-heavy single-page app. My crawler that had been running for eight months suddenly returned empty results. I couldn't tell immediately because the response codes were still 200, just with no data in the body. The fix was switching to a headless browser specifically for that endpoint, but only routing the license lookup through it while keeping the faster HTTP-based approach for everything else. That split architecture kept overall throughput reasonable. When you're building the actual crawler script, here's the structure I'd recommend:
Source configuration file — a JSON or YAML file that defines each data source, its base URL, request method, expected parameters, rate limit, and retry logic. Don't hardcode URLs in your Python scripts. You'll regret it the moment a source changes and you have to search through ten different files to find it. A deduplication layer — this is non-negotiable. Detroit businesses will appear in county records, city records, and state filings under slightly different names or addresses. I used a combination of fuzzy matching on business names (RapidFuzz library, threshold around 0.85) and normalized address parsing with USPS standards. Without this, your final list will have 30 to 50 percent duplicates depending on how broad your search is. A queue system — I used Redis for the task queue because it handles retry logic and rate limiting natively. RabbitMQ works too but adds operational complexity that isn't worth it for a project of this scale. If you're doing this solo or with a small team, stick with Redis. It's one container, one config file, and it just works.
Get the Full Details
For the output format, I'd suggest CSV with these columns at minimum: business name, address, city, state, ZIP, phone, website, source, last verified date, and a confidence score. The confidence score is something most people skip, but it matters. When a business appears in only one source, flag it as low confidence. When it appears in two or more with matching details, flag it as high. This lets you sort and filter later instead of wondering why half the phone numbers are disconnected. Now, the uncomfortable part about list crawlers for any specific metro area like Detroit. They have real limitations. Public data sources update on their own schedules, not on yours. Wayne County property records might lag by several weeks. City license data can be months behind. If you need current information for outreach or compliance purposes, a crawler alone won't give you that. You'll need a verification step, which usually means a second pass with a service like NeverBounce or ZeroBounce for email validation, or a call center for phone number verification. That's an additional cost that people forget to budget for. There's also the legal side you should consider. Crawling public data is generally fine, but the Terms of Service for some of these government portals explicitly prohibit automated access. I've seen this create problems when companies use scraped lists for commercial outreach. It's not a guaranteed legal issue, but it's a risk. If you're doing this for internal research or non-commercial purposes, you're probably fine. If you're selling the list or using it for marketing campaigns, talk to someone who knows Michigan data privacy law before you go further.
For tools, I've used a mix of custom Python scripts with Scrapy for the heavy lifting, BeautifulSoup for simpler page parsing, and a bit of Playwright for the JavaScript-rendered endpoints. Open-source solutions like Scrapy are solid if you know Python. If you don't, there are commercial list-building platforms that let you specify a geographic filter, but you're paying a premium for convenience and you have less control over the data quality. The choice really depends on whether you need one-off lists or ongoing data maintenance. One more thing that took me way too long to figure out: ZIP code boundaries in Detroit don't align neatly with city limits. Some 482xx ZIP codes extend into surrounding suburbs, and some Detroit neighborhoods span multiple ZIP codes. If you're filtering by ZIP, you'll get non-Detroit results mixed in. The more precise approach is to use a GeoJSON boundary file for Detroit city limits and filter your results against that after the initial crawl. It adds a step but it actually matters if accuracy is important to you.
