From scattered public filings to a clearer view of future housing supply.
I built a data pipeline that collects environmental filings, permits and planning cases from government websites and connects them into a searchable database. It helps me investigate upcoming developments, identify builders and assess future competition around a property.
For AI and data professionals
- Collect public records.
Python scrapers collect environmental filings, permits and planning cases, reusing readers where portals share software. Dates, record identifiers and source references stay with the data.
- Classify and connect projects.
Rules standardize known fields; language models interpret descriptions and documents. Addresses are mapped where possible, and matching rules connect related filings in supported sources while retaining the underlying records.
- Check data and estimates.
Checks flag stale data, missing locations and coverage changes; separate evaluations compare classifications with independently labelled examples. Source-reported unit counts, acreage-based estimates and assumed delivery dates are labelled separately.
- Deliver through dashboards and AI tools.
Dashboards, spreadsheets, underwriting inputs and read-only MCP tools make the data searchable. I review uncertain matches and estimates; coverage varies, so no results does not establish no supply.
For real estate professionals
- Find upcoming development.
Planning cases, environmental filings and permits bring development activity into one place, helping identify potential housing supply before a community starts leasing or selling.
- Assess size and stage.
Filing types and permit statuses indicate progress through proposals, approvals and construction. Published unit or lot counts are separated from acreage-based estimates; an application does not establish delivery timing.
- Identify competition and builders.
Mapped projects show nearby activity and available builder, developer and applicant details. Product classifications distinguish housing types; whether homes will be rented or sold requires supporting evidence.
- Inform underwriting.
Supply joins rent comps and demand data to inform leasing, concession and rent-growth assumptions, while builder activity can guide outreach. I confirm relevant projects, likely timing and coverage gaps before relying on the results.
Ask the supply database a question in plain English — alongside the rent and demand databases it shares an MCP server with. The chat box lives on the Connectors tab, next to the server and its tools.
Try it on the Connectors tabDoes the model label filings correctly?
A language model reads each stormwater filing and says what will be built: single-family homes, townhomes, apartments, a mix, or unknown. I score it against filings I labelled by hand from the filing's PDF, the developer's website and the county's permits, never from the model's own answers. Re-run on October 2, 2026 with Claude Haiku 4.5; both scores match the June baseline.
| Metro | Labelled filings | Exact type | Right business bucket | Rent / sale errors |
|---|---|---|---|---|
| Atlanta | 93 | 82.8% (77) | 88.2% (82) | 6 |
| Phoenix | 65 | 84.6% (55) | 86.2% (56) | 7 |
Buckets are for sale (single-family and townhome), for rent (apartments), mixed and unknown, plus commercial in Phoenix. A rent / sale error is the one that matters: a for-sale subdivision counted as rental supply distorts a rent underwrite, so any one of them fails the run and names the filing. In production, Atlanta adds a developer lookup on top of the model. I don't publish its score on these filings, because the lookup was built from them.
Where the AI runs, and what it costs
Every model call writes a row to a log, so this is measured, not estimated: the last 30 days to October 2, 2026.
| Step | Model | Calls | Cost | Why a model |
|---|---|---|---|---|
| Reading filing PDFs | Claude Haiku 4.5 | 363 | $3.38 | Stormwater notices and subdivision reports put the site, acreage and owner in forms and prose that rules cannot parse reliably. |
| Filling gaps in multifamily permits | Haiku 4.5 + web search | 198 | $5.23 | Finds the developer, project name and unit count a permit leaves blank. Large backfills go through the batch API at half price. |
| Labelling product type | Claude Haiku 4.5 | 208 | $0.18 | The classifier scored above. It runs only on new filings that have no label yet. |
| The “Ask the databases” chat (Connectors tab) | Claude Sonnet 4.6 | 112 | $2.50 | Chooses the tools, then writes the answer from what they return. Capped per visitor, per hour, and in dollars per day. |
The pipeline's own model calls cost about $9 a month. Everything else is ordinary code, on purpose: fetching, field mapping, address matching, linking filings into projects and the freshness checks. Those steps must give the same answer every week across hundreds of thousands of records, and a model would add cost and variance there without adding judgment.
One failure, and the check it left behind
On June 3 a migration pointed the summary view behind the Overview and Builders pages at a stored copy of itself. Every refresh copied the same numbers back, so the analytics showed mid-May data. Nothing errored, and the project map, which reads the source tables directly, stayed current. It was found and fixed on June 5. The view was rebuilt as a live union of each metro's pipeline and checked metro by metro against the frozen copy before the swap.
The lasting change is a check that compares the newest record each dashboard serves with the newest record ingested beneath it, so a frozen view is flagged by the daily checks instead of waiting for someone to notice. More failures and the tests they produced are in the Chain contracts section of the Blog.
Tracked feeds are the sources the pipeline currently collects, one public source in one metro; retired sources are not counted. Records are the rows those feeds hold today, not unique projects.
- 254tracked source feeds
- 441,243source records
- 9metros
- 212feeds active in the last 8 days
- 115,834planning and zoning cases
- 2,498active multifamily and townhome permit projects
Coverage measured from the live database on September 30, 2026.
Every feed is one public source in one metro: a county's permit portal, a city's zoning map, a state's stormwater filings. What decides how hard a feed is to keep alive is not the agency but the way it publishes, so that is how I group them. 200 of the 254 come from something structured — a map service, a portal's own JSON, an open-data API or a spreadsheet — and a reader written once for a vendor covers every jurisdiction running that software. The rest are web pages, PDFs and a handful of portals that only work in a real browser, which is where most of the maintenance goes.
| Method | Feeds | Metros | Records | One real example |
|---|---|---|---|---|
| GIS map service A county or city ArcGIS layer queried like an API. Structured and cheap; breaks loudly when a layer is renamed. | 93 | 9 | 218,455 | NSHDavidson County building permits issued, from Metro Nashville's ArcGIS hub 33,291 records |
| Permit-portal JSON The JSON a permit or agenda portal feeds its own search page. One reader per vendor covers every jurisdiction on that software. | 88 | 9 | 75,024 | LASHenderson entitlement plans, from the city's Tyler EnerGov portal 8,864 records |
| HTML page scrape Server-rendered search pages, parsed field by field. The most layout-sensitive of the structured methods. | 22 | 6 | 49,209 | HOUTexas construction stormwater notices of intent, from TCEQ's search pages 18,236 records |
| PDF documents Agendas, weekly lists and public reports. Text is extracted, and a language model reads what rules cannot. | 17 | 5 | 27,808 | PHXArizona subdivision public reports: a CSV index plus one PDF per report 25,735 records |
| Browser automation Portals that only work in a real browser, driven headless. Slowest and most fragile, so it is the last resort. | 12 | 3 | 2,603 | LASClark County planning cases, from its Accela citizen portal 592 records |
| Open-data API A published open-data API with documented fields. The easiest feeds to keep healthy. | 11 | 5 | 31,118 | AUSCity of Austin subdivision review cases, from its Socrata open-data API 5,707 records |
| Spreadsheet download An xlsx or csv file published on a schedule. One request, then a column mapping that must survive format changes. | 8 | 4 | 36,233 | CLTNorth Carolina construction stormwater permits, one xlsx from NC DEQ 14,539 records |
Records are rows currently held from that method's feeds. The other 3 feeds download nothing: they are filtered views over a table another feed already loads. Example: Nashville planning cases filtered to rezonings, plats and PUDs. In the PDF example a language model reads each report to pull out figures such as the lot count, because they sit in prose rather than in a field.
Supply shows up in public records in a rough order. First comes the early signal: a stormwater notice or a land-disturbance filing, usually months before anything is marketed. Then the entitlement: a rezoning, a plat, a site plan. Last comes the permit. A metro is only well covered when it has all three, and the bar in each row shows its mix.
| Metro | Feeds | Early · entitlement · permit | Records | Active, 8 days | Planning cases | MF/TH projects | Their units |
|---|---|---|---|---|---|---|---|
| ATL Atlanta, GA | 57 | 10 · 30 · 17 | 37,748 | 38 of 57 | 10,944 | 535 | 45,027 |
| AUS Austin, TX | 43 | 6 · 11 · 26 | 38,031 | 40 of 43 | 6,223 | 636 | 90,531 |
| TPA Tampa, FL | 32 | 3 · 17 · 12 | 46,430 | 27 of 32 | 15,058 | 239 | 8,804 |
| HOU Houston, TX | 29 | 5 · 12 · 12 | 61,114 | 24 of 29 | 6,345 | 51 | 3,178 |
| NSH Nashville, TN | 23 | 5 · 10 · 8 | 90,256 | 22 of 23 | 13,567 | 50 | 963 |
| PHX Phoenix, AZ | 23 | 3 · 12 · 8 | 82,031 | 21 of 23 | 26,642 | 112 | 18,486 |
| LAS Las Vegas, NV | 19 | 5 · 6 · 8 | 31,391 | 17 of 19 | 24,744 | 193 | 5,211 |
| CLT Charlotte, NC | 18 | 3 · 10 · 5 | 43,543 | 18 of 18 | 9,343 | 675 | 14,173 |
| HSV Huntsville, AL | 10 | 2 · 7 · 1 | 10,699 | 5 of 10 | 2,968 | 7 | 1,095 |
- Early signal
- Entitlement
- Permit
The last two columns cover only active multifamily and townhome projects from the building-permit layer, with the unit counts the permits state. They are not a metro's whole pipeline: in Houston, Nashville and Huntsville much of it is visible only in plats and stormwater filings, which count toward records, not here.
Planning and zoning cases, by type
Cases, not projects: one subdivision can file a rezoning, a preliminary plat and a final plat. 24,370 of them state a unit or lot count; the rest give acreage or nothing, and I would rather leave those blank than guess.
Active permit projects, by product
| Product | Projects | Planned | Permitted | Unknown | Units |
|---|---|---|---|---|---|
| Apartments | 1,463 | 255 | 724 | 484 | 165,054 |
| Townhomes | 617 | 76 | 517 | 24 | 4,258 |
| Duplex to fourplex | 150 | 13 | 73 | 64 | 322 |
| Condos | 120 | 14 | 22 | 84 | 4,274 |
| Other / unclassified | 148 | 21 | 96 | 31 | 13,560 |
Units are what the permits state, never an estimate from acreage. A project whose permits disagree with each other is counted once, at its largest stated size. "Unknown" means the permit's status field does not say how far along it is.
A scraper that throws an error is easy to notice. The expensive failure is the one that keeps running and quietly returns nothing because a county moved its portal. So every feed is judged by the last time it showed signs of life: the later of its last successful load and the newest record it holds. 212 of 254 were active in the last 8 days. The 32 that were not are a mix of sources frozen on purpose, kept for their history, and portals that have stopped answering. The daily checks flag the second kind.
Feeds by days since their last activity, coloured by stage as in § 02. Most feeds run weekly, so a healthy feed sits in the 2–8 day band. "No date to judge by" is a feed that holds neither a load time nor a record date to measure.
Here is what the rows look like: a stable random draw of 15 named projects per metro with a stated size, all from public filings in the last year. Where several filings have been matched to one project, the record column says how many. Some metros only have single filings to draw from so far.
| Project | Where | Product | Stage | Units / lots | Record |
|---|
Units marked "est." come from acreage and are labelled as estimates wherever they appear. Plats count lots. Blank cells are fields the filing does not state. Las Vegas rows are built up from the individual home permits in each tract.