AI & data build notes

When I compare rental-housing deals, “recent rent” needs to mean the same thing on every property. Early chat-based runs kept making slightly different choices about the lookback period, renewals and zero-rent units. The spreadsheets looked reasonable individually. Comparing them was the problem.

I moved recurring work into skills: written instructions an agent reads before running a task, backed by Python tools. I supply the underwriting definitions and decide what a reviewer needs to see. AI helps implement them. The agent can interpret an unfamiliar export, but the calculation should come from an explicit rule that survives the conversation.

The income-statement skill shows the split. A T-12 is twelve months of income and expenses; a T-3 is the latest three. The source-sheet helper adds those three months. Annualizing that result, multiplying by four, happens separately in the analysis. Those are different numbers, and putting both under an ambiguous “T-3” label would make a correct formula misleading.

1. Written workflowDefines the period, categories and expected output.
2. AgentInterprets the file and supplies its layout.
3. Python toolsBuild formulas, write the workbook and check the saved cells.

In the current tools, one rules module produces a plan of cell values, formulas and formatting. The Mac and Windows writers apply that same plan. A check then reads the saved, closed workbook back against it. This gives me a way to distinguish a wrong rule from a writer that failed to put the rule into Excel.

For example, a source-sheet test expects =SUM(P6:R6) for the three-month helper. It checks the literal formula, including its selected columns. A local check with invented January–June headers selected April–June in either chronological direction: 100 + 110 + 120 = 330. That exercises the rule, not a live workbook. Unreadable headers still need attention.

The cost is upkeep. If I change what “recent” means, the workflow, code and expected answer have to move together. A saved specification can preserve a mistake as consistently as a good decision. I still review the definitions; I no longer want to invent them again on the next deal.

An underwriting workbook can calculate a plausible answer from the wrong cells. When AI writes the code that fills it, I need more than the builder’s explanation that everything looks right. I need someone to challenge the assumptions behind the change.

I split the work into planning, building and review. I define the business outcome, constraints and what would count as finished. The planner turns that into a task brief. A builder implements a small change and commits it, saving an exact version. A separate reviewer reads that version and reports possible defects. I resolve the underwriting decisions; the builder must reproduce technical findings before changing the code.

One August review caught a useful example. A formula checker was removing spaces before comparing Excel formulas. Ignoring cosmetic spacing sounds harmless, but spaces inside a quoted worksheet name are part of the address. Using invented sheet names, these references point to different sheets:

='Rent Roll'!A1
='RentRoll'!A1

The reviewer flagged the comparison rule. The builder confirmed it and changed the helper to preserve spaces inside quoted names. Writing the test exposed another problem: two helpers shared a name, so a later definition had replaced the intended one. The fix gave the specialized helper its own name. That second issue came from testing the finding, not from the review itself.

1. BriefOutcome, constraints and expected evidence.
2. Committed changeA fixed version the reviewer can inspect.
3. AdjudicationReproduce the finding, decide, fix and check.

The archived decision record ties that finding to fix 235a1e5. It also states what was not exercised, including the Windows writer. I find that limit more useful than a blanket “review passed.”

A fresh model can still share the builder’s blind spots, even across vendors. Review also costs time and creates another context handoff. I use it to surface claims I can investigate. I remain responsible for deciding whether the evidence is enough to trust the workbook.

One of my analysis tools once returned a number a thousand times too small. An AI-written fix arrived with a passing regression test, a check meant to catch the same bug later. The problem was still present in the path that mattered. The test had learned its expected answer from the code it was supposed to check.

That changed the order I require. Before the builder fixes anything, I want the expected answer derived from the source document or business requirement. Then I want to see the check fail against the broken behavior. Only after that should the implementation change and the same check run again.

Here is a small illustration using invented values. It demonstrates the failure pattern; it is not a replay of the original incident. Suppose a statement labels amounts “in thousands.” A displayed 2.4 means 2,400 dollars. The expected answer comes from that unit label.

# Broken implementation ignores the source's unit label.
def dollars(amount):
    return amount

assert dollars(2.4) == 2400

The proposed change is deliberately small:

def dollars(amount):
    return amount * 1000

assert dollars(2.4) == 2400  # Same expected answer.

A test that instead asserted dollars(2.4) == 2.4 would pass before the fix and reward the mistake. That is why I ask where an expected answer came from, especially when the same model writes both the function and its tests.

The test also has to reach the function used by the workflow. Correcting an unused helper does nothing for the workbook someone receives. For a real fix, I need the failing case connected to that path, followed by an appropriate output check.

Red then green establishes that this check detects this behavior. It does not prove the unit label was read correctly or that downstream calculations are right. My job is to pin the business meaning and ask for evidence at the point where a wrong answer would affect a deal.

I know the trouble two people can cause editing the same underwriting spreadsheet. I still let two AI sessions work in the same code folder: one building a feature, another testing something unrelated. One session cleaned up its experiment by discarding uncommitted changes. That command also erased the other session’s edits.

The cleanup was broader than the task that justified it. Telling agents to be more careful would leave the same shared folder underneath. I changed the working arrangement: each builder gets a Git worktree, a separate working folder attached to its own branch of the repository. A branch alone is not enough if the builders still edit the same files on disk.

1. Builder ARent parser change in folder A, on branch A.
2. Builder BWorkbook check in folder B, on branch B.
3. IntegrationReview each commit, then combine changes deliberately.

The builders share project history, but each has its own current files and pending edits. They commit small, coherent changes so I can give a reviewer an exact version. If both change the same area, integration is where we resolve the disagreement. Separate folders do not make those changes automatically compatible.

For this Blog revision, the builder also worked in its own worktree. That is a checkable application of the rule, not evidence that agents can never interfere. They can still reach other folders if their permissions allow it.

We recovered the earlier incident’s edits from its session record. I was relieved, but I would not design a recovery process around getting that lucky again. A transcript may be incomplete; saved commits are the checkpoints I deliberately control.

The boundary matters beyond files. Two worktrees can still connect to the same database, deployment or running Excel application. Those resources need an explicit owner and coordinated writes. Parallel builders are useful to me when their work can proceed separately; I still combine their changes and approve shared actions deliberately.

A rent roll lists units, rents and occupancy, but the column names vary by property manager. One export says “Actual Rent”; another says “Current Charges.” I need the underwriting agent to recognize the layout without asking a language model to retype every number into a new table.

I split interpretation from copying. A parser reads the source cells into a grid. The model receives headers and column profiles, descriptions of the values each column contains, and identifies roles such as unit identifier, current rent and market rent. Python then reads those cells using the mapping and performs the calculations.

These invented rows make the boundary visible. The mapping below is shortened for illustration, not a recorded model response. Positions start at zero.

Unit	Plan	Status	Lease rent	Market rent
A101	2BR	O	1,800	1,950
A102	2BR	O	1,850	2,000

{"columns": {
  "unit": 0, "plan": 1, "status": 2,
  "current_rent": 3, "market_rent": 4
}, "status_vocabulary": [
  {"raw": "O", "meaning": "occupied"}
]}

The model’s contribution is “current rent is in column 3.” The 1,800 comes from the parsed source cell. In the implementation, assemble.py converts the mapping into the worksheet reader’s indexing and passes it to parse_rent_roll. The model does not need to generate a fresh copy of the rents.

A defined response format is only the first check. The code also rejects column positions outside the table, missing required roles and incompatible roles assigned to one column. If both unit identifier and current rent point to the same numeric-ID column, the reader could mistake a unit number for its rent. That mapping contains perfectly valid integers and still describes the wrong calculation.

For this example, a local run supplied the mapping directly to the current parser and returned rents of 1,800 and 1,850. The range and conflicting-role checks also passed their rejection cases. No live model was called.

The remaining weakness is semantic: the model could swap current rent and market rent while choosing two valid columns. The structural checks would not, by themselves, catch that. Reconciliation against source totals and human review still matter. I can inspect the mapping separately from the arithmetic, which makes it easier to locate the error when a workbook looks wrong.

When an underwriting number looks wrong, my first question is where it came from. A plausible total is hard to investigate if the output only gives me the final answer. I need to see the source, the calculation and the limits of the check.

The deal record carries provenance: information about where a value came from and how it was produced. A directly read field can point to a document and location. A derived field records the inputs it depends on. The review output then exposes those references beside the calculation, instead of making me reconstruct the chain from a model’s explanation.

The current public sample provides a concrete example. Its audit trail identifies the summary’s unit count as derived from the uploaded rent-roll rows. The Rent roll analysis → Audit trail compares the resulting 334 units with the source’s stated 334. That confirms one reconciliation, not every choice made while reading the roll.

1. Source rent rollUnit rows and the export's stated count.
2. Derived countRepeated unit rows are resolved before counting.
3. Review outputThe calculated count and source count appear together.

The same sample explains why checking a total is not enough. A unit can appear twice during turnover. If both rows carry the same market rent, keeping the wrong lease row can leave the market-rent total unchanged while changing in-place rent. A separate row-choice check is needed to catch that error.

The rendered output also defines NOT CHECKED for a total the source never stated. That result must remain visible: there is no source total to reconcile against. It should not be counted as a successful comparison simply because the calculation finished.

For this post, I traced the rendered sample and its provenance fields; I did not independently re-underwrite the original deal. You can inspect the same evidence in the underwriting sample.

A source location makes review possible. It does not prove the selected column has the right meaning, or that the document itself is accurate. My aim is to make the next question specific: which source cell, which interpretation or which calculation needs checking?

When an agent opens an underwriting workbook, it waits for Excel to answer. In an earlier Mac workflow, Excel was stuck behind a dialog. A person at the keyboard could have seen it; the agent only saw a file-open call that never finished. Retrying the same call did not explain what was wrong.

I added a watchdog, a small program that checks the application’s condition. But deciding that Excel looks unhealthy is different from deciding it is safe to quit. The Mac workflow shares an Excel process with other work. Restarting it at the wrong moment could interrupt another workbook or discard an edit.

The current decision function, assess(), checks memory, active writers, CPU activity and open workbooks. Unsaved changes and protected workbooks block a restart. So does an unreadable sensor. Crucially, an unresponsive Excel returns UNRESPONSIVE: the watchdog refuses to force-kill it. A modal dialog can still need a person to dismiss it.

Observed condition	Decision
A writer is active	Refuse restart
Excel does not answer	Refuse restart
An unsaved workbook is open	Refuse restart
High memory, idle, no workbooks	Eligible for a graceful quit

The monitor also needed to show that it had run. An empty log once looked like a dead watchdog, when the checks were correctly leaving active work alone. Its separate heartbeat file now records the latest check time, verdict and reason. WRITER_ACTIVE with a recent timestamp means something quite different from an old timestamp.

For this post, six local checks exercised refusal decisions, the eligible case and heartbeat output using mocked sensors and a temporary file. They never opened, queried or restarted Excel. That is evidence for the decision logic, not a fresh test of the running desktop installation.

This is a lesson from the desktop workflow, not the headless public demo. I want automation to report what it can observe and leave a clear reason when recovery needs me. A monitor that refuses to act can be doing exactly the right job.

The public underwriting demo produces a combined workbook. It was tempting to make that the operational deliverable too: one file, one analysis path, fewer versions to explain. A dependency review changed my mind about that rollout.

Other tools were already reading the existing workbooks. They located files by name, selected specific tabs and searched for exact row labels. Those conventions were interfaces, even though they were never packaged as a formal API. A workbook can look equivalent to a person and still break a program that reads it.

The September 15 review found a particularly awkward case: a reader could open the new workbook and find part of the data, then miss other fields. The demo’s T-12 used labels such as “Net Operating Income” and “Total Operating Expenses” where a consumer expected “NOI” and “Total Expenses.” Partial success is harder to notice than a file that fails to open.

This minimal example uses invented values to show the same kind of exact-label dependency:

rows = {"Net Operating Income": 100}
assert rows.get("NOI") is None
assert rows["Net Operating Income"] == 100

Both labels make sense to an analyst. The first lookup still returns nothing. Changing a visible label therefore needs the same care as changing a field name in a service response: find the consumers, update them or preserve compatibility, then check the full result.

I kept the existing colleague-facing workbooks. That decision separates two changes that had become bundled together: replacing the calculation engine and replacing the files people receive. Shared calculations could eventually feed the familiar workbook layouts, while the public sample continues to serve its own demonstration purpose.

That is the direction, not a completed migration. The current plan still records unfinished engine consolidation. The optional engine path defaults off; its live setting was not verified in the dependency review. The evidence here is the dated dependency review and the local label example, not a claim that a new engine has already replaced the operational writers.

Preserving the files leaves duplication to resolve. It also gives me a smaller next step: verify shared calculations behind the existing outputs before asking every consumer to accept a new workbook. I would rather make that compatibility work explicit than discover another half-populated report after changing the deliverable.

Adding another county to a housing-supply pipeline is partly an integration problem. Its permit portal may use software I already support, but its field names, search options and record types can differ. Reusing the platform reader helps; assuming the entire county behaves like another one does not.

I separate shared portal mechanics from jurisdiction-specific setup. The shared code handles repeated work such as navigating the platform and retrieving records. Each integration supplies local configuration and maps the source’s fields into the pipeline’s common record shape. That is where I need to discover what the portal actually means by a count or a date.

For example, “Number of dwelling units” and “Unit count” might map to the same field. “Number of buildings” should not be mapped there merely because it is numeric. This is an illustrative mapping problem, not a claim that every portal uses those labels. Shared syntax does not remove the need for source-specific interpretation.

The weekly runner gives each jurisdiction its own result. It catches a reader’s Python exception and continues to the other readers, recording which integration failed. The current result key is forced to the module’s jurisdiction name so freshness reporting can join the run to the correct registered source.

1. Platform readerReusable retrieval mechanics.
2. Jurisdiction configurationLocal fields, supported searches and mapping.
3. Per-source resultRecords and an attributable run outcome.

Local stub checks confirm that an exception becomes a failed jurisdiction result without terminating the next call. There is an important limitation: the wrapper marks a non-exception return as successful. It can therefore overwrite a reader’s own failure flag. The existing isolation protects against thrown errors; it does not prove that every apparently successful response contains fresh, useful records.

That limits what I can conclude from a green run. I still need checks on source dates, expected fields and the downstream data. A shared-reader change can also affect several jurisdictions at once. I would test representative source cases and the known exceptions before rolling it out broadly. Reuse makes the next integration smaller, but the county-specific verification remains part of the work.

A public housing filing can have clean fields for its date and acreage while describing the project in a sentence that needs interpretation. My supply pipeline has to handle both. I do not want the presence of a language model to turn retrieval, date parsing and storage into model tasks as well.

The boundary is specific to each path. Ordinary code fetches records and reads structured fields. A classifier can then interpret ambiguous project text and return a dwelling-type label. Document extraction is a separate model-assisted task where the source format requires it. The system has several such paths; my earlier description of exactly one model call per document was too broad.

In the current filing classifier, the prompt asks for a constrained label. Code normalizes the answer into an allowed value, including a fallback that looks for known label text inside the response. That is a real boundary, but it is not strict JSON-schema validation. The model can still produce a wrong allowed label.

Filing description	What needs deciding
New apartment buildings with shared amenities	Whether the description supports multifamily
Residential development and associated infrastructure	Whether there is enough evidence to name a dwelling type

Another path applies a curated term-based override after the base classifier. That enriched path still calls the base model first. The override is not evidence that those records bypass the model or avoid its cost. Reading the actual call order changed how I should explain the design.

I checked the current classifier and evaluation policy, then used local stubs to exercise response handling. No live model calls were made and no current accuracy or cost reduction was measured. Extraction accuracy and classification accuracy also need separate answer keys; one successful score cannot stand in for both.

A model is useful here because descriptions vary, not because every step benefits from interpretation. If a source starts supplying a reliable structured dwelling-type field, I would revisit the model step for that source. Until then, I need to evaluate the meaning of its answer as well as whether the returned label is allowed.

A housing development can appear in several public-records systems before it delivers a single home. Counting those filings is straightforward. Counting the underlying projects is harder, because the agencies may use different names, identifiers and dates for the same development.

I group supported matches under a canonical project, a common identity that links the source records. The records remain available as separate signals. That preserves the distinction between an early filing, a later approval and another agency’s record instead of replacing all of them with one unexplained row.

This invented example shows the intended relationship. The names and records are illustrative, not a real development or a measured match result.

Source record	Name on filing	Signal retained
A	Juniper Crossing	Early environmental filing
B	Juniper Crossing Phase 1	Subdivision approval
C	Juniper Residential	Permit record

Similar names are a starting point, not enough evidence by themselves. The current Phoenix canonicalization code combines normalized names, location and county context to build groups; source identifiers stay with the individual records. I need to inspect both false merges, where separate developments become one, and missed matches, where one development remains duplicated.

Phases make that tradeoff concrete. Two nearby filings with the same developer might describe separate phases whose unit counts should not be collapsed. Conversely, a renamed project can look unrelated even when its site is the same. There is no universally safe rule that “nearby means identical” or “different name means different project.”

The checked evidence for this post is the current grouping implementation and synthetic local examples. No matching-accuracy benchmark or live project audit was run. The current canonical identifiers are also rebuilt with the grouping output; I should not promise that they are permanent external IDs.

That last detail matters to integrations. A client saving a canonical ID forever would need a stronger identity contract than a dashboard that reads the latest grouping. For my supply comparison, I want the common project view and its retained sources together, so I can challenge a surprising count without losing the evidence that produced it.

A public filing can tell me that a housing project exists without telling me how many homes it contains. Some records report lots or units. Others give only disturbed acreage, the land area affected by construction. Treating those records as equally precise would make the supply comparison look more certain than its sources.

My published methodology distinguishes source-reported counts from estimates. When acreage is the available input, the estimate multiplies that area by an assumed density for the dwelling type. The dashboard also caps an acreage-based estimate at 300 units: 80 acres at an illustrative six units per acre produces 300, not the uncapped 480. The result depends on those rules. It is not a count somebody measured on the site.

These invented records show the three cases I want a reader to distinguish. The density below is illustrative, not a recommended planning assumption.

Source evidence	Displayed interpretation
The filing reports 72 proposed units	72, source-reported
12 acres × an assumed 4 units per acre	48, estimated
No usable count or acreage	Count unavailable

A missing count must not quietly become zero. Zero says there are no units; unavailable says the record cannot support a count. Likewise, a reported proposed count is not evidence of completed delivery. The project can change after the filing, or never reach construction.

The checked dashboard code carries an estimate flag into the “(est.)” badge, and its CSV export includes the count basis. The public methodology explains the density calculation but omits the cap. It is not a fresh audit of every live dashboard row or a calibration study comparing estimates with eventual deliveries.

There is useful sensitivity in making the assumption explicit. In the example, changing density from four to five units per acre moves the estimate from 48 to 60 without any new source evidence. A reader who sees only “60 units” cannot tell that the change came from an assumption.

I use estimates to screen where to investigate, then verify a count that matters to an acquisition decision. The estimate extends what I can compare, but I need its basis to remain visible when the number leaves the source record and enters a chart or export.

A leasing website can advertise one software platform while loading its availability from another. That matters when I collect rent comparables: recognizing a vendor’s name on the page does not prove that its reader can retrieve the rents.

My scraper separates those two questions. It first fetches the page and finds the platform readers whose detection rules match. It then tries those readers in priority order. A reader that returns neither rent nor availability does not prevent a later matching reader from trying. The model fallback comes after that chain.

That is a different contract from “detect a platform, then scrape it.” Detection is a list of candidates. Successful extraction is the evidence that a candidate worked for this page. A usable group can contain rent or availability; that does not establish that its prices are correct.

1. Two readers matchThe page contains signals for both platforms.
2. The first returns nothingThe scraper continues to the next candidate.
3. The second returns rentsThe result is accepted without using the model fallback.

The local dispatch tests supply a page with two vendor signals and confirm that both candidates survive in priority order. They also check fetch-failure diagnostics. The continuation loop was inspected in source; the full scraper was not executed. They do not measure how often a real leasing site is recovered by the second attempt.

There is also a reporting limit worth keeping visible. A failed page fetch and a page with no matching platform are distinguished in logs, but both can return an empty list to the caller. Someone looking only at that return value cannot tell which happened. A richer result could carry the failure reason through the same interface as the rents.

Trying another reader costs time, and shared detection rules can become stale when a vendor changes its site. I would measure recovery and latency before adding more attempts. For now, the useful decision is to let an observed extraction result settle the choice of reader, instead of treating a platform hint as certainty.

A rent feed can show the asking price while the leasing page advertises several weeks free. For an acquisition comparison, those are parts of the same offer. Ignoring the banner overstates the advertised cost; applying it to every unit can understate it.

I keep the promotion’s original wording alongside the structured rent data. The concession scanner recognizes supported free-rent language and separates it from dollar credits, waived fees and other offers. Net-effective rent means the monthly cost after spreading an eligible concession over the lease term. It requires more than spotting a number beside the word “free.”

The parser deliberately leaves a dollar credit as text. It does not turn that credit into a number of free weeks, even when a monthly rent is available. That would introduce assumptions about the lease and the credit’s conditions at the parsing step.

Banner	What can travel forward
Six weeks free	A supported free-rent duration and original text
$500 off your first month	Original text; no invented free weeks
Special on selected floor plans	An eligibility question that still needs resolving

Local tests exercise the dollar-credit boundary with and without a supplied monthly rent: the scanner keeps the wording and returns no free-week value. These are parser checks using synthetic offers. They do not establish that a promotion is available for a particular apartment today.

Eligibility is still the harder part. The current code can treat an offer naming multiple bedroom types as community-wide without confirming the property’s full bedroom mix. That is an imperfect shortcut. A property with another, unmentioned floor plan could receive an overly broad discount. I would need to join the restriction to actual inventory before claiming complete unit-level coverage.

I want the comparison to preserve the asking rent, the promotion and the basis for any adjustment. That lets me inspect an appealing net-effective number before treating it as a rent comparable. Some offers remain unquantified, which is more useful to me than silently filling in terms the page never gave.

When I compare rents, I often care about the low end of the available inventory. A community’s advertised minimum and maximum cannot tell me the average of its lowest five observed rents. They have already discarded the prices in between.

That distinction changed the data contract, the shape of the record each reader must return. Readers preserve actual unit observations, including repeated prices. If two available homes both ask the same rent, those are two observations. Removing duplicates would change the weight of that price in the comparison.

Consider these invented sets of five available homes. Both have the same advertised range, but they produce different averages.

Community	Observed asking rents	Average
A	1,800; 1,800; 1,800; 1,800; 2,200	1,880
B	1,800; 2,200; 2,200; 2,200; 2,200	2,120

Both ranges say 1,800–2,200. Creating a fake observation at each endpoint would make the two communities look identical and invent a sample size. It would also erase the repeated prices that explain the difference.

The tracker therefore leaves the observed-rent list absent when a source supplies only a range. A range-only record can still tell me an advertised starting price or broad price band. It cannot support the same calculation as a list of available units. The current comparison falls back to the advertised minimum when observations are absent. That is a different basis from an observed average and should be read that way.

The local contract checks cover preserved repeated observations and the separation between range fields and observation lists. The small example above is synthetic arithmetic, not scraped inventory. It demonstrates why the distinction matters without pretending to measure a market.

The model-output guard catches a particular two-endpoint reconstruction, not every invented distribution. Observed rents have their own limits. Availability can change after collection, and an advertised unit is not a signed lease. I still need collection dates and source context. But when the input really is a set of observed units, I can explain what went into the benchmark; when it is only a range, the record tells me that too.

A rent observation gets older even when its collection job runs exactly as designed. Once markets follow different collection cycles, “nothing arrived this week” can mean a missed run or an intentional off-week. A monitor that ignores the schedule cannot distinguish them.

I put the cadence rules in a shared module used by collection and freshness checks. Cadence is simply when a market is due. The runner uses that decision to select work; the monitor uses it to judge whether the missing work is actually late. Keeping separate copies of the schedule would create another place for the two systems to disagree.

One detail is easy to miss in the language: every four weeks is not once per calendar month. Four-week intervals repeat after twenty-eight days and drift through month boundaries. Calling that “monthly” in a monitor can produce an incorrect expectation even if the collection code has no bug.

Schedule and run evidence	Interpretation
The market is not due	An off-week is expected
Due, an attempt failed	A known collection failure
Due, no run started	A missing scheduled attempt

The local schedule tests use fixed dates rather than the wall clock. That makes week selection and boundary cases repeatable. Freshness checks can then ask what should have run on that date before inspecting what actually ran. A test that merely expects “recent data” would miss the product decision encoded in the collection interval.

This still leaves two separate concerns. A scheduler can correctly choose a market and its reader can return stale or empty data. Conversely, valid older observations may be the intended output of a slower schedule. Job timing does not establish source quality, and a green scheduling check should not imply that it does.

A longer interval reduces collection frequency while giving me older comparables between runs. That is a tradeoff I need to choose for the investment question. Shared schedule logic makes the expectation consistent; it does not make an infrequent observation fresh.

A housing-type label is an intermediate result in my supply pipeline. It eventually affects which projects appear in a market comparison. Before changing the classifier, I needed a way to tell whether its new answers were better.

My first evaluation used labels the pipeline had already filled in as the expected answers. That measured agreement with an earlier output. It could detect a change, but it could not establish that either answer was correct. A stable mistake would look like success.

I rebuilt the answer key around evidence outside the classifier’s predictions: source documents, developer material and separate permit records. A golden case is a saved input paired with an expected answer and a reason to trust that answer. The comparison itself can be ordinary code. The difficult judgment is deciding what the source actually supports.

Input	A filing describing an apartment building
Expected label	Multifamily
Independent support	A separately reviewed permit document
Model prediction	Kept separate until comparison

The current evaluation structure distinguishes representative cases from collections built around known failures. Both are useful, but they answer different questions. A targeted set tells me whether a particular weakness remains. It cannot tell me how often that weakness occurs across all incoming filings. Mixing those cases into one headline score would hide the sampling choice.

The source review checked the evaluation policy and stored-case structure. It did not independently reread every underlying document or run a new model evaluation, so I am not presenting a current accuracy score. A field saying that a label was reviewed is a record of a process, not a substitute for the supporting evidence.

An answer key can also contain mistakes. If a source is ambiguous, forcing a specific dwelling type into the expected answer would reward the model for guessing. I need an unknown option and a review trail for changed labels. When a score changes, I want to know whether I changed the model, the sample or my definition of the correct answer.

In a June incident, my supply dashboard kept showing plausible numbers while the data behind them was frozen. Ingestion ran, the weekly job finished and the surrounding health checks reported success. A migration had made a database view copy its own stale snapshot instead of refreshing from the underlying records.

That is why model evaluation could not catch this failure. The classifier did not have to give a wrong answer. The error was farther downstream, between stored records and the numbers the dashboard served. I needed to check that part of the journey directly.

I added a comparison between the newest relevant source date and the newest date in the served view. The comparison needs matching source semantics: a late-stage permit feed and an early filing feed can legitimately have different dates. It also needs to handle missing data and implausible future dates, rather than letting either produce a reassuring result.

The database assertion below is the actual checked-in query. It returns violations; no returned rows means that this particular assertion found none.

select metro, view_latest, arm_latest, lag_days, status
from observability.dashboard_view_freshness
where status is distinct from 'ok'

Imagine new source records reaching June 8 while the dashboard remains at May 25. Those dates are synthetic. A successful-job check can still pass because the refresh completed; a comparison of served and source dates can expose that it refreshed the wrong thing.

The first monitoring pass also produced an alarm that was too sensitive. I had to distinguish a real cross-market contamination problem from normal repeated names. That adjustment is recorded in the dated migration. I inspected the incident receipt and assertion code; I did not rerun production queries or reconstruct historical logs.

A later September review found another blind spot: ingestion could continue while classification failed. That did not invalidate the dashboard freshness check; it exposed a different condition the check did not cover. I cannot turn a successful check of one stage into a health claim for the whole pipeline.

I now ask what evidence a green indicator actually examined. Process completion, model quality and the correctness of the number on screen need separate checks. When the dashboard is wrong, I have to follow its data path, even if every nearby job says it finished.

My supply classifier distinguishes single-family homes, townhomes and multifamily buildings. Two wrong labels can count equally in an accuracy score while affecting different parts of the comparison I use. I wanted the evaluation to expose that difference without hiding the underlying mistakes.

The comparator reports exact-label agreement and a second, task-specific grouping. In the current grouping, single-family and townhome sit together; multifamily sits separately. A single-family prediction for a townhome fails the strict check but agrees at the grouped level. Calling that same townhome multifamily crosses the grouping boundary.

That grouping is a pipeline convention. It is not proof of ownership or rental tenure: single-family homes and townhomes can be rentals, and building form alone cannot establish whether units are rented or sold. My earlier explanation blurred that distinction. Even a single-family versus townhome error can affect a form-sensitive density estimate. The second score examines a chosen reporting boundary; it does not make that error harmless.

Expected → predicted	Strict result	Grouped result
Single-family → townhome	Mismatch	Agreement
Single-family → multifamily	Mismatch	Mismatch

Unknown answers need another view. If the source cannot support a dwelling type and the model supplies one anyway, the problem is unsupported certainty. If the source supports a type but the model leaves it unknown, the problem is missed information. Those failures can need different fixes even though neither supplies the desired answer.

Local, synthetic comparator checks exercise these distinctions. They verify how cases are scored; they do not measure current model quality or attach a financial loss to an error. I have not built a measured dollar-cost model for these mistakes.

I keep the strict score visible because a business grouping can make a model look better by forgiving distinctions that another user needs. A tool comparing building forms for construction planning might care deeply about the townhome versus detached-home distinction. The grouping is a question I bring to the person using the output, and I would revisit it when the decision changes.

Model calls happen in several parts of my underwriting and data workflows. A total bill tells me that money was spent. It does not tell me which feature spent it, which calls failed or whether a change made one step slower.

I use a shared call wrapper to record system and feature labels alongside usage, latency and errors. A wrapper is a common entry point around the model request. It gives calls from different tools the same reporting shape without requiring every tool to invent its own logging format.

The important operating decision is what happens when that reporting breaks. If primary telemetry storage fails, the storage router tries a local SQLite database. If both destinations fail, it emits a warning and drops the record. A failure to store a metric should not, by itself, discard a useful model response or interrupt the main workflow.

Failure	Response
Primary metrics write fails	Try the local fallback database.
Both metric stores fail	Warn and lose the record; preserve the call's result.
A budget breach is recognized	Stop the request before making the model call.

The budget check is separate because its purpose is to prevent spending. A recognized budget breach intentionally raises an error that stops the call. But an unexpected failure while checking the budget can fail open, allowing the request to proceed. That distinction matters: this is not a guarantee that every spending limit survives every infrastructure failure.

Ten local storage tests passed using fake primary writers and temporary SQLite files. They covered primary success, local fallback and the case where both writes fail. These checks exercise storage behavior without contacting the model provider or the production database. A separate stubbed call check covers the recognized-budget-stop path.

Keeping work running through a telemetry outage leaves gaps in the very dashboard I use to understand it. Local fallback helps, but I still need to notice missing records and inspect warnings. I can choose to tolerate a lost metric more readily than an unnecessary failed underwriting run; I should not then mistake the remaining cost chart for a complete accounting of every call.

A broker’s income statement passes through several of my tools before anyone prices a deal off it. One reads the file, another writes the underwriting model, and a third decides whether the pack is ready for review. A test at the first step cannot tell me whether the last tool read the right number.

In my September retrospective, a separate AI reviewed a month of bug records across seven projects. About one in four fixes were reworking an earlier fix or repairing something a fix had broken. Some were caught before anything shipped, so that is a rework rate, not a count of production failures.

I already asked the AI to write a failing test before fixing a bug. But those tests often stopped at the function being repaired. They did not follow the saved file into the next tool. Expected answers were also often copied from what the code produced that day.

I started calling the broader check a chain contract: a test that follows a source through the real processing code, the saved output and the tools that consume it. Its expected answer comes from evidence outside the code being tested. For an income statement, that includes the statement’s own printed totals.

This synthetic example shows the gap:

Statement's printed annual income: $120,000

Parser reads                       $120,000
Saved analysis contains            $120,000
Model reader returns                     $0  <- contract fails

Checking the parser alone would miss that failure. Deriving the expected answer from the model reader’s own incorrect output could also produce a passing test.

The contract uses scrubbed source files that retain their layouts, runs the real code, then saves and reopens the output. It checks the named downstream readers against independently recorded expectations. Coverage stops at those selected layouts and readers; it does not establish that every possible statement works.

For deal review, I want to trace an actual in the model back to the statement that supplied it. This gives me a way to test that path before using the workbook.

My underwriting and supply tools rely on live databases, including the queue where I track bugs. One test wrote fake rows into that production queue. It passed because its assertions checked the result, not whether it had reached a live service.

Other tests had crossed the same boundary: a scratch script ran a rewritten database change against production, and a unit test started a paid scrape of leasing sites. I had treated tests as a safe place to experiment without making that separation real.

I added a guard that loads before the application code in Python tests. It blocks the covered network, database and model API paths and removes credentials, including ones a settings file tries to restore during a run. Tests that need a database use a disposable one built from the project’s schema files.

A guard needs its own evidence. Each project replays an old incident against a harmless stand-in for production. The unguarded run has to reach the stand-in; the guarded run has to stop before it does.

This synthetic replay illustrates the two checks:

Controlled replay	Required result
Guard disabled	Stand-in receives the attempted operation
Guard enabled	Isolation error; stand-in receives nothing

An error by itself would not prove the guard worked. The script might have crashed before attempting the operation. The unguarded run establishes that the test can reach the point the guard is meant to stop.

Review also found a route around the guard when a proxy was configured. That was a useful reminder to check how a library connects, not just its usual path.

I can now replay those incidents without borrowing the live systems I depend on for deal work. The scope still matters whenever I add another way to connect.

The T-12 column in an underwriting model puts the property’s actual trailing income and expenses next to the broker’s pro forma. In one run, that column shipped as zeros and three readiness checks passed.

The generated T-12 analysis contained formulas, but the saved workbook did not contain their calculated answers. My reader received a missing value and converted it to zero. The model then displayed zero as an actual.

Rejecting every zero was also wrong. A property can genuinely have no pet income. I needed to preserve two separate facts: the amount, and whether the reader successfully obtained it.

This illustrative Python shows the distinction using synthetic input. The dictionary is a simplified example, not the production schema:

raw_value = None  # Formula's calculated answer is missing.

# This shortcut loses the distinction:
amount = raw_value or 0

# Preserve whether the value was read:
if raw_value is None:
    result = {"state": "unreadable", "value": None}
else:
    result = {"state": "read", "value": raw_value}

The full read also needs to distinguish a missing field:

Synthetic source condition	Meaning
Saved numeric value is 0	Successfully read zero
Formula exists; calculated answer is missing	Unreadable
Expected field is missing	Absent; apply its required or optional rule

Four sanitized statements with different layouts now pass through the real builder, saved workbook and downstream readers through to the audit verdict. Expected totals come from the statements’ printed totals, never the parser’s output.

Building that test found another problem: rebuilding a model without a T-12 left the previous run’s actuals in place. That column is now cleared. I also ran the new readiness check over about sixty current deal folders before merging; no verdict changed.

I want every actual in that column to be traceable to the statement or visibly missing. The test checks that distinction within the layouts and readers it covers.

Before new deal-pack code runs on real deals, I want to know which checks ran on that version. In September I found two different gaps in that evidence.

For twelve days, the hosted test service refused every job before execution because of an account billing limit. Those runs visibly failed, but nothing escalated them. Separately, my local runner could summarize a skipped test as though it had passed. A failure without follow-up and a misleading summary both left me unable to establish what had been checked.

I added a release record tied to an exact code revision and platform. Each project keeps a list of required tests. An exclusion must name a tracked issue that owns removing it. The record step checks out that revision fresh, runs the list and stores the result in a database table that only accepts new rows.

The first real record was refused. Every required file but one had passed; the last had announced a skip in its output. Now an unapproved skip leaves the revision unverified, even when every test that actually ran passed. Refused attempts are stored too.

This synthetic record shows the distinction:

Revision: example-revision
Platform: Windows

Workbook reader       PASS
Model writer          PASS
Source-to-model test  NOT RUN — dependency missing

Verdict: REFUSED

The platform is part of the evidence. A passing Mac result does not establish that the same check works with Windows dependencies or Excel automation. I need a record from the machine that will run the work.

That leaves an operational gap to close. For now, I have a record that names what failed or did not run, instead of a summary that can make missing evidence look reassuring.

My supply database feeds the market counts I use in deal review. Zero new records can be valid: perhaps no new permits were filed. But zero records after every request was denied means the loader never established what was available.

I had already fixed a loader that reported success when every request failed. The fix, it turned out, had landed in code that no longer runs.

I tested the active path with recorded source responses running through the real loaders into a disposable database. Then I replaced the responses with access-denied errors. Roughly a third of one family of sources and about a fifth of another still reported success with zero rows.

This synthetic comparison describes the distinctions I want to demonstrate:

Source response	Rows	Required interpretation
Successful response, no records	0	Valid empty result
Every request denied	0	Collection failed
Some requests denied	Some	Incomplete coverage; not an unqualified success

The partial-failure row is a rule to test, not a claim that every loader already implements the same behavior.

The broader contract found five paths that could report success despite failure; those fixes are deployed. Nine historical fixes are also saved as mutations: tests deliberately restore an old bug in an isolated copy and require the relevant check to catch it for the right reason. A crash somewhere else does not count.

The contract also checks the monitors. One had remained in an error state for 34 days. Replaying that history against a new rule, which flags two consecutive errors, raises the alert on day three. That is replay evidence, not a claim that the original incident was caught sooner.

When a market shows no new supply, I need to know whether collection succeeded before interpreting the count. These checks give me evidence for that distinction, with coverage limits still to resolve.

When I look at an acquisition, I ask questions such as “what competing construction is near this address?” My database organizes records by source and stage. An AI client should not have to rediscover that table layout each time it helps me investigate a property.

I exposed named tools through MCP, the Model Context Protocol. MCP gives the client a structured way to discover and call those tools. The supply logic still lives in the server and database. The model chooses a tool and its arguments; the server performs the query and returns records for the model to explain.

For example, permits_near takes coordinates, a radius, a lookback period and a result limit. This is an illustrative argument set, not a recorded live request:

{
  "lat": 33.4,
  "lng": -112.1,
  "radius_mi": 3,
  "days_back": 180,
  "limit": 10
}

The implementation first asks the database for candidates inside a rectangular boundary around the point. Python then calculates their distance and removes candidates outside the circle. The rectangle narrows the query; the distance check handles its corners. Returned rows carry a common set of project fields plus distance_mi, so the client can compare results from different sources.

That mechanism is visible in the current server code. Local checks of the normalizer and distance helper use synthetic records and points; they do not establish current database coverage or authenticated client connectivity. A computed distance is also only as precise as the coordinates behind it.

I prefer this interface to asking the model to compose a new database query from a long schema description. The argument names expose the choices I expect a user to make, and I can inspect those choices when an answer looks surprising. A radius of three miles and a lookback of six months describe a different question from the same point over five years.

The constraint is that a named tool has a defined scope. If users repeatedly need a new comparison, I have to decide whether it belongs in a parameter or needs a separate tool. MCP makes the interface available to the client; it does not design that interface or make the underlying coverage complete.

A supply query can be read-only and still expose information to the wrong caller. Once my MCP server is reachable from a remote AI client, I need a way to grant access, recognize later requests and withdraw that access.

The server implements an OAuth sign-in flow. A client receives an authorization code after approval, exchanges it for a token and includes that token on later requests. The business tools remain separate from this authentication state: querying projects is one responsibility; deciding whether the caller may query them is another.

The code exchange uses PKCE, which binds the exchange to the client that initiated it. The client starts with a random verifier and sends its hashed challenge. When redeeming the code, it supplies the verifier; the server checks the match. This makes an intercepted authorization code insufficient on its own. The implementation uses the S256 challenge method described in PKCE’s specification.

1. Verify the tokenCheck its signature and expiry.
2. Check current access stateFind the issued token in the registry and reject revoked access.
3. Allow the tool requestProceed only after those checks succeed.

The registry check matters for revocation. A token can still have a valid signature and remaining lifetime after I withdraw access. Checking its current status on each request lets the server reject it before that lifetime ends. The same path refuses unknown tokens and database-check failures instead of treating an unavailable access check as approval.

Local PKCE checks used synthetic challenges and verifiers, including mismatched, short and unsupported-method cases. The token-expiry and revocation branches were inspected in source, not executed against a live client. This is evidence for selected implementation paths, not an end-to-end compatibility test or security audit. The MCP authorization specification describes the broader protocol requirements.

That extra state brings operational work: token lifecycle, database availability and useful client errors. Read-only tools reduce what an authorized caller can change, but they do not remove the need to control what the caller can read. I need both parts when moving a useful local tool into someone else’s client.

An uploaded deal needs local demand, competing supply and rent comparables. Those datasets already exist in my other systems. Rebuilding their query logic inside the underwriting upload container would create another version to maintain and another place for the results to diverge.

I connected the upload workflow to one authenticated HTTP report endpoint. The client requests JSON from /report, then a mapper translates the response into the deal record the workbook writers already use. This is a web-service integration, not an MCP tool call. The upload container needs the service credential, rather than direct access credentials for each underlying database.

One small mismatch showed why that mapping layer matters. Python dictionaries can use integers as keys; JSON object keys are strings. A report organized by numeric radius can cross the network successfully and still fail a downstream lookup that expects the integer 3.

This shortened example illustrates the conversion with invented data:

# A radius-keyed section after JSON decoding
received = {"3": {"population": 12000}}

# Restore the numeric-key contract expected downstream
mapped = {int(radius): values
          for radius, values in received.items()}
assert mapped[3]["population"] == 12000

The actual mapper handles the report’s expected sections and restores numeric keys where required. That lets the writers keep their existing input contract. Successful JSON decoding alone would not detect the mismatch: the response is syntactically valid in both forms.

The local market-client tests exercise a fake HTTP service and synthetic responses, including this key conversion and failure handling. No live market endpoint or production database was queried. Those checks show that the client and mapper handle the fixtures; they do not prove that the deployed service is reachable or that its market data is fresh.

The request also has a bounded wait. If the service or a required module is unavailable, the affected output carries an absent reason instead of fabricated market values. An integration can fail partially, so I need that reason at the module level rather than assuming one successful response means every section was supplied.

Reusing the report service keeps the market logic in one place, but adds a network dependency to the underwriting run. I can inspect and test that boundary separately, which is useful when a workbook is complete except for one missing market section.

“No projects found” sounds like a market conclusion. It can also mean that a tool does not expose the relevant source for that market. If an AI assistant turns both situations into the same sentence, a technical coverage gap becomes misleading investment advice.

I made some of that coverage information part of the supply tool’s response. When find_subdivisions receives a metro whose early filing layer is not connected to that tool, it returns an explanation and suggests other tools. The data may exist elsewhere in the database. The limitation is the path the client tried to use.

Here is a shortened description of that existing behavior. It is explanatory text, not a universal status schema used by all tools:

[{
  "info": "This metro's early radar is not wired into this tool. Use the permitted-project or entitlement tools instead."
}]

The distinction matters because an empty search result is a different claim. A supported query that runs successfully and finds no matching rows has examined something. An unavailable layer has not answered the question at all. A client should preserve the latter limitation when it explains the result.

Geographic precision is another part of the answer. Some permits_near records carry geocode_precision, indicating whether the location is exact, approximate to a street or approximate to a ZIP area. A distance calculated from a ZIP-level location should not be presented as a precise measurement from the development’s entrance.

Local stub checks verify the explanatory coverage branch. Source inspection confirms the precision field in geographic results. They do not demonstrate that every AI client repeats those qualifications correctly. The contract is also incomplete: some unsupported source selections still return an empty list. I cannot describe the response format as consistent across the server.

A next improvement would be explicit, consistent statuses for coverage limitations, successful empty results and failed queries, followed by tests of the client’s final answer. That would make the distinction easier to check than prose alone. For now, I want the tool’s limits to travel with its records, and I still inspect whether the assistant’s summary kept them.