Building a Local-First Rehab Cost Analyzer with an AI Agent (Full Build + Replication Guide)

Why this post exists

This is both a retrospective and a replication guide. Over a single working session, an AI agent (Hermes Agent, running locally on my Mac) took a vague idea — "estimate rehab cost from public property data, locally" — and produced a working, committed, pushed-to-GitHub application. No human wrote the code by hand; the agent did, and I (the human) steered, reviewed the rendered output, and flagged defects at a craft level.

If you want to reproduce the result, the exact commands, the environment gotchas, the API schemas we reverse-engineered, and the bugs we hit are all here. The repository is public:

• Repo: https://github.com/kliewerdaniel/rehab-analyzer

The thesis underneath it: intelligence is not the model — it's the accumulated, encapsulated decisions that shaped the build. The agent is a vehicle for those decisions; the decisions are what survive.

What the application does

rehab-analyzer answers one question: given an address, what will it cost to rehab this property, and what's it worth after?

1. Geocode the address (Nominatim) → lat/lon.
2. Fetch metadata from the free-tier ATTOM provider — covering year built, square footage, beds/baths, lot size, last sale price, property taxes, and features.
3. Reconcile the record: canonical provider, unit-vs-building scope detection, per-field provenance, and conflict flags (e.g. divergent year_built). With a single metadata source the reconciler notes that no cross-provider comparison is available; it re-engages automatically if a second provider is added. ARV is pinned to the last sale price when available.
4. Estimate rehab cost with a local, OpenAI-compatible LLM (I run gemma4 on :8080 — no cloud, no API bill).
5. Pull real photos of the property from a chain of legitimate sources, so the human sees the actual building, not a synthesized guess.

The governing principle throughout was metadata > pixels: drive every decision from verified provider records; treat photos as supporting evidence, not as the input to a model.

Architecture

The project is a local-first Next.js 14 app (App Router, TypeScript, Tailwind, salmon/teal theme, Framer Motion) backed by a node:sqlite cache. A legacy FastAPI layer is retained for its tested provider adapters and as a porting reference, but the Next.js app is what runs.

rehab-analyzer/
├── nextjs/            # ACTIVE app
│   ├── app/           # /, /history, /history/[id], /api/analyze, /api/history, /api/history/[id], /api/photo/[file]
│   ├── lib/
│   │   ├── adapters/  # attom.ts (metadata-only)
│   │   ├── photos.ts  # getPropertyPhoto() source chain
│   │   ├── db.ts      # node:sqlite (address-keyed cache)
│   │   ├── llm.ts     # local OpenAI-compatible client
│   │   ├── orchestrator.ts
│   │   ├── config.ts  # reads all env vars
│   │   └── types.ts
│   └── components/    # ResultView, ui/*
├── app/               # LEGACY FastAPI layer (tests still green)
└── assets/            # README screenshot

The photo source chain

This is the part that took the most iteration, and it's the most reusable idea here. The app tries sources in priority order and returns the first one that resolves:

Two engineering details matter:

• Runtime photo serving. next start serves public/ from a build-time snapshot, so anything written to public/photos/ after next build 404s. The fix is a dynamic route, app/api/photo/[file]/route.ts, that always reads from disk.
• WebP vs JPEG content-type. Redfin's CDN serves URLs ending in .jpg but returns WebP bytes. We sniff the real magic bytes and store the correct extension so the /api/photo route sends the right Content-Type (otherwise Safari renders blank). This is now applied to every source.

Replicating the build

0. Prerequisites

• macOS (the agent runs in the Hermes desktop app; the app itself is cross-platform but the commands below assume a Mac with Homebrew).
• Node >= 22.5 — the app uses the built-in node:sqlite module. Node 24 via Homebrew is what we used.
• Homebrew, git, gh (GitHub CLI, authenticated).
• A local OpenAI-compatible LLM or the deterministic demo fallback.
• Optional: free-tier API keys (see below).

1. Hermes Agent setup (the part nobody tells you)

The single biggest reproducibility risk is the agent's shell environment, not the code. Hermes runs a Python venv (we had 3.11 on the agent side). When the agent shells out to your own service venvs (py3.14 in our case), the inherited environment bleeds over and crashes the child process mid-request. The poison variables:

PYTHONPATH=/Users/.../hermes/.venv/lib/python3.11/site-packages
PYTHONHOME=/Users/.../hermes/.venv

If you launch a child service (uvicorn, a ComfyUI-style server, a local LLM) from inside Hermes without stripping these, the child pulls the agent's numpy/py3.11 into a py3.14 interpreter and dies. The rule, burned in:

Always export a clean env for any child service the agent starts:

env -u PYTHONPATH -u PYTHONHOME ./.venv/bin/python -m uvicorn app.main:app --host 127.0.0.1 --port 8000

and for Node:

export PYTHONPATH=""; export PYTHONHOME=""
env -u PYTHONPATH -u PYTHONHOME npm run build

This is not optional. It is the difference between "works on my machine" and a silent crash you'll spend an hour chasing.

Also: the agent should never start, stop, or restart the user's LLM. In our setup gemma4 runs on :8080 and is user-launched; the agent only talks to it. Keep the human's inference server sovereign.

2. Scaffold the Next.js app

mkdir -p ~/Projects/rehab-analyzer && cd ~/Projects/rehab-analyzer
npx create-next-app@14 nextjs --ts --tailwind --app --eslint --src-dir=false --import-alias="@/*"
cd nextjs
export PATH=/opt/homebrew/opt/node@24/bin:$PATH   # node:sqlite needs >=22.5
npm install

Add the runtime deps:

npm install framer-motion lucide-react clsx tailwind-merge
npm install playwright        # used at RUNTIME for Redfin, not just dev
npx playwright install chromium

Note: playwright must be a runtime dependency (dependencies), not devDependencies, because getPropertyPhoto() imports it dynamically to fetch Redfin. If it's only in dev deps, production builds can't load it.

3. Provider adapters (free-tier schemas we verified)

Free tiers are metadata-only by design — no photos, no AVM. We reverse-engineered the real response shape for the one provider we use so you don't have to.

Note: the app originally reconciled RentCast + ATTOM across providers. RentCast was later dropped (it 404s for many addresses and adds no free-tier value); ATTOM is now the sole metadata source. The reconciler still handles a single record gracefully — it emits a "Single metadata provider — no cross-provider comparison available" note instead of fabricating a comparison. The reconciliation logic (scope detection, provenance, conflict flags) is intact and re-engages automatically if a second provider is added back.

ATTOM — base https://api.gateway.attomdata.com/propertyapi/v1.0.0, header apikey:

• GET /property/detail?address=<addr> → 200. Structure: summary.yearbuilt, building.size.universalsize.
• /property/snapshot → 200, no photos.
• /property/avm, /valuation, /image, /phototour → 404 on free tier.

The adapter contract is an abstract ProviderAdapter with get_property(address) and health_check(). Each adapter maps the provider's response onto a shared PropertyRecord (metadata-only: avm=null, with data_warnings for missing photos/avm). Reconciliation then detects scope mismatch (unit record vs building record) and flags conflicting fields rather than averaging them blindly; with a single provider it notes that no cross-provider comparison is available rather than fabricating one.

4. Local LLM estimate

lib/llm.ts is an OpenAI-compatible client pointed at your local endpoint:

const client = new OpenAI({
  baseURL: process.env.REHAB_LLM_BASE_URL || "http://localhost:8080/v1",
  apiKey: process.env.REHAB_LLM_API_KEY || "not-needed",  // blank for local
});

The prompt enforces metadata > pixels: the model estimates rehab cost only from the reconciled record, never from a photo. We parse the response strictly — per_category, total_rehab_low, total_rehab_high are required keys, validated through RehabAnalysis.model_validate. A too-lenient parser was an early bug; the strict version is what shipped.

If you don't have a local LLM, set REHAB_DEMO=1 for a deterministic rule-based estimator (cost table by category: kitchen [25,75] per sqft-ish, bath [8000,20000], roof [5,12]/sqft, flooring [4,12], hvac [6000,15000]).

5. The photo pipeline

lib/photos.ts exports getPropertyPhoto(address) which runs the chain in the table above. The Redfin fetcher is the interesting one:

const { chromium } = await import("playwright");
const browser = await chromium.launch({
  headless: true,
  args: ["--disable-blink-features=AutomationControlled", "--no-sandbox"],
});
const ctx = await browser.newContext({
  userAgent: BROWSER_UA,        // Mac/Chrome 124 constant
  viewport: { width: 1280, height: 900 },
  locale: "en-US",
});
await ctx.addInitScript(() => {
  Object.defineProperty(navigator, "webdriver", { get: () => undefined });
  (window.navigator as any).chrome = { runtime: {} };
});
// goto redfin.com, fill search box, submit, wait for listing,
// extract <img src> matching cdn-redfin.com

Redfin showed no verify-human wall in testing — a full branded 404 for a bad deep-link, and a real listing when driven through the search box. We extract the listing photo URLs straight from the DOM (no lightbox clicking needed) and download them like any other source. Classification is heuristic (hero = exterior, rest = interior); labels are advisory.

6. The env file

.env.local (gitignored, never committed):

ATTOM_API_KEY=...
REHAB_LLM_BASE_URL=http://localhost:8080/v1
REHAB_LLM_MODEL=gemma-4-26B-A4B-it-ultra-uncensored-heretic-Q4_K_M.gguf
REHAB_LLM_API_KEY=            # blank for local
REHAB_DEMO=0
GEOAPIFY_KEY=...              # optional, real map+marker
GOOGLE_MAPS_API_KEY=...       # optional, Places + Street View
MAPLLARY_TOKEN=...            # optional
REHAB_REDFIN=1                # default on
REHAB_ZILLOW_SCRAPE=0         # off (ToS)
REHAB_DB_PATH=./rehab_cache.db

Secret hygiene: the agent must never print these keys to logs, never commit .env.local, and treat it as unreadable except via terminal. We keep .env.example tracked and .env.local gitignored.

7. Run it

cd ~/Projects/rehab-analyzer/nextjs
export PATH=/opt/homebrew/opt/node@24/bin:$PATH
export PYTHONPATH=""; export PYTHONHOME=""   # strip agent poison
./start.sh                                  # or: npm run start -- --port 3100

Then curl -X POST http://127.0.0.1:3100/api/analyze -H "Content-Type: application/json" -d '{"address":"311 Cedar St, Seattle, WA 98121"}'.

8. Post-build refinements: CRUD, downloads, hardening

After the first push we hardened the app into something you'd actually keep using:

• Saved-analysis CRUD. The node:sqlite cache gained DELETE and a partial UPDATE (rename label, free-text notes, and a manual rehab-cost override low/high that takes precedence over the model's estimate). A self-migrating schema adds these columns on first run.
• GET / PATCH / DELETE on /api/history/[id]. PATCH validates its body and only writes the keys you send; the UI's history list has inline rename + delete, and the detail page has a notes field and override inputs.
• One-click downloads. Every gallery image and the hero now carry a /api/photo/<file>?download=1 link that sets Content-Disposition: attachment (verified: correct filename + Content-Type). The detail view also offers "Copy JSON" and "Download all".
• Hardened analyze. The route now validates address is a non-empty string capped at 200 characters and returns sanitized errors (no stack traces leak).
• ATTOM-only metadata. RentCast was removed entirely (it 404s for many addresses and added no free-tier value). The reconciler handles a single record gracefully and re-engages cross-provider comparison if a second source returns.

Lessons learned (the real value)

1. Strip the agent's env before launching any child service. This was the most expensive lesson. The PYTHONPATH/PYTHONHOME poison from Hermes's own venv will crash a different-version interpreter. Build the env -u prefix into every launch command. (python-multipart missing and a missing cache dir were the other two launch blockers — both one-line fixes once you know to look.)

2. next start freezes public/. Anything your app writes to public/ after build is invisible. Serve runtime artifacts through a dynamic route. Don't fight the framework; route around it.

3. Trust bytes, not URLs. CDNs lie about extensions. Sniff magic bytes and store the true type. One-line bug, hours of "why is Safari blank" debugging.

4. Free tiers are metadata-only by design. Don't guess the API surface — scrape a reference MCP-server repo or hit the endpoint and read the 200/404 reality. Our verified schemas above are the artifact of that.

5. Reconcile, don't average. Unit-level vs building-level records for the same address disagree in structure. Detecting scope mismatch and surfacing conflicts is more honest than blending them.

6. The agent is a craft-iteration partner, not a one-shot generator. The human reviewed rendered output and gave terse, correct corrections ("interior labels are wrong", "WebP blank in Safari"). The agent's checker is the blind spot; the human's read is almost always right. Root-cause, don't defensively re-verify.

7. Keep the user's inference server sovereign. The agent never starts/stops gemma4. It only calls the endpoint. This boundary prevents the agent from poisoning the user's running services (and, relatedly, from inheriting their env).

8. Two-write publishing. Blog/docs that build from a separate source dir need the file in both places. A draft in the canonical archive never reaches the site. (We applied this discipline to the project README too.)

Current state & what's next

The app is committed and pushed to github.com/kliewerdaniel/rehab-analyzer (public). It runs locally, pulls a real Redfin gallery (most authoritative for the exact building) with Google Places as a fallback, and estimates rehab cost from a local LLM. Beyond the initial build it now supports saved-analysis CRUD (rename, notes, manual rehab-cost overrides), one-click photo downloads, and a hardened analyze route (input validation + sanitized errors). One visible refinement remains: the Redfin "interior" label is a heuristic (floor-plan thumbnails get mislabeled) — worth a follow-up pass.

The deeper point stands: a local-first tool, built by an agent, reviewed by a human, running on your own hardware with your own model. No cloud dependency for the intelligence. That's the shape of sovereign software.