Source dictionary and freshness controls

The Canada ETL fails instead of inventing a fallback exchange rate when the official API is unavailable.

Official Bank of Canada Valet API observation used only for CAD-to-USD display conversion.

.github/workflows/monthly-data-refresh.yml · scripts/loadCanadaData.py

Downloaded monthly; unemployment is not a daily metric.

Official BLS public time-series file.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Monthly workflow probes the latest released quarter and broadcasts county job growth to ZIPs.

Official BLS public QCEW aggregate employment data.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Monthly workflow probes the latest quarterly BLS QCEW NAICS 236117 file and writes aggregate establishment rows plus data_source_runs audit records.

Official BLS public QCEW aggregate NAICS 236117 establishment data. Not named-builder, community, lot, owner, or contact data.

.github/workflows/homebuilder-establishments.yml · scripts/ingestBlsQcewHomebuilderEstablishments.mjs

Monthly workflow checks the latest annual file; values should not be interpreted as daily permits.

Official Census BPS file. Current OpenRoof load uses annual state files.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Monthly workflow probes the latest Census county components file and maps county ratios to ZIPs.

Official aggregate Census county population-estimates components; no individual health records.

.github/workflows/monthly-data-refresh.yml · scripts/backfillBirthDeathRatio.mjs

Discovers the current official CFPB CSVs weekly, writes aggregate delinquency rates, and fills missing aggregate distress scores from those rates.

Official CFPB aggregate delinquency CSVs from Mortgage Performance Trends. No ATTOM, no paid property-data API, no individual borrower data.

.github/workflows/cfpb-mortgage-performance.yml · scripts/ingestCfpbMortgagePerformance.ts

Uses the latest two-bedroom row/apartment 3+ unit observation; the source is annual.

Statistics Canada table 34-10-0133-01, sourced from the CMHC Rental Market Survey.

.github/workflows/monthly-data-refresh.yml · scripts/loadCanadaData.py

Registry sources carry their own cadence. Daily sources are handled by the daily public-record workflow; weekly/monthly sweeps cover the rest.

Official/public records only. No ATTOM, no paid property API, no login/CAPTCHA bypass.

.github/workflows/ohio-distress-registry-ingest.yml · scripts/ingestDistressPublicRecords.ts

Paused behind CREA_LEGAL_APPROVED; obtain and document a commercial data-use agreement before re-enabling.

CREA prohibits commercial distribution without prior written authority. Do not ingest until OpenRoof has written permission.

scripts/loadCanadaData.py

Keeps genuinely daily public sources current without hammering annual/monthly county lists.

Runs only registry sources marked daily. Still no search/login portal scraping.

.github/workflows/distress-daily-public-records.yml · scripts/ingestDistressPublicRecords.ts

Runs after public-record ingests and once daily as a safety net.

Derived from already-ingested public-record facts.

.github/workflows/distress-materialization.yml · scripts/materializeDistressCache.ts

Monthly workflow checks the latest FEMA NRI county service and broadcasts county risk scores to covered ZIPs.

Official FEMA public aggregate hazard-risk data. No ATTOM, no paid property-data API.

.github/workflows/monthly-data-refresh.yml · scripts/backfillFemaNri.mjs

Downloads state/metro aggregate mortgage-performance rates and uses them only as aggregate fallback distress coverage.

Official FHFA National Mortgage Database aggregate statistics. No ATTOM or paid property-data API.

.github/workflows/nmdb-mortgage-performance.yml · scripts/ingestNmdbMortgagePerformance.ts

Downloads state-level outstanding mortgage balances, mark-to-market LTV, and underwater share. ZIP/county/metro equity values are modeled from the state baseline by the monthly ETL.

Official FHFA National Mortgage Database outstanding mortgage aggregate statistics. No borrower-level data, no ATTOM, and no paid property-data API.

.github/workflows/nmdb-mortgage-equity.yml · scripts/ingestNmdbMortgageEquity.ts

Used as a supporting input to affordability and mortgage-payment composites.

Official FRED endpoint using FRED_API_KEY.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Cached in December when available; county annual values are broadcast to covered ZIPs.

Official IRS SOI aggregate migration files; no personal/skip-tracing data.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Monthly workflow checks the latest released presidential county dataset and assigns county shares to ZIPs.

Harvard Dataverse CC0 aggregate county returns; no individual voter records.

.github/workflows/monthly-data-refresh.yml · scripts/backfillElectionVoteShare.mjs

Runs the complete live registry monthly so lower-cadence sources and any item missed by a narrower schedule remain self-healing.

Full registry safety sweep over official/public records only. No ATTOM, no paid property API, no login/CAPTCHA bypass.

.github/workflows/distress-ingest.yml · scripts/ingestDistressPublicRecords.ts

Monthly workflow assigns each ZIP to the nearest NOAA station with annual mean-temperature normals.

Official NOAA/NCEI public climate normals; no scraped weather pages.

.github/workflows/monthly-data-refresh.yml · scripts/backfillNoaaClimateNormals.mjs

Runs after monthly source loads and FEMA NRI refresh; values are suppressed in the UI when confidence is too low.

Deterministic forecast derived from source-backed aggregate market signals. Not an appraisal, AVM, lending, insurance, or FCRA decisioning model.

.github/workflows/monthly-data-refresh.yml · scripts/backfillForecast12mo.mjs

Recomputed after monthly source files are loaded.

Derived from OpenRoof source-backed metrics; not a third-party data source.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Runs after main ETL in the monthly workflow.

Derived from source-backed Realtor, Redfin, distress, and Census/BPS fields.

.github/workflows/monthly-data-refresh.yml · scripts/backfillPhase2Composites.mjs

Runs after main ETL in the monthly workflow.

Derived/model proxy values. Must be labeled as proxy, not observed facts.

.github/workflows/monthly-data-refresh.yml · scripts/backfillPhase3Proxies.mjs

Processes a bounded daily batch, persists progress throughout the run, assigns ZIPs to address-only public-record rows, and rebuilds affected ZIP fact rows.

Uses the official public Census geocoder for address-to-ZIP attribution. No skip tracing, no consumer enrichment, no paid property-data API.

.github/workflows/distress-geocoding.yml · scripts/distress/geocodeRawRecords.ts

Downloaded on the 18th monthly; powers listing activity and several supply composites.

Public economics CSV feed. Do not scrape realtor.com listing pages.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Workflow validates the official Realtor.com weekly CSV; production promotion writes when Supabase write secrets are configured.

Public Realtor.com research/data downloads only. Do not scrape realtor.com listing pages.

.github/workflows/weekly-market-tape.yml · scripts/ingestWeeklyMarketTape.mjs

Workflow validates the official Redfin Housing Market Tracker weekly CSV; production promotion writes when Supabase write secrets are configured.

Public Redfin research/data-center downloads only. Do not scrape Redfin search/listing pages.

.github/workflows/weekly-market-tape.yml · scripts/ingestWeeklyMarketTape.mjs

Downloaded on the 18th monthly; used for closed-sales, above-list, and speed metrics.

Public Redfin S3 market-tracker file. Do not scrape Redfin search/listing pages.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Static 2021 Census facts are re-downloadable and remain labeled 2021 until a newer Census profile exists.

Official Statistics Canada Census Profile files for census divisions and FSAs.

.github/workflows/monthly-data-refresh.yml · scripts/loadCanadaData.py

Paused behind TERANET_LEGAL_APPROVED; obtain written commercial permission before re-enabling.

Teranet-National Bank grants non-commercial personal use only without prior written permission. Do not ingest into OpenRoof until licensed.

scripts/loadCanadaData.py

The February monthly run probes for the newest published ACS vintage; demographic values are not daily signals.

Official Census data. Annual estimates lag the market by design.

.github/workflows/monthly-data-refresh.yml · scripts/fetchCensusData.ts

The monthly workflow discovers the newest Gazetteer vintage and recomputes ZIP and aggregate density.

Official Census Gazetteer land-area file; density is population divided by ZCTA land square miles.

.github/workflows/monthly-data-refresh.yml · scripts/backfillPopulationDensity.mjs

Downloaded on the 18th monthly, after Zillow's normal publication day, and written through the main market ETL.

Public Zillow research CSVs; no ATTOM or paid property-data API.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts

Downloaded on the 18th monthly; ZIP gaps may fall back to Census rent estimates.

Public Zillow research CSVs; no ATTOM or paid rent API.

.github/workflows/monthly-data-refresh.yml · scripts/loadMarketData.ts