Architecture

System Overview

┌─────────────────────────────────────────────────────────────────┐
│  Daily Cron  php artisan rates:daily                            │
│                                                                  │
│  1. Extract Non-QM XLSX  → /storage/app/rates/non-qm/          │
│  2. Extract FlexPoint XLSX → /storage/app/rates/flexpoint/      │
│  3. Fetch matrix PDFs    → /storage/app/matrices/raw/           │
│  4. Parse PDFs           → /storage/app/matrices/*.json         │
│  5. Seed database        → MySQL (flex_rate_api)                │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  MySQL Database                                                  │
│                                                                  │
│  lenders ──┬── rate_sheets ──┬── programs ──┬── loan_products   │
│            │                 │              ├── rate_ladder_…   │
│            │                 │              ├── program_doc_…   │
│            │                 │              ├── program_elig…   │
│            │                 │              └── ltv_matrix_…    │
│            │                 └── prepayment_penalty_tables      │
│            └── (FlexPoint programs, separate rate_sheet)        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  POST /api/pricer/quote  →  Pricer service                      │
│                                                                  │
│  ScenarioRequest → EligibilityEvaluator → PriceCalculator       │
│                              │                                   │
│                    per (program, doc_type):                      │
│                    1. scalar rule checks (FICO, loan amt, DTI…)  │
│                    2. LTV matrix lookup (FICO × loan × occ)      │
│                    3. LTV / CLTV validation                      │
│                    4. Rate ladder traversal + LLPA pricing        │
│                    5. Rank by final_price desc                   │
└─────────────────────────────────────────────────────────────────┘

Data Pipeline

Rate Sheet Extraction (rates:extract-non-qm)

Reads storage/app/incoming/non-qm-latest.xlsx. Each worksheet is a program (Full Doc, DSCR, ITIN, etc.). The extractor outputs:

• Per-program {slug}.json — raw rate ladder and LLPA grids
• _manifest.json — list of programs with slugs, effective date, rate sheet ID

Matrix PDF Fetching (MatrixFetcher)

Downloads 6 PDF files from the lender's public website. Files are cached by SHA-256 so unchanged PDFs do not re-extract. Text is extracted with pdftotext -layout into /storage/app/matrices/raw/{sha256}.txt.

PDF Parsing (MatrixExtract command)

Each program has a dedicated parser class:

Parsers emit MatrixExtractionResult objects with:

• eligibilityRules — min FICO, max DTI, loan limits, eligible citizenship, etc.
• ltvGrid — rows of {fico_min, fico_max, loan_amount_min, loan_amount_max, occupancy, purpose, max_ltv, dscr_ratio_min, dscr_ratio_max}

DSCR, ITIN, Multi-Unit, and Foreign National programs use hand-transcribed data in StaticMatricesSeeder because their PDF layouts are not reliably machine-parseable. The seeder runs after the PDF parsers in RatesDatabaseSeeder.

Database Seeding

RatesDatabaseSeeder calls in order:

1. NonQmSeeder — upserts rate_sheets, programs, loan_products, rate_ladder_entries, prepayment_penalty_tables
2. FlexPointSeeder — same for the conventional/government channel
3. MatricesDatabaseSeeder — syncs parsed program_doc_types, program_eligibility_rules, ltv_matrix_entries
4. StaticMatricesSeeder — inserts hand-transcribed LTV grids for the 4 manual-entry programs

Pricing Engine

EligibilityEvaluator

For each (Program, doc_type) pair, EligibilityEvaluator::evaluate() runs checks in sequence — the first failure short-circuits to a rejection:

1. Scalar rules — each program_eligibility_rules row is evaluated against the scenario:

   • gte / lte — numeric threshold comparisons (FICO, DTI, loan amount, DSCR ratio)
   • in — membership check (citizenship, property type, eligible states)
   • eq — exact match

2. LTV matrix lookup — queries ltv_matrix_entries for the most permissive matching row:

   WHERE program_id = ?
     AND doc_type = ?
     AND fico_min <= credit_score AND fico_max >= credit_score
     AND loan_amount_min <= loan_amount
     AND (loan_amount_max IS NULL OR loan_amount_max >= loan_amount)
     AND (occupancy IS NULL OR occupancy = ?)
     AND (purpose IS NULL OR purpose = ?)
     AND (dscr_ratio_min IS NULL
          OR (dscr_ratio_min <= dscr_ratio
              AND (dscr_ratio_max IS NULL OR dscr_ratio_max >= dscr_ratio)))
   ORDER BY max_ltv DESC

   If no row matches → no_ltv_matrix_cell rejection (the specific combination is ineligible per the matrix).

3. LTV / CLTV validation — ltv > max_ltv or cltv > max_cltv → rejection with the threshold and provided value.

PriceCalculator

For every passing scenario, PriceCalculator::resolveBasePrices() first picks the storage shape:

• Non-QM programs ship a rate_ladder_entries table — one row per note rate per program, with a single base price. The scenario term × 12 must match a loan_products.amort_term.
• FlexPoint programs ship rate_blocks + rate_block_entries — one block per amortization term × (conforming / High Balance) variant, with three lock-day columns (lock_15 / lock_30 / lock_45). The resolver picks the block whose name contains the scenario term token (e.g. "30 Yr"), preferring the non-HB block unless loan_amount > 766,550 (the 2026 FHFA single-family conforming limit), then takes the column for scenario.lock_days (defaulting to lock_30).

For each surviving (rate, base_price) pair, PriceCalculator::calculate():

1. Starts with the base price resolved above.
2. Resolves each LLPA grid into the right (row_label, band) cell for the scenario via the category dispatcher (resolverFor() in PriceCalculator). One resolver per category — FICO/CLTV, Purpose, Doc Type, Property Type, Occupancy, Product, DSCR, Loan Amount, Miscellaneous Adjustments.
3. Adds each matching cell's value to price_adjustment and pushes a structured entry into adjusters[].
4. If any required cell is NA, short-circuits to a llpa_na rate rejection (the offer never reaches eligible[]).
5. Applies the PPP adjuster (Non-QM only — FlexPoint has no per-program PPP tables) and the state adjustment (FlexPoint only — resolves program.stateAdjustments against scenario.state, picking the Fannie or Freddie column based on the program name; missing-state rows surface as a state_not_offered rate rejection).
6. Applies pricing caps and the minimum-price floor (see Pricing caps & floors).
7. Returns the final priced offer.

Multi-lender orchestration

Pricer::price() now iterates every lender's most recent rate sheet (MAX(rate_sheets.id) per lender_id), evaluates each program through the chain above, and tags every offer + rejection with the originating lender_slug ("non-qm" or "flexpoint"). The combined result is sorted descending by final_price with a stable (lender_slug, program_slug, doc_type) tiebreak; the top 25 are returned.

FlexPoint agency-shape programs (Conv Fixed / Conv ARM / HomeReady / HomePossible / RefiNow / FHA / VA / USDA) have no program_doc_types rows and no ltv_matrix_entries rows seeded today. EligibilityEvaluator detects that shape and bypasses the doc-type-registered and LTV-matrix-cell gates rather than auto-rejecting; remaining scalar program_eligibility_rules (state, occupancy, etc.) still apply when seeded. The orchestrator synthesizes a ["full_doc"] lane when program.docTypes is empty so every agency program has exactly one (program × doc_type) tuple to evaluate.

Miscellaneous adjustments (scenario toggles)

Unlike the seven category-keyed grids — where each grid contributes one cell per scenario via the dispatcher — the Miscellaneous Adjustments grid contributes zero-to-many rows per scenario. Each toggle on ScenarioRequest maps to a single labelled row:

This split lets pricing reflect cumulative risk (e.g. an interest-only DSCR cash-out with a 30-day mortgage late will fire three separate misc rows, plus the category-grid adjustments). Wire shape is identical to other adjusters; the category field is the discriminator if a consumer needs to bucket them.

Both lenders feed this same path. Non-QM Miscellaneous Adjustments / Other Adjustments categories come straight from the XLSX grid. FlexPoint's per-program misc_adjusters (Escrow Waiver, Lock Extension per Day, etc.) is split at seed time by FlexPointSeeder: numeric/NA rows are promoted into a synthetic Miscellaneous Adjustments LlpaCategory (band value) so they ride the same matchMiscAdjustmentRows() dispatcher; non-numeric guidance (e.g. Temporary 2-1 Buydown) lands in the program_notes KV bucket as metadata only. key_value_attributes no longer carries any value that influences a price; it is metadata-only, and the misc_adjusters group is rejected by the Nova guard on KeyValueAttribute.

Pricing caps & floors

After LLPA summation, PriceCalculator::applyPricingCaps() clamps final_price against per-program ceilings stored on programs.cap_*. Cap selection depends on occupancy, PPP term, loan amount, and state-PPP restrictions; tightest threshold wins. Every clamp emits an entry in caps_applied[] for traceability.

The Minimum Price floor is the only "cap" that can flip an offer to ineligible — breaching it produces a min_price_floor rate rejection instead of a clipped offer.

DSCR Ratio Tiering

DSCR and Foreign National programs split their LTV matrix into two tiers:

A scenario with dscr_ratio = null only matches rows where dscr_ratio_min IS NULL.

Key Models

Integrity & Test Gates

The pricing engine has no external sanity check — once an LLPA grid is parsed into the database, the engine trusts that every row will be matched by some resolver. To prevent silent value drift from breaking adjustments without breaking any test, the suite includes:

• LlpaRowCoverageTest::test_every_alias_dispatched_row_is_reachable_by_some_scenario (tests/Feature/Pricer/LlpaRowCoverageTest.php) — iterates every alias-dispatched llpa_adjustments row across every seeded program in every lender and asserts that some probe scenario in the enum space resolves to each row. Catches the "row-level silent miss" class: an alias list that omits, mis-orders, or carries an ambiguous needle. Unreachable rows must be added to an explicit allowlist with written justification.
• LlpaRowCoverageTest::test_every_lender_category_has_a_dispatch_strategy_or_is_explicitly_pending (task 012) — distinct from the row-level test above. Asserts every (lender, program, category_name) tuple is either handled by PriceCalculator::resolverFor() / isMiscAdjustmentCategory() or is in the categoriesAwaitingDispatcher() allowlist. Catches the "category-level silent miss" class: a seeded category whose name matches no dispatcher branch and silently contributes 0 to every priced offer. FlexPoint surfaces this class today (Additional Purchase/R/T/Cash-Out LLPA Adjusters, Government Adjustments, Cumulative LLPA Cap); each cluster maps to a pending tech-debt entry.
• Per-program FlexPoint pricer tests (task 012) — PricerQuoteFlexPoint{ConventionalFixed,ConventionalArm,HomeReady,HomePossible,FhaFixed,VaFixed}Test lock in the exact final_price (to 4 decimals) for a representative scenario per program (mid-tier FICO, 75 LTV, primary purchase, CA, 30-day lock). When a dispatcher gap closes or an LLPA shifts, these tests fail and force the developer to re-derive the expected price against the active rate sheet in the same PR — preventing silent baseline drift. The ARM variant currently asserts 0 eligible because PriceCalculator::matchRateBlock() cannot resolve ARM rate blocks for a 30-year scenario; that test flips to an exact-price assertion when ARM resolution lands.
• FlexPointStateAdjustmentCoverageTest (task 012) — for every FlexPoint program with seeded state_adjustments, exhaustively iterates each seeded state and asserts a state_adjustment adjuster fires; for each program also asserts an un-seeded state (HI) produces a state_not_offered rejection. Catches the "state column wired but never read" class.
• Per-parser snapshot tests — pin extractor output structure so an XLSX/PDF layout shift fails loudly rather than silently dropping rows.
• Pricer scenario tests — exercise each program with representative scenarios and assert exact final_price + adjusters[] ordering.

The corresponding tech debt for matcher gaps is tracked in Tech Debt. The coverage allowlists (isKnownUnmappedDimension() and categoriesAwaitingDispatcher()) are the single source of truth for "known-skipped" rows and categories — adding to either requires a written justification linking the pending dispatcher work.