Parser Hardening — Design Spec

Source: plans/step-5.1.2-cv-parser-hardening-design.md · synced 2026-06-12

Date: 2026-05-20 Phase: A — Underlying logic. 9-step coverage: Step 5a hardening. Status: ✅ Shipped on feat/parser-hardening branch. Depends on: Existing CV parser in calibre-app/src/lib/cv-parser/

Note (2026-05-25): Reference set is now ~107 CVs across 5 sub-functions. Hardening logic itself is unchanged but a re-batch across the full set is pending (see context/current-data.md §“Parser Calibration Audit”). “JD-side parsing — Sprint 4” comment in this doc is overtaken — JD work sits in Phase A Sprint 2–3 per SOURCE-OF-TRUTH.md §7.

Context

A 7-CV calibration audit (2026-05-20) found that the parser’s content is accurate but its confidence scores are not honest above ~0.92. Specifically:

• Hallucinated list items at high confidence. Adrian Tan flagged with “LNG” specialisation (0.88) when LNG appears nowhere in his CV. Jonathan Lam flagged with “Crude Oil” (0.95) when source only mentions “crude” as a market reference, not a specialisation.
• Stale CVs treated as current. Jonathan Lam’s CV ended Jan 2019 but the parser called SOCAR his current_role with 1.0 confidence — the CV is 7 years out of date.
• Row fabrication. Rachel Seah’s single Concordia tenure split into 3 rows with invented date boundaries at 0.95 confidence.

These errors would compound into the title normaliser (Sprint 1) and Apollo sourcing (Sprint 2). Cleanest fix: deterministic post-checks the model cannot fool.

Architectural Decisions

Check 1: Source-grounding for list fields

Fields checked

• signals.commodity_specialisations.value
• signals.certifications.value
• signals.tools_and_systems.value

Grounding rule

For each item:

1. Lowercase the item and the raw source text.
2. Tokenise the item: split on whitespace and punctuation, keep tokens of ≥4 characters that are not stop words (the, and, for, with, from, into).
3. An item is grounded if all significant tokens appear as substrings in the source text.
4. Two-letter and three-letter acronyms (e.g. LNG, LPG, FRM, VBA, SAP) bypass the 4-char threshold and must appear as whole-word matches (\bLNG\b).

Action on ungrounded items

• Item is kept in the value array (don’t silently delete; keep visible for review).
• The containing field’s confidence is clamped to min(current, 0.5).
• An entry is appended to parse_metadata.ungrounded_items: { field: "commodity_specialisations", item: "LNG" }.

Why not delete?

Deleting hides the model’s behaviour. Keeping the item with capped confidence makes the failure auditable — useful for the grant audit trail.

Trade-offs

• Risk: too strict. “Crack Spread Products” might fail if source says “Crack Products.” Mitigation: the grounder is informational at first (not a delete), so we can adjust thresholds after seeing what gets flagged.
• Risk: too lenient. “Oil” tokenises to a single 3-char acronym, won’t be checked → matches almost every CV. The 4-char floor + acronym whole-word rule blocks this.

Check 2: Stale-CV detection

Logic

1. Look at current_role.end_date.value and every employment_history[].end_date.value.
2. If current_role.end_date.value is null → not stale, return.
3. Otherwise find the most recent end_date across current_role and history.
4. Compare against cutoff = today - 12 months.
5. If max_end_date < cutoff → stale:
   • Set parse_metadata.stale_cv = true.
   • Set parse_metadata.last_role_end = <max_end_date> for downstream consumers.
   • Leave current_role populated (so downstream can still read the role) but document the staleness in metadata.

Why not null current_role?

The title normaliser needs a role to classify against. Better: keep the role but stamp the metadata flag. Downstream consumers (UI, normaliser, outreach) read stale_cv and decide.

Trade-offs

• Edge case: deliberately omitted end_date. Some candidates leave their current job’s end_date blank. If the parser correctly nulled it, stale detection won’t trigger — good. If the parser mistakenly filled it in, this catches it.
• Edge case: career break. Someone on maternity/sabbatical might have a 2-year gap. False positive risk. Mitigation: stale_cv is informational; downstream can choose to surface or ignore.

Check 3: Row-fabrication detection (deferred)

Rachel Seah’s split into 3 Concordia rows is harder to catch deterministically — requires understanding the source’s narrative structure. Two options:

• Prompt change (cheap): Add to the parser system prompt: “Do not split a continuous tenure at one employer into multiple rows unless the source explicitly lists separate role-change dates.”
• Detection pass (expensive): Compare consecutive rows with the same employer; if dates abut (no gap) and the source text has only one tenure block for that employer, flag.

Decision: Apply the prompt change in this pass. Detection pass deferred — wait to see if the prompt change alone is sufficient on the next batch run.

File Map

Verification Plan

After changes land, re-run npm run batch-parse and check:

1. Adrian Tan: commodity_specialisations confidence is now ≤0.5, “LNG” appears in ungrounded_items.
2. Jonathan Lam: parse_metadata.stale_cv === true, “Crude Oil” appears in ungrounded_items (or is correctly grounded if Marketscan write-up text contains “crude”), confidence ≤0.5 on commodities.
3. Sim Qi Xuan: no regressions — should still parse honestly at ~0.89.
4. Rachel Seah: re-check whether prompt change reduces the 3-row fabrication. If still present, log as known issue and decide on detection pass next session.
5. Aggregate: parse_status="partial" count should rise (from 0) to reflect ungrounded items. That’s a feature, not a bug — partial means “humans should look at this.”

What This Plan Does NOT Cover

• Re-prompt to ask Claude for cleaner output (deterministic > prompt-engineering).
• LinkedIn URL validation — out of scope.
• JD-side parsing — Sprint 4.
• Title normaliser changes — Sprint 1, next plan.

Next Step

If approved: invoke writing-plans to produce the step-by-step implementation plan, then executing-plans to implement.