Step 5.0 — Candidate-record merge (Design Spec)

Source: plans/step-5.0-candidate-merge-design.md · synced 2026-06-12

Date: 2026-06-11 Covers steps: 5.0 (bridge between Step 4 sourcing and Step 5.1.1 parsing). Status: Draft — awaiting greenlight. Depends on: ✅ Step 5.1.1 CV parser shipped. ⏳ Step 4 sourcing plan (none yet — will consume CandidateRecord schema defined here). Closes: D8 from _internal/engine-map-audit-2026-06-11.md.

1. Context

Step 4 produces structured profile JSON from Apify (LinkedIn), Apollo (employer/decision-maker enrichment), Anymail Finder (verified email). Step 5.1.1 was built to parse CV file blobs (PDF/DOCX) into ParsedCV. There’s no defined bridge.

Without Step 5.0, the runtime engine has two disjoint candidate ingestion paths that produce different schemas. Step 5.2 (CV normaliser) only knows how to read ParsedCV from file. Scraped profile data can’t reach the matching pipeline.

Step 5.0 produces a unified CandidateRecord from any combination of: scraped profile JSON (always), uploaded CV file blob (sometimes), enrichment data from Apollo (sometimes), verified email from Anymail Finder (sometimes). Downstream sub-steps read CandidateRecord only.

2. Architectural Decisions

Decision	Choice	Rationale
Single merged record vs parallel paths	Single — `CandidateRecord`	Downstream code is simpler; one schema to reason about; one place to fix bugs.
Source-of-truth precedence	CV file (when present) overrides scrape on overlapping fields; scrape overrides Apollo enrichment on identity fields	CV is candidate-authored; scrape is platform-curated; Apollo is third-party inference. Cascade is most-trustworthy-wins.
Conflict resolution audit	Every overridden field is logged with `source`, `original_value`, `chosen_value`, `chosen_reason`	Audit-defensible. Recruiter can inspect “why did Step 5.0 pick this employer name.”
Missing data	Fields default to `null`. Downstream rubrics handle null per-dimension.	No fake data. Better to score-down a missing field than fabricate one.
Identity (primary key)	LinkedIn URL (when present) → email (when verified) → `cv_id` (parsed CV file path)	LinkedIn URL is the most stable cross-source identifier.
Re-merge cadence	Re-run Step 5.0 whenever any source updates (scrape refresh, new CV uploaded, Apollo re-enrichment)	Candidate records are mutable. New data wins per the precedence cascade.
File location	`calibre-app/src/lib/candidate-merge/`	Sibling to `cv-parser/` and `jd-parser/`.

3. End-to-End Flow


Step 4.1 Apify scrape ─┐
Step 4.2 Apollo       ─┼─→  Step 5.0 merge  ─→  CandidateRecord  ─→  Step 5.1.1 (if file)
Step 4.3 Anymail      ─┤                                              Step 5.2 (always)
Uploaded CV (manual)  ─┘

4. CandidateRecord Schema


type Source = "linkedin_scrape" | "apollo" | "anymail" | "cv_file" | "manual";
 
interface CandidateRecord {
  // Identity
  candidate_id: string;              // ULID
  linkedin_url: string | null;       // primary cross-source key
  primary_email: string | null;      // from Anymail (verified) or CV
  primary_email_confidence: number;  // 0–1
  cv_file_path: string | null;       // if a CV was uploaded
 
  // Demographics
  full_name: string;
  full_name_source: Source;
  current_location: string | null;
  current_location_source: Source;
 
  // Current role
  current_employer: string | null;
  current_employer_source: Source;
  current_title: string | null;
  current_title_source: Source;
  current_seniority: "Junior" | "Mid" | "Senior" | "Lead" | "Head" | null;
 
  // Employment history (merged, deduped)
  employment_history: EmploymentEntry[];
  employment_history_provenance: Source[];  // which sources contributed
 
  // Skills + signals
  raw_signals: {
    languages: string[];
    certifications: string[];
    tools_systems: string[];
    commodity_specialisations: string[];
    vertical_signals: string[];
  };
  raw_signals_provenance: Record<keyof CandidateRecord["raw_signals"], Source[]>;
 
  // Merge audit
  merge_metadata: {
    merged_at: string;               // ISO 8601
    sources_present: Source[];
    conflicts: ConflictResolution[]; // every field where ≥2 sources disagreed
  };
}
 
interface EmploymentEntry {
  employer: string;
  raw_title: string;
  start_date: string | null;         // ISO 8601 or "YYYY-MM"
  end_date: string | null;           // null = current
  source: Source;
  confidence: number;
}
 
interface ConflictResolution {
  field_path: string;                // e.g. "current_employer"
  candidates: Array<{ source: Source; value: string; confidence: number }>;
  chosen_source: Source;
  chosen_value: string;
  chosen_reason: "precedence" | "confidence" | "recency" | "manual_override";
}

5. Merge Algorithm

For each field on CandidateRecord:

Collect all values from present sources.
Apply precedence cascade for canonical fields (name, employer, title, location):
- cv_file > linkedin_scrape > apollo > anymail
For non-canonical fields (skills, languages, certs), union all sources, dedupe by normalised string.
For employment history, dedupe by (employer, start_date) tuple. When duplicate entries exist across sources, prefer cv_file then linkedin_scrape.
Record every disagreement in merge_metadata.conflicts.

6. Implementation Plan (separate file pending greenlight)

Will live at plans/step-5.0-candidate-merge-impl.md. Outline:

Task	File	Test
1	`types.ts` — schemas above	type-only
2	`precedence.ts` — cascade rules	unit
3	`merge.ts` — main merge function	unit + integration (mock all 4 sources)
4	`index.ts` — public `mergeCandidateRecord(...)` API	—
5	Batch script	run on a synthetic test set (3 profiles, 3 source-combinations each)

7. Dependencies + Unblocks

Depends on: Step 4 sourcing plan (defines the scrape output schemas this consumes).
Unblocks: Step 5.2 (CV normaliser will read CandidateRecord, not ParsedCV directly, post this plan). Step 4 sourcing impl (needs to know the target merge schema).

8. Greenlight Checklist for Ben

Approve precedence cascade (cv_file > linkedin > apollo > anymail for canonical fields)
Approve union strategy for skill-like fields
Approve CandidateRecord schema shape
Authorise impl plan draft (step-5.0-candidate-merge-impl.md)