Sourcing Pipeline — Design Spec (Step 4)

Source: plans/pipeline-4-sourcing-design.md · synced 2026-06-12

Date: 2026-06-11 Covers steps: 4.1 (Apify candidate scraping) · 4.2 (Apollo enrichment) · 4.3 (Anymail Finder) · 4.4 (Employer-side BD scraping) Status: ⏸️ Deferred 2026-06-11. Build postponed — orchestrate against existing 41 JDs × 107 CVs first. Design plan stays in workspace (audit-defensible — AD1 sourcing scope is still on file). Revisit after the matching engine (Steps 5–8) proves itself end-to-end on existing data. Depends on: ✅ Step 3.2 JD normaliser design (output NormalisedJD is the search-query input). ⏳ Step 5.0 candidate-merge design (step-5.0-candidate-merge-design.md) — defines CandidateRecord schema this pipeline emits into. Grant mapping: Annex 3 Deliverable AD1 — “AI-Powered Resourcing” — both candidate-side AND employer-side sourcing. See grant/scope.md §“AI-Powered Resourcing”.

1. Context

Step 3.2 produces a NormalisedJD (sub-function, sub-vertical, tag list, seniority floor, years floor). Step 4 turns that JD spec into a longlist of candidate profiles + a parallel longlist of hiring-employer leads (BD).

Two parallel branches:

• Candidate-side (4.1 → 4.2 → 4.3): Apify scrapes LinkedIn → Apollo enriches employer + decision-maker context → Anymail Finder produces verified emails. All three feed Step 5.0 candidate-merge.
• Employer-side (4.4): Scrapes LinkedIn Jobs / careers pages / agency directories for hiring agencies that match Calibre’s verticals. Feeds Calibre’s BD funnel (not the candidate matching engine).

Grant compliance: AD1 explicitly requires BOTH candidate and employer sourcing. Skipping 4.4 breaks the deliverable.

2. Architectural Decisions

3. End-to-End Flow

NormalisedJD
   │
   ▼
4.1 Apify scrape ───→ ProfileScrape[]
   │                        │
   ▼                        ▼
4.2 Apollo enrich    4.4 BD scrape ───→ EmployerLead[]  ───→ Calibre BD funnel
   │
   ▼
4.3 Anymail Finder ──→ VerifiedEmail
   │
   ▼
[scrape + enrichment + email] ──→ Step 5.0 merge ──→ CandidateRecord[]

4. Sub-step Specs

4.1 — Apify candidate scraping

Input: NormalisedJD + per-tenant Apify config. Output: ProfileScrape[] — one record per surfaced LinkedIn profile.

interface ProfileScrape {
  linkedin_url: string;
  full_name: string;
  current_title: string;
  current_employer: string;
  current_location: string;
  employment_history: ProfileScrapeEmployment[];
  skills: string[];
  certifications: string[];
  scraped_at: string;          // ISO 8601
  search_query: SearchQuery;   // for audit trail — which JD + tags led to this
  apify_run_id: string;
}
 
interface SearchQuery {
  jd_id: string;
  sub_function: string;
  sub_vertical: string;
  must_have_tags: string[];
  seniority_tier: string;
  location_filter: string[];   // ["Singapore"] for SG-only mandates
  volume_cap: number;
}

Apify actor: start with apify/linkedin-profile-scraper + apify/linkedin-jobs-scraper. Actor ID configurable per-tenant.

Code target: calibre-app/src/lib/sourcing/apify-client.ts + calibre-app/src/lib/sourcing/search-query.ts.

4.2 — Apollo enrichment

Input: ProfileScrape[] from 4.1. Output: ApolloEnrichment[] — per-profile augmentation (employer firmographics, decision-maker chain at employer, more accurate employment history).

interface ApolloEnrichment {
  linkedin_url: string;        // join key back to ProfileScrape
  employer_firmographics: {
    industry: string;
    headcount_band: string;
    funding_stage: string | null;
  };
  decision_makers_at_employer: Array<{
    name: string;
    title: string;
    linkedin_url: string;
  }>;
  employment_history_enriched: EmploymentEntry[];
  enriched_at: string;
}

Code target: calibre-app/src/lib/sourcing/apollo-client.ts. Cache: keyed on linkedin_url. TTL 30 days.

4.3 — Anymail Finder

Input: (full_name, current_employer) tuples from 4.1. Output: VerifiedEmail[] — one per resolved email.

interface VerifiedEmail {
  linkedin_url: string;        // join key
  email: string;
  confidence: number;          // 0–1
  verification_method: "smtp" | "pattern" | "catchall";
  verified_at: string;
}

Accept threshold: confidence ≥ 0.8. Below → mark candidate as email_status: "unresolved".

Code target: calibre-app/src/lib/sourcing/email-finder.ts. Cache: keyed on (full_name, employer). TTL 60 days.

4.4 — Employer-side BD scraping

Input: Calibre’s vertical config + grant-required vertical list (commodities, maritime, offshore, tech & gaming + risk-analytics, finance-fo, ops-back-office per SOT §3). Output: EmployerLead[] — hiring agencies in those verticals with open mandates.

interface EmployerLead {
  employer_name: string;
  employer_url: string;
  sub_function_match: string;
  open_mandate_titles: string[];
  rationale: string;            // 1-2 sentence "why this is a BD target"
  source: "linkedin_jobs" | "careers_page" | "agency_directory";
  scraped_at: string;
}

Code target: calibre-app/src/lib/sourcing/bd-scraper.ts.

Not piped into matching engine. Lands in bd_leads table for Calibre’s BD team to action.

5. Implementation Plan (separate file pending greenlight)

plans/pipeline-sourcing-impl.md. Outline:

6. Cost + Risk

Risk: LinkedIn ToS. Apify scraping is in a grey zone. Mitigation: use only Apify-curated actors that respect rate limits + robots.txt. If LinkedIn shuts down access, fall back to public CV uploads + Apollo-only enrichment.

7. Dependencies

• Step 3.2 (JD normaliser) must ship before Step 4 has structured search input.
• Step 5.0 (candidate-merge) must ship before Step 4 has a sink to write to.
• Step 5.2 (CV normaliser) reads merge output, but doesn’t gate Step 4 build.

8. Greenlight Checklist for Ben

• Confirm Apify + Apollo + Anymail Finder stack (no substitutions)
• Approve 100-profile-per-JD volume cap
• Approve 0.8 email-confidence threshold
• Confirm EmployerLead[] goes to a separate BD table, not the matching engine
• Authorise impl plan draft (pipeline-sourcing-impl.md)
• Approve fallback path if LinkedIn ToS forces a pivot