Step 6 Phase 1 — Rubric Bootstrap (Design Spec)

Source: plans/amendment-3.1+5.3+6.1-bootstrap-rubric-extension.md · synced 2026-06-12

Date: 2026-06-11 Phase: Step 6 Step 6.1 — the per-sub-vertical rubric content that the Layer 1 hard-fit scorer reads. Also extends Step 3.1 mining output to cover Step 5.3 hard-rules content. Status: Draft — awaiting greenlight. Depends on: ✅ Step 3.1 harness built + tested (calibre-app/src/lib/vocab-bootstrap/, 47 tests). ✅ Step 5.1 CV parser shipped (parsed CVs in outputs/parsed-cvs/). ✅ 41 JDs in reference/JD/. ✅ 3-level taxonomy locked (SOURCE-OF-TRUTH.md §3). ✅ Layer 1 rubric formulas locked (plans/pipeline-1.1+2.1+3.2+6.1+7.1+7.2+8.1-matching-engine-design.md §“Layer 1 — Hard Fit”). Closes: D5 from session 2026-06-11. Partially closes D8 (5.4 hard-rules content) — Daniel + Jon scaffold-review session still required after mine fires. Supersedes: None. Extends Step 3.1 design (step-3.1-vocab-bootstrap-design.md) — does NOT replace it.

⚠️ AMENDED 2026-06-12 — the 2026-06-11 mined xlsx must be RE-EMITTED before the Dan + Jon ratification session (see SOURCE-OF-TRUTH.md §8 2026-06-12 entry):

Rubric weights → 4-dim scheme. sub_function_match is no longer weighted (it’s the Step 6.0 entry gate). scoring_weights defaults become: tag_overlap 0.40 / seniority_delta 0.20 / years_experience 0.20 / red_flag_penalty 0.20 (still sum to 1.0).

Hard-gate sheet splits by rule type. drop rules → Step 6.0 config (fires BEFORE Layer 1); block_at_l2 / require_evidence → Step 6.2 config (fires AFTER Layer 1).

Boost/penalty deltas re-scale for the net ±0.5 cap (sum-then-clip). The ±0.3–1.5 per-rule values are superseded. Step ID note: “5.3” in this doc now reads as “6.0/6.2” — the filter was renumbered into Step 6.

1. Context

The original Step 3.1 mine produces ONE artefact: the locked tag vocab at config/tag-vocab.json. Two other artefacts have the same input requirements (107 CVs × 41 JDs), the same mining pattern, and the same ratification cadence — but no plan exists for them yet:

Step 5.3 hard-rules content — recruiter heuristics encoded as deterministic config. Scaffold exists in step-5.3-industry-filter-design.md (10 example rules), but the actual rule set must be derived from observed CV↔JD patterns, then ratified.
Step 6.1 per-sub-vertical rubric content — must_have_signals, nice_to_have_signals, disqualifying_signals, title_normalisations, and per-dimension scoring_weights for 22 sub-verticals (per SOURCE-OF-TRUTH.md §3b). The 4 existing stubs at reference/rubric-configs/ use the wrong 4-grant-vertical schema and are empty.

Both depend on the same mine + Dan’s ratification. Running three separate mines triples cost (~USD 60–120) and triples ratification sessions (~6 hours of Dan’s time).

This plan extends the Step 3.1 mine to emit all three artefacts in a single live run with one ratification session.

Without placement-outcome data (treat as zero per session 2026-06-11), the mine is hypothesis-driven: Claude infers fit-discriminating patterns from CV/JD content alone, then Dan validates. Same defensibility shape as Step 3.1 vocab — grant asks for “recruiter-validated rubrics,” not statistically-validated rubrics. Dan’s ratification IS the validation.

2. Architectural Decisions

Decision	Choice	Rationale
Extend or new mine	Extend Step 3.1 harness	Same input set (107 CVs × 41 JDs), same Claude tool-use pattern, same 5-stage shape. Reusing the harness avoids duplicating ~1,500 lines of code.
Single mine or three	Single mine, three outputs	One API run (~USD 20–40 vs ~USD 60–120). One ratification session (~2h vs ~6h).
Hypothesis-driven vs outcome-driven	Hypothesis-driven (no placement data assumed)	Calibre has no placement-outcome records per session 2026-06-11. Outcome data, when available later, becomes a separate refinement pass.
Rubric weight derivation	Frequency-weighted starter weights, Dan tunes	Signals appearing in >70% of JDs in a sub-vertical get high weight. Dan overrides at ratification.
5.4 split (a/b)	Yes — emit hard gates (5.4a) and boost/penalty (5.4b) as separate JSON files	Per session 2026-06-11 D1 rec: cleaner audit, two configs one purpose each.
Rubric file granularity	One JSON per `(sub_function, sub_vertical)` cell — 22 files	Matches SOT §3b sub-vertical count. Replaces 4 grant-vertical stubs.
Thin sub-function handling	Emit thin rubrics + flag at ratification	`finance-fo` (5 JDs) + `ops-back-office` (4 JDs) produce weak signals. Dan decides at ratification: accept thin / merge sub-verticals / defer.
Ratification format	Single session, three worksheets (vocab + 5.4 rules + rubrics)	Dan + Jon, ~2 hours. Sequential walk through each artefact.
Re-run cadence	Same as Step 3.1 — auto-regen every 30 days or when CV/JD count crosses threshold	Approved entries persist; new content surfaces for review.
Output file paths	Replace, don’t extend	Existing 4 grant-vertical stubs move to `_archive/` post-ratification. New 22 sub-vertical files land at `reference/rubric-configs/{sub-function}.{sub-vertical}.json`.

3. End-to-End Architecture


┌─────────────────────────────────────────────────────────────────────┐
│  Pre-locked structural data (no change vs Step 3.1)                │
│  - 5 sub_functions                                                  │
│  - 22 sub_verticals (SOT §3b)                                       │
│  - Layer 1 rubric dimensions (jd-cv-matching-design.md §Layer 1)    │
│  - Step 5.3 rule schema (5-phase-4-industry-filter-design.md §Rule │
│    Schema) — both hard gates + boost/penalty                        │
└─────────────────────────────┬───────────────────────────────────────┘
                              │
┌─────────────────────────┐   ┌─────────────────────────────────────────────────┐
│  41 JD PDFs             │──▶│  Stage 1: Term + signal extraction (per doc)    │
│  reference/JD/          │   │  Claude tool-use per JD/CV:                     │
│                         │   │  → categorical terms (same as Step 3.1)         │
│  ~107 parsed CVs        │──▶│  → recurring binary patterns (5.4 candidates)    │
│  outputs/parsed-cvs/    │   │  → JD-stated must/nice/disqualifier signals      │
└─────────────────────────┘   │    (6.1 rubric candidates)                       │
                              └─────────────────────┬───────────────────────────┘
                                                    │
                                                    ▼
                              ┌─────────────────────────────────────────────────┐
                              │  Stage 2: Bucket by (sub_function, sub_vertical)│
                              │  (unchanged from Step 3.1)                     │
                              └─────────────────────┬───────────────────────────┘
                                                    │
                          ┌─────────────────────────┼─────────────────────────┐
                          │                         │                         │
                          ▼                         ▼                         ▼
              ┌──────────────────┐    ┌──────────────────────┐    ┌──────────────────────┐
              │ Stage 3a:        │    │ Stage 3b:            │    │ Stage 3c:            │
              │ Cluster + name   │    │ Cluster recurring    │    │ Cluster signal       │
              │ terms per cell   │    │ binary patterns      │    │ candidates per cell  │
              │ (Step 3.1       │    │ → 5.4a hard gates +  │    │ → 6.1 must_have /    │
              │  behaviour)      │    │   5.4b boost/penalty │    │   nice_to_have /     │
              │                  │    │   candidates         │    │   disqualifying      │
              └────────┬─────────┘    └──────────┬───────────┘    └──────────┬───────────┘
                       │                         │                            │
                       ▼                         ▼                            ▼
              ┌──────────────────┐    ┌──────────────────────┐    ┌──────────────────────┐
              │ Stage 4a:        │    │ Stage 4b:            │    │ Stage 4c:            │
              │ tag-vocab        │    │ expert-rules         │    │ rubric-configs       │
              │ .candidate.json  │    │ .candidate.json      │    │ .candidate.json[]    │
              │                  │    │ (split into hard +   │    │ (22 sub-vertical     │
              │ (Step 3.1       │    │  soft files)         │    │  files, weights      │
              │  output)         │    │                      │    │  proposed from       │
              │                  │    │                      │    │  signal density)     │
              └────────┬─────────┘    └──────────┬───────────┘    └──────────┬───────────┘
                       │                         │                            │
                       └─────────────────────────┼────────────────────────────┘
                                                 ▼
                              ┌─────────────────────────────────────────────────┐
                              │  Stage 5: Ratification session                  │
                              │  Dan + Jon (~2h), 3 worksheets sequentially:    │
                              │  - tag vocab worksheet (Step 3.1)              │
                              │  - 5.4a hard gates + 5.4b boost/penalty         │
                              │  - 22 sub-vertical rubric worksheets            │
                              │                                                 │
                              │  → config/tag-vocab.json                        │
                              │  → reference/expert-rules/hard-gates.json       │
                              │  → reference/expert-rules/boost-penalty.json    │
                              │  → reference/rubric-configs/                    │
                              │       {sub-function}.{sub-vertical}.json × 22   │
                              └─────────────────────────────────────────────────┘

4. Three Artefacts — Output Schemas

4a. Tag vocab (unchanged — Step 3.1 output)

config/tag-vocab.json — schema and stages 1, 2, 3a, 4a unchanged from step-3.1-vocab-bootstrap-design.md. This plan does not modify Step 3.1’s vocab logic.

4b. Step 5.3 expert rules — split into two files

reference/expert-rules/hard-gates.json — drop / block_at_l2 rules only.


interface HardGatesFile {
  version: string                       // "bootstrap-YYYY-MM-DD" or "ratified-YYYY-MM-DD"
  tenant_id: string                     // "calibre" (multi-tenant hook)
  generated_from: {
    run_id: string                      // ULID of the mining run
    cv_count: number
    jd_count: number
  }
  rules: ExpertRule[]                   // schema in step-5.3-industry-filter-design.md
                                        // restricted: effect.kind ∈ {"drop", "block_at_l2"}
}

reference/expert-rules/boost-penalty.json — boost / penalty / require_evidence rules only.


interface BoostPenaltyFile {
  version: string
  tenant_id: string
  generated_from: { run_id, cv_count, jd_count }
  rules: ExpertRule[]                   // schema in step-5.3-industry-filter-design.md
                                        // restricted: effect.kind ∈ {"boost", "penalty", "require_evidence"}
}

Both files share the same ExpertRule schema (see step-5.3-industry-filter-design.md §Rule Schema). Split is purely by effect.kind — runtime loader reads both, applies them at the appropriate pipeline stages.

Cardinality target: 30–60 rules total across both files for the initial scaffold. Dan can extend at ratification.

4c. 22 per-sub-vertical rubric configs

reference/rubric-configs/{sub-function}.{sub-vertical}.json — one file per sub-vertical. Examples: cargo-ops.energy.json, ship-ops.tanker.json, risk-analytics.credit.json, finance-fo.rm.json, ops-back-office.aml.json.


interface RubricConfig {
  // Identity
  sub_function: SubFunction             // SOT §3a
  sub_vertical: SubVertical             // SOT §3b
  display_name: string                  // e.g. "Cargo Operations — Energy (LNG/LPG/Crude/Petrochem)"
  version: string                       // "bootstrap-YYYY-MM-DD" or "ratified-YYYY-MM-DD"
  generated_from: { run_id, cv_count_in_cell, jd_count_in_cell }
 
  // Signals (the substance — what makes good/bad fit in this cell)
  must_have_signals: Signal[]           // boost the tag_overlap dimension; absent → low score
  nice_to_have_signals: Signal[]        // boost composite when present; absent → neutral
  disqualifying_signals: Signal[]       // drop or heavy penalty
 
  // Title normalisations (raw_title → normalised_title mapping for this cell)
  title_normalisations: TitleNorm[]
 
  // Per-dimension scoring weights (must sum to 1.0)
  scoring_weights: {
    sub_function_match: number          // default 0.20 (low — already gated upstream)
    tag_overlap: number                 // default 0.30
    seniority_delta: number             // default 0.20
    years_experience: number            // default 0.15
    red_flag_penalty: number            // default 0.15
  }
 
  // Cutoffs
  minimum_pass_score: number            // default 4.0 (matches SOT)
 
  // Status
  status: "bootstrap-draft" | "ratified" | "deprecated"
  ratified_by?: string                  // "team@calibre.sg"
  ratified_at?: string                  // ISO 8601
  notes?: string                        // free text for thin-data / known issues
}
 
interface Signal {
  type: "tag" | "keyword" | "certification" | "employer_class" | "geography" | "language"
  values: string[]                      // e.g. ["lng", "lpg"] or ["CFA", "FRM"]
  weight: number                        // 1–5 (relative within the signal kind)
  source_docs: string[]                 // filenames of JDs/CVs where this signal was observed
  observed_in_pct: number               // 0–1, frequency across the cell's JDs (drives bootstrap weight)
}
 
interface TitleNorm {
  from_pattern: string                  // regex — e.g. "(?i)cargo (operator|operations)"
  to_canonical: string                  // e.g. "Cargo Operator"
  seniority_signal?: "Junior" | "Mid" | "Senior" | "Lead" | "Head"
}

Bootstrap-weight rule: signals with observed_in_pct ≥ 0.7 land in must_have_signals with weight = 5. Signals with 0.3 ≤ observed_in_pct < 0.7 land in nice_to_have_signals with weight = 2–3 (proportional). Below 0.3 → discarded as noise unless Dan rescues at ratification.

Default scoring_weights (above) are starter values. Dan tunes per-sub-vertical at ratification — e.g. for cargo-ops.energy he might raise tag_overlap to 0.40 because LNG/LPG/crude expertise is highly specific.

5. Stage-by-Stage Changes from Step 3.1

Stage 1 — Term + signal extraction (extended Claude prompt)

Step 3.1’s prompt extracts categorical_terms, tools_and_systems, etc. This plan extends the prompt to ALSO extract:

From each JD:

stated_must_haves: string[] — explicit “Required:” / “Must have:” lines
stated_nice_to_haves: string[] — explicit “Preferred:” / “Bonus:” lines
stated_disqualifiers: string[] — explicit “Not eligible:” / “Not suitable for:” lines
stated_seniority_floor: string — “Senior”, “5+ years”, etc.
stated_years_floor: number | null
recurring_hard_rules: HardRuleCandidate[] — observed binary patterns (e.g. “JD mentions Singapore-only”, “JD excludes specific employer”)

From each CV (reuse parsed JSON, no new Claude call):

All existing Step 3.1 fields
Plus: derived signal patterns (e.g. “all 5-year-tenure CVs at top-tier employer X”).

Cost impact: ~+0.5 USD per JD vs Step 3.1 prompt. Total mine cost: ~USD 20–40 (vs Step 3.1’s USD 5–10).

Stage 2 — Bucket by (sub_function, sub_vertical) (unchanged)

Same as Step 3.1.

Stage 3a — Cluster terms (unchanged from Step 3.1)

Produces tag vocab candidates.

Stage 3b — Cluster recurring binary patterns → 5.4a/5.4b candidates (NEW)

Per (sub_function, sub_vertical) cell, Claude tool-use call:

Input: recurring_hard_rules[] + stated_disqualifiers[] + Step 5.3 example rules (10 from existing scaffold).

Output: proposed ExpertRule[] split by effect kind:

effect.kind ∈ {"drop", "block_at_l2"} → 5.4a hard gates candidate
effect.kind ∈ {"boost", "penalty", "require_evidence"} → 5.4b boost/penalty candidate

Cardinality target per cell: 2–5 hard gates + 3–7 boost/penalty.

Stage 3c — Cluster signal candidates per cell → rubric content (NEW)

Per (sub_function, sub_vertical) cell, Claude tool-use call:

Input: all stated must/nice/disqualifier signals from JDs in this cell + recurring CV signal patterns.

Output: proposed Signal[] for each of must_have_signals, nice_to_have_signals, disqualifying_signals per cell.

Frequency calculation done deterministically post-Claude — observed_in_pct = (count of JDs mentioning this signal) / (total JDs in cell).

Stage 4a/4b/4c — Emit three artefact families (extended)

Step 3.1 emits config/tag-vocab.candidate.json. This plan also emits:

reference/expert-rules/hard-gates.candidate.json
reference/expert-rules/boost-penalty.candidate.json
reference/rubric-configs/{sub-function}.{sub-vertical}.candidate.json × 22

All .candidate extensions get stripped at ratification (Stage 5).

Stage 5 — Single ratification session, three worksheets

Dan + Jon, ~2 hours. Sequential walk:

Worksheet	Format	Decisions per row
1 — Tag vocab	One row per proposed tag	approve / rename / merge with X / reject
2 — Expert rules	One row per rule	approve / tune effect.delta / tune trigger / reject / add new
3 — Rubrics × 22	One sheet per sub-vertical, signals + weights	approve signal / move tier (must→nice / nice→must) / drop signal / adjust per-dimension weight / approve thin rubric / merge sub-verticals / defer cell

At end of session, run-script writes .ratified.json siblings; deploy script promotes them to canonical paths.

6. Cost + Time Budget

Item	Cost	Wallclock
Mine — Claude API (extended prompt)	~USD 20–40	~30–45 min
Ratification session	$0	~2h Dan + Jon
Plan implementation (extend harness + write 3 emit functions + ratification CLI)	$0 — my time	~3–4 hours
Total fresh dollars to fire this	USD 20–40	—

7. What This Unblocks

Per SOURCE-OF-TRUTH.md §6 critical path:

Once ratified	Unblocks
Tag vocab	Step 3.2 (JD normaliser) + Step 5.2 (CV normaliser)
Expert rules	Step 5.3 implementation
Rubric configs	Step 6.1 (Layer 1 hard-fit)

All four phases unblock in parallel. Step 7 (Layer 2 deep assess) then has its rubric criteria locked. Step 8 (shortlist) score bands are already locked.

Net effect: the matching engine (Steps 3.2, 5.2, 5.4, 6.1, 7.1, 7.2, 8.1) can build end-to-end from a single live run + one ratification session.

8. Risks + Caveats

#	Risk	Mitigation
R1	No placement-outcome data — hypothesis-driven, not statistically validated	Dan’s ratification IS the validation. Grant defensibility unchanged (asks for “recruiter-validated”, not “statistically-validated”). When placement data accumulates later, refinement pass becomes possible.
R2	Thin sub-functions (`finance-fo`: 5 JDs, `ops-back-office`: 4 JDs) produce weak rubrics	Flag at ratification. Dan picks: accept thin + mark “growth area” / merge sub-verticals / defer cell until more data lands.
R3	22 rubric files at ratification = high cognitive load on Dan	Mitigation: sub-verticals within same sub-function share dimension structure — most differences live in `must_have_signals`. Worksheets group by sub-function so Dan can pattern-recognise across siblings.
R4	Mine prompt drift — extended prompt is more complex than Step 3.1’s	Mitigation: split into 3 separate Claude calls per cell (Stage 3a/3b/3c) rather than one mega-prompt. Each call is small, testable.
R5	4 existing rubric stub files at wrong paths cause loader confusion	Mitigation: stub deletion is a Stage 5 post-ratification step. Move to `_archive/2026-06-XX-rubric-stubs-4-vertical/`.
R6	Ratification session runs over 2 hours	Mitigation: pre-circulate worksheets 24h ahead. Dan + Jon arrive with strikes already pencilled in. Session is for discussion of disputed rows, not first read.

9. Dependencies

Hard preconditions (must exist before mine fires)

✅ Step 3.1 harness built (calibre-app/src/lib/vocab-bootstrap/)
✅ Step 5.1 CV parser shipped, parsed CVs at outputs/parsed-cvs/
✅ 41 JDs in reference/JD/
✅ 3-level taxonomy locked (SOT §3)
✅ Layer 1 rubric formulas locked (jd-cv-matching design)
✅ Step 5.3 ExpertRule schema locked (5-phase-4-industry-filter design)
⏳ Implementation plan for this design (next step after greenlight)
⏳ Dan + Jon calendar slot for ratification (~2h, target within 1 week of mine completion)

What this plan does NOT cover

Live execution of the mine (separate run; greenlight first)
Ratification CLI implementation (will live in scripts/review-rubric-bootstrap.ts)
Step 6.1 scorer code itself — that consumes this plan’s output; separate impl plan
Step 5.3 runtime loader — separate impl plan that consumes the rule files

10. Next Steps

After greenlight:

Write implementation plan: plans/step-6-phase-1-rubric-bootstrap-impl.md — Tasks 1–N extending the existing 3.1 harness.
Build extended Stages 3b + 3c + 4b + 4c on a feature branch off calibre-app/feat/vocab-bootstrap.
Run unit tests (clone existing 47-test pattern).
Fire live mine (USD 20–40, ~30–45 min).
Pre-circulate 3 worksheets to Dan + Jon.
Run ratification session (~2 hours).
Promote .candidate.json → canonical paths.
Archive 4 old grant-vertical rubric stubs.
Update PLANS-LOG.md + STATE-LOG.md + SOT §6 timeline.
Open impl plans for downstream: 3.2 JD normaliser, 5.2 CV normaliser, 5.4 runtime loader, 6.1 scorer.

11. Greenlight Checklist for Ben

Approve combined-mine approach (vs three separate plans)
Confirm 5.4 split into 5.4a + 5.4b (this design assumes yes per session D1 rec)
Confirm 22-sub-vertical granularity (vs fewer — e.g. merge thin sub-verticals upfront)
Confirm USD 20–40 mine budget
Confirm 2h Dan + Jon ratification slot is bookable within 1 week of mine completion
Authorise implementation plan draft (step-6-phase-1-rubric-bootstrap-impl.md)

Once boxes are ticked, I write the impl plan + open a PR with extended harness. Mine doesn’t fire until you greenlight that PR too.