Skip to content

bug(merge-tracker): a re-evaluation that scores LOWER is silently discarded #2411

Description

@Souptik96

merge-tracker.mjs discards a re-evaluation whose score is lower than the row it matches. Nothing warns, no counter reflects it, the new report is orphaned, and the TSV is archived into merged/ as though it had landed.

data/applications.md is user-layer data that is never regenerated and is gitignored, so the loss is unrecoverable.

The path

merge-tracker.mjs:728

if (newScore > oldScore) {
  // ... rewrite the row, updated++
} else {
  console.log(`⏭️  Skip: ${addition.company}${addition.role} (existing #${duplicate.num} ${oldScore} >= new ${newScore})`);
  skipped++;
}

The dedup tiers above this correctly identified the row as the same opening. The only question left is which evaluation is newer, and the answer is always the incoming one — but a lower score routes it to else and it is dropped.

Why a lower score is the case that matters most

A downgrade is rarely noise. In practice it means one of:

  • the candidate's targeting changed — a role that was on-strategy no longer is, so the same JD honestly scores lower;
  • a staleness signal surfaced late — e.g. reading first_published for the first time and finding the requisition is ~12 months old;
  • a gap was re-weighted — something previously judged a "ramp-up item" turns out to be a hard gate.

All three are strictly better information than the score already in the row. Discarding them leaves the tracker asserting a stale optimistic number, which is worse than having no row at all: it actively misleads the next prioritisation pass.

Reproduction

Tracker row #7 at 3.8/5, then a re-evaluation of the same requisition at 3.0/5:

📥 Found 1 pending additions
⏭️  Skip: Initech — Payments PM (existing #7 3.8 >= new 3)

✅ Moved 1 TSVs to merged/

📊 Summary: +0 added, 🔄0 updated, ⏭️1 skipped

Row #7 still reads 3.8/5, still points at report [7], and report [8] exists on disk with nothing referencing it. verify-pipeline.mjs then reports it as an orphan report, which is the only surviving trace.

Equal scores take the same branch, so a re-evaluation that confirms the score but carries fresher notes and a newer report link is also dropped.

Suggested fix

Let a re-evaluation write through in both directions, and make a downgrade visibly distinct rather than silent:

  • log it with its own marker (e.g. 🔽 ... — DOWNGRADE, re-eval scored lower) so it cannot be mistaken for an upgrade in a batch log;
  • count it as an update, not a skip;
  • record the superseded report number in Notes (Superseded report [7] (was 3.8/5)), so that if the fuzzy matcher mis-paired two genuinely different roles the previous evaluation is still reachable from the tracker alone.

That last point matters because role-matcher.mjs treats "Senior" as a stopword and drops tokens of ≤3 characters, so near-identical titles can pair when they shouldn't. Writing through unconditionally without leaving a pointer would turn a bad match into data loss of a different kind; the Notes breadcrumb keeps it recoverable.

skipped stays meaningful — it still counts malformed and genuinely rejected rows.

Relationship to #2392 / #2394 / PR #2399

PR #2399 fixes four other data-loss paths in this same merge loop (stale row snapshot, no intra-run dedup, Notes overwritten on upgrade, missing separator row). None of them cover this branch — that PR only changes what happens inside the newScore > oldScore arm and never revisits the else.

The two do touch adjacent lines, so whichever lands second will need a small rebase. Happy to sequence behind #2399 if that is easier for review.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions