feat(event): score first-story detections with FAR and miss rates - #65
Conversation
Keep TDT first-story labels distinct from promoted event instances and require computed false-alarm, miss, and RMSE evidence against known truth.
|
Warning Review limit reachedNext included review available in 13 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (13)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Revalidated current head |
# Conflicts: # CHANGELOG.md # DOCUMENTATION.md # docs/TRACEABILITY.md
|
Current-head refresh completed for |
|
Current-head review refresh for :\n\n- The first-story detection contract reports calibrated false-alarm and miss behavior against known truth without promoting a detection to a state transition or latent scientific claim.\n- , documentation validation, workspace contract, docstring contract, and passed locally.\n- All hosted exact-head checks, including Rust tests, PostgreSQL integration, coverage, security scans, and Strix, pass.\n- No new actionable defect was found at this exact head; no approval is being self-issued. |
|
Current-head review refresh for 928f905:
|
|
Exact-head review receipt: current-main conflict repair was completed and pushed at |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head54bb76dd3d25ae872d1555b78e42fa80a83b1240. -
Head SHA:
54bb76dd3d25ae872d1555b78e42fa80a83b1240 -
Workflow run: 32602891776
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (6 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (6 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (8 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (8 files)"]
R2 --> V2["docs review"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage Decision
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (6 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (6 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (8 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (8 files)"]
R2 --> V2["docs review"]
|
Stale coverage-evidence CHANGES_REQUESTED. Exact head 54bb76d has coverage-evidence success. Not an approval.
Drive first_story_false_alarm_rate and first_story_miss_rate with a four-mention mixed stream whose class-conditional error rates are exactly one half, and require residual RMSE below machine scale.
|
Pushed |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Union of CHANGELOG/DOCUMENTATION doctoring rows; event_core error-variant union (DetectionIsNotTransition/PredictionIsNotFact alongside first-story refusals); ADR 0016 maturity paragraphs consolidated into one statement; ADR index 0016 rows deduped keeping combined wording; APA register keeps Allan et al. (1998) before Allen (1983); validation ledger keeps the first-story FAR/miss row and main's nested-ICC membership row.
There was a problem hiding this comment.
🔍 Stray agent-state files committed at repo root
registered_agents.json and task_agent_mapping.json are new empty {} files unrelated to the first-story detection feature. They appear to be accidentally committed agent runtime state and likely should be removed or gitignored.
Was this helpful? React with 👍 or 👎 to provide feedback.
| fn rate_over_class( | ||
| truth: &[FirstStoryLabel], | ||
| decided: &[FirstStoryLabel], | ||
| class: FirstStoryLabel, | ||
| error_label: FirstStoryLabel, | ||
| ) -> Result<f64, EventError> { | ||
| if truth.is_empty() || truth.len() != decided.len() { | ||
| return Err(EventError::InvalidWirePayload); | ||
| } | ||
| let mut class_count = 0_u32; | ||
| let mut error_count = 0_u32; | ||
| for (truth_label, decided_label) in truth.iter().zip(decided) { | ||
| if *truth_label == class { | ||
| class_count += 1; | ||
| if *decided_label == error_label { | ||
| error_count += 1; | ||
| } | ||
| } | ||
| } | ||
| if class_count == 0 { | ||
| return Err(EventError::InvalidWirePayload); | ||
| } | ||
| Ok(f64::from(error_count) / f64::from(class_count)) | ||
| } |
There was a problem hiding this comment.
📝 Info: Divergent empty-class behavior across two first-story APIs
The new rate helpers in first_story.rs fail closed with InvalidWirePayload when a class is absent, while the pre-existing first_story_detection_rates returns 0.0 for the same case (crates/event_core/src/intelligence.rs:114-136). Two coexisting APIs for the same concept with opposite edge-case semantics can mislead consumers.
Was this helpful? React with 👍 or 👎 to provide feedback.
# Conflicts: # DOCUMENTATION.md # crates/event_core/src/error.rs # crates/event_core/src/lib.rs # docs/TRACEABILITY.md # docs/adr/0016-tdt-chronos-event-intelligence-boundary.md
| ( | ||
| EventError::UnknownFirstStoryLabel, | ||
| "unknown first-story label", |
There was a problem hiding this comment.
🟡 Merged test tuple drops two error assertions
The UnknownFirstStoryLabel test case is missing its closing paren and the reopening paren for the next case, so it fuses with EventTrackIsNotEventInstance into a single four-element tuple. The two variants are no longer asserted separately, and the array of two-element tuples no longer type-checks.
| ( | |
| EventError::UnknownFirstStoryLabel, | |
| "unknown first-story label", | |
| ( | |
| EventError::UnknownFirstStoryLabel, | |
| "unknown first-story label", | |
| ), | |
| ( |
Was this helpful? React with 👍 or 👎 to provide feedback.
| //! and scientific estimation. Mentions and first-story detections never | ||
| //! silently become instances, and TDT detections and CHRONOS predictions | ||
| //! remain measurement or hypothesis artifacts until independently promoted. |
There was a problem hiding this comment.
🟡 Duplicated, incoherent module docstring
The new summary sentence was added without removing the old overlapping text, so the module doc repeats "and scientific estimation. Mentions..." and the TDT/CHRONOS clause twice, producing a broken, self-contradicting paragraph.
Prompt for agents
The module-level doc comment in crates/event_core/src/lib.rs now contains duplicated text. The newly added lines 7-9 ("and scientific estimation. Mentions and first-story detections never silently become instances, and TDT detections and CHRONOS predictions remain measurement or hypothesis artifacts until independently promoted.") overlap with the pre-existing lines 10-12 ("and scientific estimation. Mentions never silently become instances. TDT detections and CHRONOS predictions remain measurement or hypothesis artifacts until independently promoted. Track assignments, story..."). The result is a single garbled paragraph that repeats the same phrases twice. Consolidate into one coherent paragraph: keep the new first-story mention, then continue directly into the 'Track assignments, story segmentations, ...' sentence, removing the duplicated 'and scientific estimation. Mentions never silently become instances. TDT detections and CHRONOS predictions remain measurement or hypothesis artifacts until independently promoted.' fragment.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
#37 merged on protected main (
babd591). #45 kept ready (0007) but remains review-blocked (no independent APPROVE; still CONFLICTING after #37). Next no-0008 gap: first-story false-alarm/miss calibration in existingevent_core(does not recreate TDT/CHRONOS crates, does not wait on #48/#49/#53, and does not allocate 0008).refuse_first_story_as_instancenever promotes a detection to an instancedecide_first_storyuses an inclusive probability thresholdfirst_story_false_alarm_rateandfirst_story_miss_ratefail closed on empty, mismatched, or single-class streamsTest plan
first_story_contractfailed withE0432/E0599cargo test -p event_core --offlinecargo clippy -p event_core --all-targets --offline -- -D warningspython3 scripts/validate_documentation.pyandcheck_docstrings.pyDo not self-approve or merge. Prefer merge #45 when exact-head required checks and a qualifying independent APPROVE exist. Do not allocate 0008.