feat(estimation): ADR 0200 points 2+4 — expected-information estimator, schema union, provenance persistence (no activation) - #583
Conversation
…t 2, no activation) Replaces main's unconditional-refusal estimation stub with the full estimator ADR 0200 specifies: MLS2PLM over dichotomized channel scores with multilevel cluster intercepts, weights = normalized EXPECTED item information over the fitted latent distribution, computed on the fitted person parameters with the package's own predict_proba -- answering the theta-conditionality critique in main's ADR 0145 rejection instead of ignoring it. Non-converged fits are rejected outright (convergence_status must be 'converged'). Method code: mls2plm_expected_information. NOTHING ACTIVATES: the product loader's authorized anchor set remains empty, so persisted-weight activation still fails closed exactly as main's ADR 0145 requires -- this stage lands the estimator and its parameter-recovery evidence only. The demo generative design's follow probabilities are re-declared (0.80/0.72/0.66) so the fixture estimate preserves the designed A-100 demo fork under expected-information weighting; the estimate over that design converges and recovers the declared ordering. Supersedes main's test_unanchored_channel_scores_never_run_a_fit by design: running the fit is now permitted, activation is not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015HgzNGsCxqiTaT4YuJEb5J
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| probabilities = numpy.asarray(predict_proba(result.params, factor_id), dtype=float) | ||
| if probabilities.shape[1] != len(channels): | ||
| return None | ||
| information = (discriminations**2) * probabilities * (1.0 - probabilities) | ||
| expected_information = information.mean(axis=0) |
There was a problem hiding this comment.
🔍 predict_proba second argument reuses the factor map
The expected-information step calls predict_proba(result.params, factor_id) reusing the item-to-factor array (all zeros, length n_channels) as the second argument, then averages over axis=0 as if it were fitted persons. This is correct only if predict_proba(params, factor_id) returns a per-person-by-item matrix from the persons embedded in params. The polytomous path in lineageweave/period_report.py:195 instead passes an explicit theta array to information_polytomous. If the second argument is a theta grid, every item is evaluated at theta 0 and the weights are wrong. The fast_mlsirm signature could not be verified from the repo.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if len(pair_channel_scores) != len(group_ids): | ||
| raise ValueError("pair_channel_scores and group_ids must align") | ||
| if len(pair_channel_scores) < _MIN_SAMPLE_PAIRS: | ||
| return None | ||
| channels = sorted(pair_channel_scores[0]) | ||
| if not channels: | ||
| return None | ||
| for scores in pair_channel_scores: | ||
| if sorted(scores) != channels: | ||
| raise ValueError("every pair must score the same channel set") |
There was a problem hiding this comment.
📝 Info: Small-sample inputs skip channel-set validation
The docstring promises a ValueError for pairs that omit a channel, but the < _MIN_SAMPLE_PAIRS return at line 149 runs before the channel-set check at lines 154-156. A misaligned input with fewer than 200 pairs returns None instead of raising. No runtime consequence: the function fails closed either way and no fit runs.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # ADR 0200 point 2: Birnbaum item information is conditional, | ||
| # I_j(theta) = a_j^2 P_j(theta) Q_j(theta) -- so the fusion weight is | ||
| # the normalized EXPECTED information over the fitted latent | ||
| # distribution, approximated by averaging over the fitted person | ||
| # parameters (the empirical distribution the multilevel model | ||
| # produced), using the package's own item response function | ||
| # (predict_proba) rather than a re-derived one (van der Linden, | ||
| # 2005, on expected/target information as the design quantity). | ||
| probabilities = numpy.asarray(predict_proba(result.params, factor_id), dtype=float) | ||
| if probabilities.shape[1] != len(channels): | ||
| return None | ||
| information = (discriminations**2) * probabilities * (1.0 - probabilities) | ||
| expected_information = information.mean(axis=0) |
There was a problem hiding this comment.
📝 Info: Expected information uses fitted persons, not the ADR mixture
ADR 0200 point 2 integrates item information over the fitted multilevel latent distribution. The code instead averages over the fitted person point estimates, which are shrunk toward cluster means and understate latent variance. The comment acknowledges this approximation; it diverges from the ADR's exact formula.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Accurate reading — the implementation approximates E_θ[I_j] on the fitted person parameters (the empirical latent distribution) rather than integrating the model-implied cluster-intercept mixture analytically. That is the documented approximation in the module docstring; the empirical average converges on the mixture integral as pairs grow and avoids re-deriving the mixture density outside the package's official API. If a future fast-mlsirm release exposes the latent mixture directly, swapping the averaging set is a one-line change with the same method code semantics.
…nt 4, no activation) Migration 0200 unions the two lines' lineage_channel_weight schemas: primary key (channel_set_code, channel_code) -- one persisted set per active-channel combination -- carrying main's full per-run provenance contract and integrity constraints. Validated live against BOTH predecessor shapes (main's 0135 provenance table and the customer-master line's 0135+0136 set table), replay-idempotent per the ADR 0166 window, with a rollback that restores main's single-set shape. Pre-provenance rows are deleted, not backfilled with invented provenance -- the loader refuses them either way and re-estimation is the operator's next action. The loader becomes set-aware (exact active-channel match of exactly one set) while keeping every provenance-integrity gate; a pre-0200 schema is probed via the catalog (never a failing statement that would abort the caller's transaction) and read as the single implicit deterministic set. ACTIVATION IS UNCHANGED: the authorized anchor set stays empty, so every loaded vector is still refused. The operator script becomes the full estimator front-end: fetch on one short-lived connection, no connection held while fitting, persist with full provenance (fresh run uuid, installed estimator version, honest anchor_method_code=unanchored_internal_structure, reproducible source-snapshot digest over the ordered sampled rows, knowledge cutoff = max sampled created_at) on a fresh connection. Its report names activation as blocked until an anchor is authorized. The llm channel is deliberately absent -- bulk synchronous provider calls are banned; llm scoring arrives with the queued worker (ADR 0200 point 5). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015HgzNGsCxqiTaT4YuJEb5J
seonghobae
changed the title
| sets: dict[str, list] = {} | ||
| for row in all_rows: | ||
| sets.setdefault(row["channel_set_code"], []).append(row) | ||
| rows = next( | ||
| ( | ||
| candidate | ||
| for candidate in sets.values() | ||
| if {row["channel_code"] for row in candidate} == active_channels | ||
| ), | ||
| [], | ||
| ) |
There was a problem hiding this comment.
📝 Info: Scoped-set match depends on channel-count distinctness
load_estimated_channel_weights groups rows by channel_set_code and takes the first set whose channel codes equal active_channels (lineage_ingestion.py). Unambiguous matching depends on the migration CHECK allowing only the 3-channel deterministic set and the 4-channel llm set. Malformed data (an llm set missing its llm row) could make two sets collide and the pick order-dependent. The later set(persisted) != active_channels guard is now redundant but harmless.
Was this helpful? React with 👍 or 👎 to provide feedback.
| do $$ | ||
| begin | ||
| if not exists ( | ||
| select from information_schema.columns | ||
| where table_name = 'lineage_channel_weight' | ||
| and column_name = 'estimation_run_id' | ||
| ) then | ||
| delete from lineage_channel_weight; | ||
| end if; | ||
| end $$; | ||
|
|
||
| alter table lineage_channel_weight | ||
| add column if not exists channel_set_code text not null | ||
| default 'channel_set_deterministic', | ||
| add column if not exists estimation_run_id uuid not null, | ||
| add column if not exists estimation_method_code text not null, | ||
| add column if not exists estimator_version text not null, | ||
| add column if not exists anchor_method_code text not null, | ||
| add column if not exists source_snapshot_sha256 text not null, | ||
| add column if not exists sample_pair_count bigint not null, | ||
| add column if not exists knowledge_cutoff timestamptz not null, | ||
| add column if not exists estimated_at timestamptz not null default now(); |
There was a problem hiding this comment.
📝 Info: NOT NULL column adds safe via prior-row deletion
0200 adds NOT NULL columns without defaults (0200_channel_weight_schema_union.sql). This is safe only because the preceding block deletes all rows when estimation_run_id is absent (the customer-master predecessor), and for main's 0135 those columns already exist so the adds are skipped. Constraint names match migrations/0135*.sql exactly, so the re-assert block never double-adds, and replay is idempotent.
Was this helpful? React with 👍 or 👎 to provide feedback.
96f9dad
into
docs/adr-0200-weight-reconciliation
1 participant
Summary
Stacked on #574 (ADR 0200). Implements points 2 and 4 with zero activation surface: the product loader's authorized anchor set stays empty, so persisted-weight activation still fails closed exactly as main's ADR 0145 requires. The fit may run and rows may persist as inert evidence; nothing reaches reconstruction until a follow-up authorizes an anchor method under ADR 0200 point 3.
Point 2 — expected-information estimator (commit 815b577)
estimate_channel_weights: MLS2PLM (mmle, multilevel cluster intercepts per Fox & Glas 2001); weights are the normalized expected item informationw_j ∝ mean_i[a_j² P_j(θ_i)Q_j(θ_i)]computed with fast-mlsirm's ownpredict_probaon the fitted person parameters — engaging the θ-conditionality critique from main's ADR 0145 rejection on the merits (Birnbaum 1968; Lord 1980; van der Linden 2005). Method codemls2plm_expected_information; non-converged fits are rejected outright.test_unanchored_channel_scores_never_run_a_fitby design: the rejection semantics move from "never fit" to "never activate without an authorized anchor".Point 4 — schema union + provenance persistence (commit c6ba024)
(channel_set_code, channel_code)+ main's full provenance columns/constraints. Validated live against both predecessor shapes (main's 0135; the customer-master line's 0135+0136), replay-idempotent (ADR 0166 window), rollback restores main's shape. Pre-provenance rows are deleted, never backfilled with invented provenance.anchor_method_code='unanchored_internal_structure', reproducible source-snapshot digest, knowledge cutoff. Output names activation as blocked. No llm mode — bulk synchronous provider calls are banned; llm scoring arrives with the queued worker (point 5, next PR).Tests
Full light suite: 787 passed, 12 skipped (includes new set-matching, pre-0200 catalog-probe, snapshot-digest, and provenance-stamping tests; parameter-recovery runs with fast-mlsirm, skips honestly without).
Sequencing
Phase C (queued judge scoring per the no-bulk-sync directive, issue #289) follows stacked; activation (one-line authorized-anchor addition) lands only with ADR 0200's acceptance.
🤖 Generated with Claude Code
https://claude.ai/code/session_015HgzNGsCxqiTaT4YuJEb5J