Calibration script fixes for issue #39 validation on constrained hardware - #348
Closed
voorhs wants to merge 1 commit into
Closed
Calibration script fixes for issue #39 validation on constrained hardware#348voorhs wants to merge 1 commit into
voorhs wants to merge 1 commit into
Conversation
…ware Changes needed to produce the laptop-6GB calibration deliverables: calibrate_advisor.py: - Record is_feasible, headroom, severity_by_metric, and resolved model_name per driver on each CalibrationRow — without these a calibration JSON cannot answer the one question the advisor exists to answer, and a local preset swap can masquerade as 'transformers-heavy' - Add inter-preset GPU leak detection (_LEAK_WARN_GB) to surface leftover VRAM that corrupts the next preset's measurement - Fix cli_smoke divergence note: the ~10x drift was an apples-to-oranges n_trials artifact, not a wrapper regression run_calibration_banking77.sh: - uv run preset-discovery fix New helper scripts: - phase1b_metadata_counterfactual.py: re-predict with correct model metadata - phase3_reduce_to_fit.py: exercise the reduce-to-fit path on real hardware - render_issue39_tables.py: render definition-of-done tables from JSONs - run_phase2_isolated.sh: one preset per process to avoid GPU contamination Advisor source code is left untouched — this is validation, not fixes.
Collaborator
Author
|
Superseded — closing. The calibration harness this PR fixes has been moved out of the library repo entirely. It was a validation instrument, not library code, and one of its files ( The harness now lives beside the results it produced, in the experiments repo alongside Nothing is lost: the code remains in this branch's history at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Script-level changes needed to produce the laptop-6GB calibration deliverables for AutoIntent-experiments#39. The advisor source code is left untouched — this is validation, not fixes.
Changes
scripts/calibrate_advisor.pyis_feasible,headroom,severity_by_metric, and resolvedmodel_nameper driver on eachCalibrationRow. Without these, a calibration JSON cannot answer the one question the advisor exists to answer, and a local preset swap can masquerade as "transformers-heavy"._LEAK_WARN_GB = 0.25): warns when VRAM still allocated after a preset finishes, which corrupts the next preset's measurement. This caught a false classic-light OOM during validation.cli_smokedivergence note: the ~10× drift was an apples-to-orangesn_trialsartifact (the CLI has non_trialsflag), not a wrapper regression. Updated the note to reflect this.scripts/run_calibration_banking77.shuv runpreset-discovery fix.New helper scripts
phase1b_metadata_counterfactual.py— re-predicts with correct model metadata to test whether verdicts are fallback artifactsphase3_reduce_to_fit.py— exercises the reduce-to-fit path on real constrained hardware (the calibrator runs report-only, so this needs a separatePipeline.fit(..., preflight=...)call)render_issue39_tables.py— renders the definition-of-done tables straight from the run JSONsrun_phase2_isolated.sh— runs one preset per process to avoid inter-preset GPU contaminationContext
Validation ran on an RTX 3060 Laptop (6 GB VRAM, 16 GB RAM). All 4 fitted presets matched their advisor prediction. See the full write-up in the experiments repo PR.