Feat/benchmark metrics - #102
Open
Safaael25 wants to merge 16 commits into
Open
Conversation
Track CPU/memory/GPU utilization and queries/tokens-per-second throughput alongside the existing hit-rate/latency/accuracy metrics, write results to CSV as well as JSON, and add fully offline unit/integration tests so the instrumentation reruns on every commit without needing a full-scale, paid benchmark run in CI. Also commits a sample results log as documented in benchmarks/ReadMe.md.
benchmarks.benchmark.Benchmark subclasses unittest.TestCase and defines test_run_benchmark(self, max_samples), an existing quirk unrelated to this PR. Importing it at module level in a test_*.py file made pytest collect and try to run it as its own bare test case, which fails because pytest has no way to supply the required `vcache` constructor argument. Moving the import inside setUp() keeps it out of the module's top-level namespace so pytest's collector never sees it.
- Compute QPS/TPS from summed per-query latency instead of the benchmark loop's wall-clock time, which was dominated by harness overhead rather than simulated model latency - Initialize NVML once and cache the device handle instead of calling nvmlInit() on every sample; shut it down via atexit - Declare psutil as a project dependency (was used but undeclared) - Shut down VerifiedDecisionPolicy's executor/callback thread in the benchmark smoke test's tearDown, fixing a background-thread leak - Strengthen resource metrics and smoke tests to assert exact computed values instead of just presence/bounds
CostAwareEvictionPolicy never received a meaningful cost value during benchmark runs: cost was measured as wall-clock time around BenchmarkInferenceEngine.create(), which returns its pre-set response instantly, so every cache miss recorded a near-zero cost regardless of the dataset's actual recorded LLM latency. - BenchmarkInferenceEngine.set_next_response() now accepts an optional cost, exposed as next_cost. - VerifiedDecisionPolicy prefers the inference engine's reported cost over wall-clock measurement when available, falling back to the original wall-clock timing otherwise (no behavior change for live engines). - benchmark.py now passes the dataset's recorded LLM generation latency through as the injected cost on cache misses. - Added mean/p95/p99 latency and a hit_rate field to the results JSON, alongside the existing throughput metrics. - Added a CostAwareEvictionPolicy vs. LRUEvictionPolicy comparison to RUN_COMBINATIONS: same dataset/model/sample count, with max_size set low enough to actually trigger eviction (unlike the existing combinations, which keep max_size far above sample count). LRU (not MRU) is the correct baseline since CostAwareEvictionPolicy reduces to plain LRU at cost_weight=0.
…tests - RUN_COMBINATIONS now compares CostAwareEvictionPolicy against LRU, MRU, FIFO, and SCU (not just LRU), all on the same dataset/model/sample count/max_size, so the benchmark reports how cost-aware eviction stacks up against every eviction strategy in the codebase. - Added deterministic unit tests for CostAwareEvictionPolicy covering the cost_weight=0/1 extremes (equivalence to LRU / pure cost-driven eviction), invalid cost_weight clamping, direct validation of the normalization/priority formulas, and negative/zero-cost inputs.
CI's ruff format --check step was failing on the two _generate_and_measure_cost() call sites in process_request() (added in the cost-signal fix) because they exceeded the line-length that would have collapsed them onto one line. Reformatted with ruff 0.11.7 (the version pinned in pyproject.toml/poetry.lock) to match CI exactly.
Addresses luis-gasparschroeder's remaining review comment: drop the SCU-baseline aside and throughput-derivation comment in benchmark.py, and trim the CI rationale paragraph in ReadMe.md.
Introduces ARCEvictionPolicy, splitting the cache into recency (T1) and frequency (T2) lists with an adaptive victim split driven by a promotion-based proxy for ARC's ghost-list hits. Wires it into the package's public exports alongside the existing eviction strategies.
…tion Wraps any EmbeddingEngine with thread- or process-pool dispatch that coalesces duplicate in-flight requests and batches concurrent calls, improving throughput for local/CPU-bound models and reducing redundant computation under concurrent vcache.infer() load. Adds default get_embeddings()/get_engine_factory() to the base EmbeddingEngine class and batched get_embeddings() support to LangChain and OpenAI engines.
Add ConcurrentEmbeddingEngine for batched, concurrent embedding execution
Force-added despite the repo's blanket *.json gitignore rule so the workload backing the ARC vs LRU/MRU/FIFO/SCU/CostAware sweep is reproducible from the repo.
Adds `benchmarks/your_datasets/workload_short.json` and its companion `workload_short_INFO.json`.
Consolidates the generate_*/build_* dataset-builder scripts (previously scattered in benchmarks/your_datasets/ alongside the run_* benchmarks) into their own directory with a README documenting each script's output and command. Output paths updated to still write into benchmarks/your_datasets/; verified each script reproduces byte-identical JSON after the move. Also gitignores the generated JSON files explicitly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces internal "advantage"/"cost" jargon with plain, real-world terminology across the generator scripts in dataset_generators/ and the run_*.py sweeps that consume their output (dataset filenames, results directory names, and the benchmark.filepath provenance label): cost_advantage_demo(_200) -> cost_sensitive_retention(_large) dual_advantage_dataset(_v2) -> mixed_cost_recency_workload(_v2) full_repeat_dataset(_v2) -> hot_cold_repeat_workload(_v2) high_locality -> temporal_locality_workload Regenerated every renamed JSON output and diffed it byte-for-byte against the pre-rename file before removing the old copy, so no dataset content changed - only names. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mirrors the Poetry dependency groups used in CI (dev, benchmarks) so the repo can be built and its tests/benchmarks run without a manual Python/Poetry setup, per the assignment's reproducibility deliverable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.