Add Dataset Benchmarking view with quality scores and embeddings - #55
Merged
Conversation
… to Titlecase using a labelOverride file except for the Crop Type filter due to excess crops.
…as laggier due to each point being a DOM element. Switched to scattergl for performance and smoothness. 2. Added metric explanation tooltip for interpretation
…he visualization for dataset benchmarking. 3. Added tooltip/help explaning significance for each metric
Implements the axis scoring math from SCORING_FORMULAS.md in its own module, and extends the benchmark data types with reproducibility info, phases_completed, and the phase 3 metrics (dataset cartography, class confusability, label noise). Updates results.json to the new schema and refreshes the watermelon dataset's benchmark run with real phase 2 data.
…ards Adds an overall + per-axis score gauge (computed via the new scoring module) and a collapsible reproducibility details panel to the top of the benchmark view, matching the updated design. Metric cards are now grouped under their scoring axis (Structural Quality, Content Difficulty, Diversity & Coverage, Annotation Reliability) instead of one flat grid. Adds three new metric cards for when phase 3 data is available: Class Confusability (confusion matrix + top confused pairs), Dataset Cartography (easy/ambiguous/hard proportion bar), and Label Noise (per-class bars). Resolution Consistency now spans two columns to match the reference design. Introduces a shared three-tier good/mid/poor color scale (a new --agml-caution-text token alongside the existing primary/warning colors) used by both the score boxes and the cartography proportion bar. Verified in light and dark mode, on a 4-class and a 30-class dataset, and at mobile width.
…on run The watermelon benchmark now has a complete phase 3 run (100% confusability accuracy, zero label noise), which meant the confusion matrix's top-pairs list can legitimately be empty. Show a plain-language fallback instead of an empty list under a dangling header in that case.
…ation header The watermelon dataset's phases_completed field had been dropped from results.json during editing, which silently zeroed out every axis score since the scoring math gates on it — restoring it as [1, 2, 3] since cartography, confusability, and label noise data are all present. Metric cards now stretch to match the tallest card in their row instead of sizing to their own content, so a short stat card next to a tall bar-chart card reads as one aligned row. Also adds the missing 'Visualization' axis header above the embedding scatter panel, matching the other axis sections.
The embed model name now renders as the same pill tag used elsewhere in the modal instead of plain muted text. Also removes the marker outline on the 3D scatter plot — in a dense cluster, hundreds of overlapping translucent ring outlines were compounding into a solid dark smear instead of separating anything, which is exactly the WebGL rendering trap the outline was meant to avoid on the 2D view.
Removing the outline entirely (previous fix) traded the black-smear problem for the opposite issue: dense clusters flattened into a solid opaque blob with no way to tell individual points apart. Bring back a very low-alpha ring plus a lower fill opacity so a couple of overlapping points still get a crisp edge and dense regions read as a layered point cloud instead of either extreme.
A long label like 'Height (mean±std)' wraps to two lines in a narrow stat grid column. With align-items: center, that centered the info icon in the gap between the two lines instead of next to either one, and since nothing reserved space for the wrap, that tile's value sat lower than its neighbors' in the same row. Align the icon to the top line instead, and reserve two-line height on every label so values stay on one row regardless of which label happens to wrap.
The previous fix used flex to align the info icon, but a flex row lays the
icon out beside the whole (possibly two-line) label box wherever its edge
lands — which is exactly what put the icon in the dead space toward the
next column. Switch to plain inline flow so the icon wraps right after the
label's own last word, like any other inline element.
Also shorten 'Height (mean±std)' to 'Height (μ±σ)', matching its siblings'
brevity ('Width (mean)', 'Aspect ratio') — it was the one label consistently
tight enough to wrap in the first place, and the info tooltip already spells
out what it means in full.
Static, non-data-driven modal explaining how the overall and per-axis quality scores are computed, opened from a new 'How scores are calculated' link on the benchmark view. Documents the Image Classification formulas from SCORING_FORMULAS.md in plain language, with task-type tabs for Object Detection and Image Segmentation showing a placeholder until those formulas are published. Rendered via a portal to document.body since the dataset modal's flip-card uses transform/perspective for its 3D effect, which creates a new containing block for position: fixed descendants and would otherwise center this modal inside that transformed box instead of the viewport. Adds two axis-accent tokens (blue, pink) alongside the existing green/amber ones for the four scoring axes, and @types/react-dom as a devDependency.
…r icon Moves the 'How scores are calculated' trigger next to the flip-back button in the modal header instead of sitting alone below the score row — it's only relevant once you're on the benchmark face anyway. Renames 'Back to Details' to 'Dataset Details' to match the plain-label style of 'View Benchmarks', and replaces the colorful bar-chart emoji with a small monochrome SVG icon that fits the header's existing glyph language.
Reorders the header so the scoring-methodology trigger reads before the flip toggle, and gives 'Dataset Details' a small back-arrow icon to match the bar-chart icon on 'View Benchmarks' now that neither button uses plain text arrows.
…trics and pipeline changes
Matches the task type used for VLM datasets elsewhere in the catalog. Falls through to the existing placeholder path since no metric formulas are published for it yet.
Resolves conflicts across 8 files from ~21 commits of divergence (leaderboard UI, semantic search, location/metadata enrichment, new datasets on main vs. the benchmarking feature + VLM dataset support on this branch): - package.json / package-lock.json: kept dependencies/devDependencies from both sides (plotly + react-plotly.js + @types/react-dom from this branch, @huggingface/transformers + @orama/orama + tsx + test scripts from main), regenerated the lockfile via npm install rather than hand-merging it. - .gitignore: kept both branches' additions (performance_staging/, .claude/). - src/lib/datasets.ts: kept this branch's multi-platform array support and VLM fields (dataset_type, qa_type, task_dimensions, etc.) alongside main's new location fields (country, lat_lon, imaging_equipment, collection_period) and its crop_types casing fix — combined rather than picking one side. - src/components/DatasetMetadataModal.tsx/.module.css: kept this branch's benchmark flip-card, scoring methodology modal, and header changes, combined with main's new Location & Collection section and Citation block. - src/pages/datasets/index.tsx: kept this branch's title-cased label upgrade and VLM dataset_type filtering, took main's newer formatPrimaryLocation in place of the older formatDisplayLocation, wired in main's semantic search. - static/data/hf_datasets.json: main did a genuine data-enrichment pass on the 227 datasets shared with this branch (fuller location detail, casing fixes, new metadata fields) — verified by sampling several conflicts, then took main's version wholesale for those rather than hand-resolving field by field. Separately identified and de-duplicated 7 records that main renamed (gUAVa -> guava, UAV -> uav casing fixes) which would otherwise have been double-counted as new records, then combined with the 37 datasets unique to main and the 6 VLM datasets unique to this branch. Verified programmatically that the merged file contains every main record plus exactly those 6 additions, with no data loss on either side. - src/lib/semanticSearchIndex.ts + tests/semantic-search-index.test.ts: fixed fallout from the platform type change (string -> string[]) that the above datasets.ts merge introduced, which main's semantic search indexing code and its test fixture weren't written against. Verified with a clean tsc --noEmit, the full test suite (26/26 passing), and a live dev server smoke test confirming both the benchmarking flip-card and main's leaderboard/semantic-search/location-detail features work together with no console errors.
… tones Metric grid now uses 4 explicit tracks with each card spanning 2 (2 cards per row) instead of auto-fit, giving cards enough width that paragraphs stop wrapping into a narrow ladder. Added a 900px breakpoint so cards drop to one per row before the general 720px mobile breakpoint, since two cards get too cramped on tablet widths first. Also shifted the light-mode caution and warning tokens toward yellow — caution was a muddy amber (hue 75) and warning was a flat brown hex, both of which read as brown rather than yellow in the score boxes and penalty card.
Warning and caution were both landing in the amber/gold family after the last pass, which left the Hard/poor tier too close to the Ambiguous/fair one to tell apart at a glance. Warning is now a clear red so the three tiers read as an actual green/yellow/red scale, matching how a reader expects severity levels to differ in color, not just lightness. Dark mode untouched.
taskBadgeClass() here never got the VLM branch that the datasets listing page's own copy of this function already has, so image-text-to-text tasks fell through to the generic 'Other' gray badge instead of the dedicated purple --agml-badge-vlm-* tokens that already existed for this. Ported the same check and added the matching .badgeVlm rule, which was likewise only defined on the listing page's CSS module.
Dark mode's --agml-warning-text was still the old tan/amber color from before the light-mode red fix, so Hard and Ambiguous were just as hard to tell apart there as they were in light mode. Same green/yellow/red fix, bumped for dark backgrounds. The Locations list under Location & Collection rendered site.location raw instead of routing it through toTitleCase like the Country field next to it does, so lowercase source values (e.g. 'multinational') stayed lowercase. Also added a 'usa' override while in labelOverrides.ts, since the general auto title-case would otherwise turn 'USA' into 'Usa' wherever country goes through the same formatDisplayLocation path.
… package-lock.json
The lockfile committed in 15bcdc7 was generated with npm 10 on Node 22, which silently drops optionalDependencies-of-optionalDependencies (e.g. the per-platform @rspack/binding-*, @swc/core-*, and lightningcss-* packages declared by @rspack/binding rather than @rspack/core directly) from the lockfile instead of recording them as unmet-platform optional entries. That made npm ci fail on CI with dozens of 'Missing from lock file' errors even though package.json and package-lock.json's direct dependencies matched. Regenerated from a clean node_modules using Node 24 / npm 11 (matching the engines field in package.json), which records the full multi-platform optional dependency set correctly. Verified with npm ci --dry-run.
js2552
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a full benchmarking view to the dataset detail modal for image classification datasets: a flip-card back face with per-metric quality cards, an overall + per-axis score gauge, and an interactive 2D/3D UMAP embedding viewer. Also adds a static "Scoring Methodology" reference modal documenting how each score is computed.
What's included
Dataset benchmarking view
2D/3D embedding visualization
EmbeddingPlot2D/EmbeddingPlot3Dusing react-plotly.js which has theme-aware chrome.Scoring Methodology reference modal
ScoringMethodologyModal) explaining every formula in plain language, with task-type tabs.Other changes on this branch