Fix Backup Inventory 503 from unsupported Cosmos GROUP BY aggregate

[Paul Lizer (paullizer)]

Fixes #1294

Problem

Backup Inventory in Admin Settings → Data Management always failed to load. GET /api/admin/data-management/backups returned 503, the panel showed Backup history is temporarily unavailable. Please try again., and every tile rendered 0 — even with successfully completed backups.

Root Cause

_get_data_management_backup_global_summary() computed its counts with:

SELECT c.backup_type, c.status, COUNT(1) AS count FROM c
WHERE c.type = @type AND c.operation = @operation
GROUP BY c.backup_type, c.status

The azure-cosmos Python client does not advertise support for GroupBy combined with a non-VALUE aggregate (COUNT(1) AS count). Cosmos rejects the request during query plan negotiation:

(BadRequest) {"code":"BadRequest","message":"Query contains the following features,
which the calling client does not support:\nNone GroupBy NonValueAggregate ..."}

That BadRequest propagated out of _query_data_management_history_items and surfaced as a generic 503.

This was not throttling, indexing, or capacity — Cosmos Maintenance consistently reported the composite index as Aligned / 0 missing / 7 containers checked, and the 400 entries with requestCharge = 0 in AzureDiagnostics are the benign cross-partition query-plan negotiation. Because the aggregate shipped with the summary feature, Backup Inventory had never worked in any deployment.

Why tests did not catch it

FakeHistoryContainer implemented GROUP BY c.backup_type, c.status in Python and returned grouped rows. The test double supported a query shape the real client cannot serve, so the suite passed against a query that always failed in production.

Changes

• functions_data_management.py — added _count_data_management_backups() and rewrote the summary to issue six bounded SELECT VALUE COUNT(1) queries (total, available, running, failed, full, partial). VALUE aggregates are supported by the client and already used elsewhere in this module.
  • Deliberately avoided projecting every backup row and aggregating client-side, which would grow unbounded as history accumulates.
  • The two SELECT TOP 1 * latest-backup queries were already valid and are unchanged.
• admin_settings.html — added a hover tooltip and an (i) toggle that expands inline guidance next to Run Retention Cleanup, covering what gets deleted, what is protected, the 25-per-run cap, and that found no expired backups to delete is expected rather than an error.
• test_data_management_history_pagination.py — FakeHistoryContainer now raises the real BadRequest ... GroupBy NonValueAggregate error instead of emulating GROUP BY, plus two new tests.
• Version bump, fix doc, and release notes.

Validation

• functional_tests/test_data_management_history_pagination.py — 14 passed
• Data Management suite (-k data_management) — 159 passed, only the known pre-existing test_backup_recovery_and_admin_progress_are_bounded_and_sanitized failure

Regression probe: restoring a grouped aggregate fails two tests with the exact production error, confirming they fail for the right reason:

RuntimeError: (BadRequest) Query contains the following features, which the calling client
does not support: GroupBy NonValueAggregate.

Impact

Behavior	Before	After
GET /api/admin/data-management/backups	503	200
Backup Inventory tiles	Always 0	Actual counts
Backup Inventory table	Error banner	Backup rows
Retention cleanup purpose	Undocumented in UI	Tooltip + expandable help