Add baseline-deviation database-state alert

[jakemorgangit]

Implements the database-state alert proposed in #1986. Thanks for the green light and the review notes — this addresses all of them.

What it does

Fires when a monitored database's current state deviates from its expected state, rather than on any non-ONLINE state (which would page every log-shipping secondary). database_config only captures state_desc at server-load, so a database going OFFLINE / SUSPECT / RECOVERY_PENDING / RESTORING after onboarding was invisible until now.

• A new per-minute database_states collector captures sys.databases.state_desc (+ is_in_standby) as a time series.
• Each database's expected state is captured automatically on first observation (a baseline), stored per (server, database), and is user-editable — change it to accept a new normal, or set (ignore) to opt a database out.
• Severity is CRITICAL for SUSPECT / RECOVERY_PENDING / EMERGENCY, WARNING otherwise. Per-database fire / cooldown / resolution, through the existing mute rules and delivery channels, plus the web viewer's alert history.

First-observation hole — closed (your review point)

A critical first observation does not baseline itself. Onboard a server while a database is SUSPECT and the engine will not learn SUSPECT as expected (which would go quiet through the outage, then fire on recovery). Instead:

• Non-critical first observation (ONLINE / OFFLINE / RESTORING / RECOVERING) → auto-accept baseline.
• Critical first observation (SUSPECT / RECOVERY_PENDING / EMERGENCY) → no baseline written; alerts immediately and stays pending until it recovers (auto-accepts the healthy state) or an operator sets an expected state.

"Pending" is modelled as the absence of a baseline row, so the deviation read is (no baseline AND current is critical) OR (baseline exists, not ignored, current != baseline). Covered by new store-level and engine-level tests.

Logistics from your checklist

• Migration claimed at PR time: database_states collector table + config.database_state_expected + the database_state_enabled column ship as V45; StorageVersion → 45; the viewer schema-probe gains a matching newest-first sentinel. Lite's DuckDB tables are catalog-generated / non-collector-added.
• New-collector tax: added to CollectorScheduleDefaults, the pinned collector set, and all six schedule preset tables (Lite s_presets + Darling CollectorSchedulePresets, @ 1/1/5); the two preset pin tests and the cross-app parity test pass.
• CI worker sizing: hypertable count 39 → 40, so build.yml / nightly.yml move to timescaledb.max_background_workers = 42, max_worker_processes = 53 (kept in step with CiClusterWorkerSizingTests).
• Grants: the viewer role's write on config.database_state_expected (for the override editor) is added to both Darling/tools/provision-roles.sql and DarlingManagedRoles, mirroring the custom_views model; SELECT is covered by the blanket config grant.
• Azure SQL DB: gated off via AppliesTo => !IsAzureSqlDb (Managed Instance stays on), matching the AG / system-health collectors.
• Based on dev; CHANGELOG entry added with a bottom link-ref; tests are in this PR.

UI / surfaces

A Configure… button in each desktop app's Settings → Alerts opens a per-database editor (current state, expected state, re-baseline, (ignore); pending databases are labelled). The web viewer auto-displays fired alerts. The MCP get_alert_settings / update_alert_settings expose the master toggle.

Testing

• New tests: engine fire / severity / cooldown / resolve + pending-critical first observation; real-DuckDB store round-trips for the seed / deviation / override CRUD including the pending-baseline behaviour.
• Full suites green on dev: Lite 2002, Darling 3936 (live-Postgres legs skipped locally). Builds clean, no warnings.

Happy to adjust anything to fit the project's direction.

[erikdarlingdata]

Thorough, high-quality contribution — thank you. You clearly read the house patterns: the V47 hand-create-table-plus-catalog-driven-hypertable idiom, the engine's family conventions (cooldown stamped even when muted, no fabricated recovery on fetch failure, resolution gated on enable), the first-observation fix implemented symmetrically in both stores with both halves pinned by tests, and a CHANGELOG entry in the right shape. I fetched the branch, read the full diff, and built the service chain locally (Alerting/Collectors/Storage/Notifications/Darling.Service: 0 warnings, 0 errors). Requesting changes for one mechanical reason (the fork is two collector-generations stale) and one behavioral gap worth closing while you're in here.

Blocking: rebase against current dev (dev moved twice since your base)

Your base ae278b72 predates the PVS alert (#1984) and this week's other merges, which took the exact surfaces this PR touches. After rebasing on dev:

• Migration 45 → 49, StorageVersion.SchemaVersion → 49, CHANGELOG text "schema v45" → v49. Dev is at 48; per the applier's ascent rule your V45 would silently skip on upgraded stores and the feature would be dead there (the tables would never exist).
• CI worker sizing: dev is already at timescaledb.max_background_workers = 43 / max_worker_processes = 54 (hypertable count 41). Your +1 makes it 44 / 55, not 42/53.
• The six schedule-preset tables and DarlingManagedRoles now carry pvs_stats rows and a restructured role model (mcp role writes config_alert_settings etc.) — your additions are right in spirit but will need re-applying against the new shape. Same for the probe: hasDatabaseStateAlert slots after hasCollectorState plus whatever #1984 added.
• git merge dev currently conflicts; CI never ran on this PR because GitHub can't build the merge commit. Once you push the rebase I'll approve the workflow runs promptly.

Behavioral: two transient-state scenarios that will page people

1. Instance restarts produce an alert storm (including CRITICALs). After any restart — patch window, failover, crash — databases pass through RECOVERY_PENDING and RECOVERING before ONLINE. The collector reconnects while crash recovery is still running (minutes, on busy instances), the engine's next sweep reads that latest snapshot, and every affected database fires: RECOVERING → WARNING, RECOVERY_PENDING → CRITICAL (it's in your critical set), each followed by a resolution notice minutes later. A monthly patch window across a fleet turns that into dozens of paired fire/resolve pairs for a completely normal restart.

Proposed fix: require the deviation to be present in the two most recent collections before firing (change the deviation read to join the two newest snapshots and require the mismatch in both). At the per-minute default that adds ~1 minute of latency to a genuine SUSPECT — cheap — while making restart transients invisible unless recovery actually sticks, which is exactly the case you do want to page on (a stuck RECOVERY_PENDING).

2. is_in_standby is collected but never read. Your collector doc says it's captured so a standby log-shipping secondary is "distinguishable from a genuinely stuck restore" — but nothing downstream uses it (I grepped: it exists only in the collector, schema, and golden-schema pin). Concretely: a STANDBY-mode secondary reports state_desc = ONLINE with is_in_standby = 1, so it baselines as ONLINE — then every periodic log restore flips it through RESTORING, and each flip the sampler catches fires a WARNING + later a resolution. Suggest composing an effective state — is_in_standby ? 'STANDBY' : state_desc — through the seed, the deviation read, and the override editor (plus a STANDBY token in the dropdown), so standby secondaries baseline as STANDBY; the two-sample rule from finding 1 then absorbs the restore window itself. Worth validating the exact state_desc/flag sequence against a live standby restore, which I'm happy to do on the next round. If you'd rather not model STANDBY at all, drop the column and the doc claim rather than shipping it write-only.

Doc nits (stale relative to your own fix)

• DatabaseStateInfo and the Lite store doc both still say a database "never alerts on first sight" — your pending-critical design (correctly) alerts on first sight for SUSPECT/RECOVERY_PENDING/EMERGENCY. Reword to "never alerts on a healthy first sight."
• DatabaseStateCollector's class doc has a leftover paragraph saying Azure SQL DB "needs no special path — the same single query runs everywhere," which contradicts the (correct) AppliesTo gate and its own doc right below.
• The engine's field comment says the cooldown key is {serverKey}{dbName}; the implementation (correctly) uses serverKey + "|" + dbName.

Minor, non-blocking

• Dropped databases leave their database_state_expected rows behind forever. Bounded growth, but a delete-on-absent sweep (or a note that the editor shows them for manual cleanup) would tidy it.
• The engine compares database names OrdinalIgnoreCase while both stores' joins are case-sensitive — a rename differing only by case would strand the old row and could seed a duplicate. Edge-case; noting for completeness.

What I verified against the checklist

First-observation hole: closed correctly in both stores (critical states never seed; pending rows alert via the no-baseline arm) with store- and engine-level tests. Collector tax: schedule defaults (1/30) ✓, pinned set ✓, presets ✓ (will re-conflict), sizing ✓ (numbers need the rebase update). Grants: viewer write on the override table matches the custom_views model ✓ (re-check placement after rebase). Azure gating: AppliesTo(!IsAzureSqlDb) with a reasoned doc ✓. Base dev ✓, CHANGELOG + bottom link ref ✓, tests in-PR ✓. Security scan of the full diff: clean.

After you push the rebase I'll approve CI, and before merge I'll run the whole path live against a Docker SQL Server (offline/suspect/recovery transitions) — that will also settle the standby question empirically.

[jakemorgangit]

Thanks for the thorough review — all points addressed and pushed. Summary:

Behavioral changes you asked for

• Two-sample rule — the alert now fires only when the deviation is present in the two most recent collections, so a restart's brief RECOVERING / RECOVERY_PENDING transient no longer false-fires. A critical state still can't be learned as a baseline (first-observation SUSPECT/RECOVERY_PENDING/EMERGENCY stays pending and alerts).
• Effective STANDBY state — instead of dropping is_in_standby, I model it: a read-only log-shipping secondary is compared at a single stable STANDBY rather than the RESTORING it flickers through on every log restore, so it baselines once and never churns. Added a non-critical STANDBY token and both stores compose is_in_standby ? 'STANDBY' : state_desc.
• Auto-baselines for databases that drop off the newest snapshot are now pruned (user overrides preserved).

Rebase onto current dev

• migration 45 → 49, schema version → 49
• CI worker sizing → 44 / 55 (HypertableCount 41 → 42)
• alert-settings ordinals: database_state_enabled slots after the three PVS columns (PVS growth alert and version-store trend chart (follow-up to #1951) #1984), realigned across the service config provider, the viewer alert-settings, and the MCP alert reader
• viewer schema-probe: the database-state arm is newest (returns 49, above the V48 PVS arm)
• schedule presets and pinned collector sets carry both plan_correction and database_states
• re-applied the config.database_state_expected viewer grant against the restructured managed-roles model

Doc nits + minor items

• Reworded the two-sample / "healthy first observation" language; removed the stale Azure paragraph in the collector (contradicted the !IsAzureSqlDb gate); fixed the engine cooldown-key comment and aligned per-db name comparisons to StringComparer.Ordinal.

Builds clean (0/0); Lite 2059 and Darling 3991 pass (the latter includes your new #2007/#2008 tests).