feat: aggregate statistics from stored telemetry

[BKPepe]

stats.php shows individual telemetry results, but gives no way to see what the
collected data adds up to. This adds a monthly aggregate report so operators can
see the distribution of the tests stored on their server.

The report covers download/upload distributions, ping/jitter, client type,
IPv4/IPv6 and country. Results are cached, public reporting is opt-in, and
groups with fewer than 100 tests are not reported separately.

Light theme

Why

The CLI is packaged for OpenWrt, and Turris OS runs it on a schedule rather than
on demand, so a router accumulates measurements instead of producing one at a
time. Those routers are not a single platform — Turris Omnia, MOX and Omnia NG
all run it, and the 32-bit PowerPC Turris 1.x cannot run the Go client at all,
which is why a Rust port exists (openwrt/packages#30186).

That raises a question an operator currently cannot answer: what do all those
tests add up to, and is the hardware itself limiting what they show?

It also matters to the people sending the data. Users ask what telemetry is
collected, where it goes, and where they could see it. A published aggregate
answers all three at once, and answers them with the data rather than with a
paragraph of assurance.

None of this is specific to Turris. Anyone collecting telemetry eventually wants
context for the population behind it, and this provides that without inventing
information the telemetry does not contain — there is no notion of DSL or fibre
in it, and none is guessed at.

Notes

Aggregation is done in PHP because the measurement columns are text, malformed
historical values exist, and percentile support differs between the supported
database backends. On SQLite, MAX(dl) over '99.9', '312.5', '1024.0',
'' and 'abc' returns 'abc', and AVG(dl) returns 287.28 where the mean of
the real values is 478.8. Percentiles use a sparse histogram, and completed
months are cached permanently.

Malformed measurements are retained as rows but excluded from aggregates.
Individual results, IP addresses and raw User-Agent strings are never published.

Which month is current is taken from the database rather than from PHP, because
the backends disagree on what their timestamp column records: SQLite's
CURRENT_TIMESTAMP is UTC, PostgreSQL's now() and MSSQL's getdate() are
server-local.

Tested against seeded SQLite telemetry, including malformed measurements and
privacy-threshold edge cases. tests/telemetry_stats_test.php covers the
helpers directly and is wired into npm test, which was previously a stub.

Follow-up worth considering

The client breakdown can only say what a client calls itself. Extending the CLI
to name its platform in the User-Agent — librespeed/speedtest-cli#131 does this
for the Go client — would let this report show which architectures are measuring
against a server, with no change to what telemetry stores.

---

@ljelinek-cznic, would you be willing to try this on librespeed.turris.cz? The
figures above come from seeded data, and what I cannot check here is whether
they read sensibly against a real month: whether the client split matches what
you would expect, whether the country parsing copes with your ispinfo, and how
long stats_build.php takes on a table that size.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Add cached monthly aggregate telemetry statistics (with optional public report)

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add cached monthly telemetry aggregates (speed, latency, client, IP family, country) to stats UI.
• Provide opt-in, read-only public report and CLI cache builder suitable for cron.
• Normalize malformed measurements to NULL and add timestamp indexes for faster queries.

Diagram

graph TD
  A["Operator"] --> B["stats.php (auth)"] --> D["stats_summary.php"] --> F[("speedtest_users")]
  B --> E["stats_render.php"]
  C["Public visitor"] --> G["stats_public.php"] --> D --> H["Cache JSON files"]
  I["Cron"] --> J["stats_build.php (CLI)"] --> D
  D --> K["telemetry_db.php"] --> F

  subgraph Legend
    direction LR
    _u["User/Caller"] ~~~ _p["PHP endpoint/script"] ~~~ _db[("Database")] ~~~ _c["Cache file"]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Database-side aggregation (views/materialized tables)

• ➕ Potentially faster on very large datasets with proper numeric columns
• ➕ Leverages DB-native percentile/histogram features where available
• ➖ Not portable across supported backends (SQLite lacks needed percentile features)
• ➖ Telemetry measurement columns are TEXT; casting risks errors or incorrect results on malformed data
• ➖ Harder to keep consistent behavior across MySQL/Postgres/MSSQL/SQLite

2. Precompute-only pipeline (cron required; no on-demand rebuild)

• ➕ Predictable DB load; never rebuilds from web requests
• ➕ Simplifies request path for stats.php and stats_public.php
• ➖ Authenticated stats page becomes stale unless cron is configured
• ➖ Worse out-of-box experience for operators who only use the UI

3. Store normalized numeric columns alongside raw text

• ➕ Future queries/aggregates become cheaper and more reliable
• ➕ Allows more SQL-side aggregation safely
• ➖ Schema migration complexity across backends
• ➖ Requires backfill strategy and dual-write behavior
• ➖ Doesn’t address historical malformed text rows without careful handling

Recommendation: The PR’s approach (PHP-side normalization + sparse histograms + caching, with an opt-in public endpoint that never queries the DB) is the best portability/safety tradeoff given multi-DB support and TEXT measurement columns. Consider the “normalized numeric columns” option only as a future evolution if telemetry volume grows enough to justify a migration.

Files changed (14) +1390 / -2

Enhancement (5) +1053 / -0

stats.phpAdd aggregate statistics section to authenticated stats page +29/-0

Add aggregate statistics section to authenticated stats page

• Loads monthly summaries via the new aggregation module, adds a month picker, and renders the cached report. Includes shared head rendering for styles/theme toggle and links to the public report when enabled.

results/stats.php

stats_build.phpAdd CLI cache builder for monthly summaries (cron-friendly) +60/-0

Add CLI cache builder for monthly summaries (cron-friendly)

• Introduces a CLI-only script to (re)build cached monthly summaries for specified months or default current/previous month. Outputs basic build metrics and enforces non-web execution.

results/stats_build.php

stats_public.phpAdd opt-in public monthly statistics endpoint served from cache +73/-0

Add opt-in public monthly statistics endpoint served from cache

• Adds a public report page gated by $stats_public_report that only reads cached summaries and never triggers DB work. Supports selecting months and listing other available cached months.

results/stats_public.php

stats_render.phpImplement shared HTML/CSS rendering for telemetry summaries +372/-0

Implement shared HTML/CSS rendering for telemetry summaries

• Adds formatting helpers and a reusable renderer that produces cards/tables for distributions, clients, IP families, and countries. Includes inline styling and a theme toggle stored in localStorage for consistency across pages.

results/stats_render.php

stats_summary.phpImplement monthly aggregation, privacy thresholding, and cache logic +519/-0

Implement monthly aggregation, privacy thresholding, and cache logic

• Adds PHP-side aggregation using sparse histograms for percentiles and exact means, avoiding DB-specific percentile functions and TEXT-cast pitfalls. Implements client/country/family grouping with a privacy threshold and JSON caching with limited refresh for the current month.

results/stats_summary.php

Bug fix (2) +127 / -0

index.phpHarden result formatting against malformed numeric fields +9/-0

Harden result formatting against malformed numeric fields

• Guards number formatting by coercing non-numeric values to 0 to avoid PHP 8 TypeError and 500 responses when historical rows contain empty/non-numeric measurements.

results/index.php

telemetry_db.phpNormalize telemetry measurements and add timestamp-based query helpers +118/-0

Normalize telemetry measurements and add timestamp-based query helpers

• Normalizes dl/ul/ping/jitter to float or NULL at insert time to prevent malformed values breaking rendering or aggregates. Adds SQLite timestamp index creation, a portable getDatabaseNow() helper, and a streaming getSpeedtestUsersBetween() query used for monthly aggregation.

results/telemetry_db.php

Tests (1) +167 / -0

telemetry_stats_test.phpAdd plain-PHP unit tests for aggregation helpers +167/-0

Add plain-PHP unit tests for aggregation helpers

• Adds a lightweight test harness covering measurement normalization, client/family/country classification, histogram percentile logic, threshold folding, and month bound calculation. Exits non-zero on failures and is runnable via npm test.

tests/telemetry_stats_test.php

Documentation (1) +16 / -0

doc.mdDocument monthly aggregate telemetry statistics and public reporting +16/-0

Document monthly aggregate telemetry statistics and public reporting

• Adds an 'Aggregate statistics' section describing the monthly summary, cache behavior, opt-in public report, and suggested cron job. Documents privacy threshold behavior and confirms no raw identifiers are published.

doc.md

Other (5) +27 / -2

package.jsonWire PHP telemetry stats tests into npm test +1/-1

Wire PHP telemetry stats tests into npm test

• Replaces the placeholder test script with a direct call to the new PHP test runner for telemetry aggregation helpers.

package.json

telemetry_mssql.sqlAdd timestamp index to MSSQL telemetry schema +6/-0

Add timestamp index to MSSQL telemetry schema

• Introduces a nonclustered index on the timestamp column to speed filtering/ordering for result listing and aggregation.

results/telemetry_mssql.sql

telemetry_mysql.sqlAdd timestamp index to MySQL telemetry schema +2/-1

Add timestamp index to MySQL telemetry schema

• Extends the table alteration to include an index on the timestamp column in addition to the primary key.

results/telemetry_mysql.sql

telemetry_postgresql.sqlAdd timestamp index to PostgreSQL telemetry schema +7/-0

Add timestamp index to PostgreSQL telemetry schema

• Adds a btree index on the timestamp column to improve performance for time-range scans used by stats and listings.

results/telemetry_postgresql.sql

telemetry_settings.phpAdd statistics configuration (title, public report flag, cache dir) +11/-0

Add statistics configuration (title, public report flag, cache dir)

• Introduces settings for statistics page title, opt-in public reporting, and configurable cache directory, with sensible defaults and inline guidance.

results/telemetry_settings.php

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

1. Empty cache on JSON failure 🐞 Bug ☼ Reliability

Description

statsMonthlySummary() writes the cache without checking whether json_encode() succeeded, and a
0-byte file_put_contents() return is treated as success, so a failed encode can replace the cache
with an empty/invalid file and force rebuilds on every request. This can repeatedly trigger
full-month table scans for authenticated views/cron until the cache can be written correctly.

Code

results/stats_summary.php[R511-514]

+        // Written through a temporary file so a reader never sees half of it.
+        $tmp = $file.'.'.getmypid().'.tmp';
+        if (false !== @file_put_contents($tmp, json_encode($summary))) {
+            @rename($tmp, $file);

Evidence

The cache writer uses json_encode($summary) inline and treats any non-false file_put_contents()
return as success, then renames the temp file into place; this can overwrite the cache with invalid
content if encoding/writing does not produce a valid JSON document.

results/stats_summary.php[477-516]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`statsMonthlySummary()` writes cached summaries using `file_put_contents($tmp, json_encode($summary))` and only checks for `!== false`. If `json_encode()` fails, the write may still “succeed” with 0 bytes and the code will `rename()` the empty temp file over the real cache, causing every later call to rebuild.

### Issue Context
The caching logic is meant to prevent repeated table scans. A corrupted/empty cache file defeats that and can keep the system in a rebuild loop.

### Fix Focus Areas
- results/stats_summary.php[507-516]

### Suggested fix
- Encode first, check for a string result, and only rename when bytes written are `> 0`.
- Optionally use `JSON_THROW_ON_ERROR` and catch `JsonException` to avoid silently corrupting the cache.
- If encoding/writing fails, delete the temp file (best-effort) and keep the previous cache intact.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Current month exposed publicly 🐞 Bug ≡ Correctness

Description

stats_public.php documents that the reported month is always finished, but it accepts any YYYY-MM
from $_GET['month'] and will serve the current month if a cache exists. This breaks the stated
contract and can unintentionally publish in-progress (shifting) aggregates.

Code

results/stats_public.php[R22-26]

+// Default to the most recent finished month.
+$month = statsPreviousMonth();
+if (isset($_GET['month']) && is_string($_GET['month']) && preg_match('/^\d{4}-\d{2}$/', $_GET['month'])) {
+    $month = $_GET['month'];
+}

Evidence

The file-level comment states the public report is for a finished month, but the implementation
allows overriding the month via query string and uses it directly when loading a cached summary.

results/stats_public.php[3-33]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`stats_public.php` claims it always reports a finished month, but it allows overriding the month via `?month=YYYY-MM` with no check that the requested month is strictly before `statsCurrentMonth()`.

### Issue Context
Even though visitors cannot trigger a build (`build=false`), the current-month cache can be created by `stats.php` or `stats_build.php`, after which it becomes publicly readable via `stats_public.php`.

### Fix Focus Areas
- results/stats_public.php[22-33]

### Suggested fix
- After parsing `$_GET['month']`, clamp it to `statsPreviousMonth()` when the requested month is `>= statsCurrentMonth()` (and also reject future months).
- Alternatively, if serving current month publicly is intended, update the module docblock/comment to match reality and consider explicitly labeling it as “partial month” in the UI.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context used

✅ Tickets: 🎫 Configurable default filesize for garbage download

Tip of the day

💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗