Update redfish grafana dashboard. - #2518
Conversation
📝 WalkthroughUpdated Redfish Grafana queries to aggregate metrics by WalkthroughThe Grafana Redfish dashboard now aggregates Prometheus results by ChangesRedfish dashboard query aggregation
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: 🟡 Moderate · up to The dashboard may merge data from different servers when aggregating by system_id, causing Redfish Up and Powered On panels to report incorrect results; one health query may also include duplicate scrape data. Merge should wait for the aggregation fix and health-query filter verification. 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Biome (2.5.6)etc/kayobe/kolla/config/grafana/dashboards/openstack/redfish.jsonFile contains syntax errors that prevent linting: Line 1: unexpected character Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: ac4adfc3-131f-46d9-b99c-92e7c32071ec
📒 Files selected for processing (1)
etc/kayobe/kolla/config/grafana/dashboards/openstack/redfish.json
Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.
📜 Review details
⏰ Context from checks skipped due to timeout. (4)
- GitHub Check: stackhpc/check
- GitHub Check: Tox releasenotes with Python 3.12
- GitHub Check: Ansible 2.20 lint with Python 3.12
- GitHub Check: Ansible 2.19 lint with Python 3.12
🔇 Additional comments (2)
etc/kayobe/kolla/config/grafana/dashboards/openstack/redfish.json (2)
330-330: LGTM!Also applies to: 900-900
3086-3086: 🗄️ Data Integrity & IntegrationApply the exporter-job filter to the averaged health query.
Line 3086 omits
job!="redfish-exporter-collectlog", unlike the otherredfish_chassis_healthqueries in this dashboard. If that job exposes the same metric,avgincludes both scrape targets and the panel still double-counts chassis. Prometheusavgincludes every matching sample. (prometheus.io)Add the matcher and verify the result with
count by (job) (redfish_chassis_health{server=~"$server"}).
| }, | ||
| "editorMode": "code", | ||
| "expr": "count(redfish_system_power_state{group=\"$group\", job!=\"redfish-exporter-collectlog\"})", | ||
| "expr": "count(redfish_system_power_state{group=\"$group\", job!=\"redfish-exporter-collectlog\"}) by (system_id)", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
file='etc/kayobe/kolla/config/grafana/dashboards/openstack/redfish.json'
printf '%s\n' '--- relevant dashboard sections ---'
sed -n '80,125p;165,210p;300,345p;875,920p;3055,3110p' "$file"
printf '%s\n' '--- panel options and query references ---'
rg -n -C 3 '"title"|"reduceOptions"|"calcs"|"values"|"expr": ".*redfish_(system_power_state|chassis_health)' "$file"Repository: stackhpc/stackhpc-kayobe-config
Length of output: 21416
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- redfish metric definitions and label usage ---'
rg -n -S -C 4 'redfish_system_power_state|system_id|redfish_chassis_health' . \
-g '!etc/kayobe/kolla/config/grafana/dashboards/openstack/redfish.json' \
-g '!*.lock' -g '!*.min.*' | head -n 500
printf '%s\n' '--- dashboard panels containing server-count or system-id aggregations ---'
rg -n -S -C 3 'count\(.*redfish_system_power_state|by \(server\)|by \(system_id\)|title": ".*(Up|Powered|Node|Health|Count)' \
etc/kayobe/kolla/config/grafana/dashboards/openstackRepository: stackhpc/stackhpc-kayobe-config
Length of output: 31094
🌐 Web query:
stackhpc redfish exporter redfish_system_power_state system_id server metric
💡 Result:
The metric redfish_system_power_state is commonly used in various Redfish Prometheus exporter implementations to report the power status of a managed server (ComputerSystem) [1]. It typically represents the current power state retrieved from the Redfish API (e.g., /redfish/v1/Systems/{system_id}) [2]. Key details regarding this metric include: Implementation and Usage - Metric Purpose: It is a gauge metric [3][2] that reflects whether a system is powered on or off. Implementations often map the Redfish PowerState string (e.g., "On", "Off") to numerical values (e.g., "On" = 1, "Off" = 0) [2]. - Identification: When exported to Prometheus, the metric is generally associated with labels that identify the specific server, such as the system_id, target address, or host name [3][4][2]. - Context: While different projects use slightly different naming conventions (e.g., redfish_system_power_state [1] vs. redfish_powerstate [3][2]), they all serve the purpose of monitoring out-of-band hardware power status [5][3]. StackHPC Specifics - The stackhpc/redfish_exporter is a Prometheus exporter designed to query Redfish-based servers (e.g., Dell, HPE, Supermicro) [5]. - In the context of the StackHPC Kayobe configuration, the exporter is enabled and configured to scrape BMC targets at specified intervals [6]. The specific metrics exposed depend on the version and configuration of the exporter deployed, as multiple open-source implementations exist under similar names (e.g., stackhpc [5], sapcc [3], dasbaum-ch [4]). To verify the exact metrics available in your specific deployment, you can query the /metrics endpoint of your running exporter: curl http://<redfish_exporter_host>:9610/metrics [5]
Citations:
- 1: https://repo1.dso.mil/big-bang/customers/airgap/-/blob/main/redfish-exporter/main.go?ref_type=heads
- 2: https://github.com/sapcc/redfish-exporter/blob/main/collector.py
- 3: https://github.com/sapcc/redfish-exporter/blob/main/README.md
- 4: https://github.com/dasbaum-ch/redfish-exporter
- 5: https://github.com/stackhpc/redfish_exporter
- 6: https://stackhpc-kayobe-config.readthedocs.io/en/stackhpc-2024.1/configuration/monitoring.html
🌐 Web query:
Prometheus aggregation operators count by label documentation
💡 Result:
In Prometheus (PromQL), aggregation operators such as count are used to aggregate elements of an instant vector [1][2]. To count by specific labels, you use the by or without clauses [1][2]. Aggregation Syntax Aggregation operators can be used with a by or without clause to control label preservation [1][3]: [without|by ()] () or () [without|by ()] by (): This clause preserves only the labels listed in the parentheses [1][2]. Even if other labels have identical values across all elements, they are dropped from the result [1][3]. without (): This clause removes the listed labels from the output, while preserving all other original labels [1][2]. Example To count the number of time series (e.g., running instances) grouped by a specific label such as app: count by (app) (instance_cpu_time_ns) [4] Other Relevant Operators count(v): Returns the number of elements in the vector [1][2]. count_values(l, v): Outputs one time series per unique sample value in v, creating a new label l whose value is the unique sample value and whose time series value is the count of how many times that sample value appeared [1][2]. count_over_time(v): A separate function (not an aggregation operator) that counts all samples for each series within a specified range vector [5][6]. Top Results: [1][2][4]
Citations:
- 1: https://prometheus.io/docs/prometheus/latest/querying/operators/
- 2: https://prometheus.io/docs/prometheus/3.13/querying/operators/
- 3: https://github.com/prometheus/prometheus/blob/f37442d2/docs/querying/operators.md
- 4: https://prometheus.io/docs/prometheus/latest/querying/examples/
- 5: https://prometheus.io/docs/prometheus/latest/querying/functions/
- 6: https://iximiuz.com/en/posts/prometheus-functions-agg-over-time/
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
import json, urllib.request
base = "https://api.github.com/repos/stackhpc/redfish_exporter"
for path in ("/git/trees/main?recursive=1", "/git/trees/master?recursive=1"):
try:
with urllib.request.urlopen(base + path, timeout=10) as r:
data = json.load(r)
print("tree", path, "status", data.get("sha"))
for item in data.get("tree", []):
name = item.get("path", "")
if any(x in name.lower() for x in ("collector", "metric", "redfish")):
print(name)
break
except Exception as exc:
print("tree lookup failed", path, type(exc).__name__, str(exc))
PY
printf '%s\n' '--- repository references if present locally ---'
rg -n -S 'stackhpc/redfish_exporter|redfish_exporter' . -g '!*.json' -g '!*.lock' | head -n 200Repository: stackhpc/stackhpc-kayobe-config
Length of output: 4595
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- local scrape labels and exporter configuration ---'
cat -n etc/kayobe/kolla/config/prometheus/prometheus.yml.d/60-redfish.yml
cat -n etc/kayobe/containers/redfish_exporter/redfish_exporter.yml
printf '%s\n' '--- configured image and version references ---'
rg -n -S -C 3 'jenningsloy318|redfish_exporter.*image|redfish-exporter|redfish_exporter.*version|redfish_exporter.*tag' etc doc releasenotes
printf '%s\n' '--- public source references ---'
for url in \
https://raw.githubusercontent.com/jenningsloy318/redfish_exporter/master/redfish_exporter.py \
https://raw.githubusercontent.com/jenningsloy318/redfish_exporter/main/redfish_exporter.py \
https://raw.githubusercontent.com/jenningsloy318/redfish_exporter/master/collector.py \
https://raw.githubusercontent.com/jenningsloy318/redfish_exporter/main/collector.py; do
echo "URL: $url"
curl -L --max-time 10 -k -sS "$url" | rg -n -C 5 'system_power_state|system_id|chassis_health|Metric|GaugeMetricFamily' | head -n 120 || true
doneRepository: stackhpc/stackhpc-kayobe-config
Length of output: 2885
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
import json, ssl, urllib.request
ctx = ssl._create_unverified_context()
repos = [
"stackhpc/redfish-exporter",
"stackhpc/redfish_exporter",
"jenningsloy318/redfish_exporter",
]
for repo in repos:
url = f"https://api.github.com/repos/{repo}/git/trees/main?recursive=1"
try:
req = urllib.request.Request(url, headers={"User-Agent": "review-inspection"})
with urllib.request.urlopen(req, context=ctx, timeout=15) as r:
data = json.load(r)
print(f"REPO {repo} TREE {data.get('sha')}")
for item in data.get("tree", []):
path = item.get("path", "")
if path.endswith((".py", ".go", ".yml", ".yaml")):
print(path)
except Exception as exc:
print(f"REPO {repo} ERROR {type(exc).__name__}: {exc}")
PYRepository: stackhpc/stackhpc-kayobe-config
Length of output: 631
🏁 Script executed:
#!/bin/bash
set -e
for path in collector/system_collector.go collector/redfish_collector.go collector/common_collector.go README.md; do
url="https://raw.githubusercontent.com/stackhpc/redfish_exporter/main/$path"
echo "--- $path ---"
curl -L --max-time 15 -k -sS "$url" \
| rg -n -C 8 'system_power_state|system_id|server|MetricDesc|NewDesc|Describe|Collect' \
| head -n 300 || true
done
printf '%s\n' '--- tags and release references ---'
curl -L --max-time 15 -k -sS \
-H 'Accept: application/vnd.github+json' \
https://api.github.com/repos/stackhpc/redfish_exporter/tags \
| head -c 12000Repository: stackhpc/stackhpc-kayobe-config
Length of output: 12666
🏁 Script executed:
#!/bin/bash
set -e
file='etc/kayobe/kolla/config/grafana/dashboards/openstack/redfish.json'
printf '%s\n' '--- changed query context ---'
git diff --unified=8 -- "$file" | sed -n '1,220p'
printf '%s\n' '--- exporter system collector implementation ---'
curl -L --max-time 15 -k -sS \
https://raw.githubusercontent.com/stackhpc/redfish_exporter/main/collector/system_collector.go \
| sed -n '130,172p'
printf '%s\n' '--- deterministic aggregation model ---'
python3 - <<'PY'
from collections import Counter
series = [
{"server": "srv-a", "system_id": "1", "value": 1},
{"server": "srv-b", "system_id": "1", "value": 1},
{"server": "srv-c", "system_id": "2", "value": 1},
]
by_system = Counter(s["system_id"] for s in series)
by_server = Counter(s["server"] for s in series)
print("count(metric) by (system_id):", dict(by_system))
print("mean of grouped values:", sum(by_system.values()) / len(by_system))
print("count(metric) by (server):", dict(by_server))
print("count(count(metric) by (server)):", len(by_server))
on = Counter(s["server"] for s in series if s["value"] == 1)
print("count(count(metric == 1) by (server)):", len(on))
PYRepository: stackhpc/stackhpc-kayobe-config
Length of output: 2987
Count servers in the overview panels.
The exporter emits redfish_system_power_state once per Redfish system, while the scrape configuration labels each target with server. by (system_id) can merge systems from different servers and returns one value per system_id; the stat panels then display or average those values.
Use count(count(redfish_system_power_state{...}) by (server)) at line 109 and count(count(redfish_system_power_state{...} == 1) by (server)) at line 192.
1 participant
Add
by ()to expressions to stop double counting of metrics, without loss of information.redfish_system_power_statereturns values for the sameserver, but multiplesystem_id. This causes duplication forRedfish UpandPowered Onpanels.redfish_chassis_healthreturns values for the sameserver, but multiplechassis_id. This causes duplication ofChassis statusandUnhealthy NodesandChassis Statuspanels.Chassis Statusdisplays max value for all returnedchassis_idGeneral Healthpanel for specificserverwould showcriticalorunhealthyif onechassis_idreturned that value. Now takes an average of all returnedchassis_idfrom metricredfish_chassis_health.