Skip to content

AIR CLI Integration: fall back to MLflow when Bricklens returns no logs for a terminal run - #6245

Open
riddhibhagwat-db wants to merge 1 commit into
air-clifrom
air-logs-empty-fallback
Open

AIR CLI Integration: fall back to MLflow when Bricklens returns no logs for a terminal run#6245
riddhibhagwat-db wants to merge 1 commit into
air-clifrom
air-logs-empty-fallback

Conversation

@riddhibhagwat-db

Copy link
Copy Markdown
Contributor

Changes

air logs <run> printed "No logs available for run . Run terminated in state SUCCESS" and exited 0 for runs whose logs were fully retrievable: air logs --download-to DIR on the same run returned the complete log. This happens when Bricklens is enabled for the workspace but never ingested the run so it answers every request successfully with zero records, yet the logs are present in MLflow.

The streaming (print) path only fell back to MLflow on errBricklensFeatureDisabled (gated off / not deployed / persistent failure); an empty-but-successful Bricklens response was treated as the final answer. The download path already reads from MLflow, which is why it worked.

This is the Go version of the Python fix: https://github.com/databricks-eng/universe/pull/2384942

Tests

  1. TestStreamBricklensEmptyFallsBackToMLflow:
  • terminal run — a TERMINATED/SUCCESS run whose Bricklens /logs returns {"log_records": []} must make streamBricklensLogs return errBricklensFeatureDisabled and emit nothing (so the caller falls back to MLflow rather than printing "No logs available").
  • static view of a past retry — same assertion for the staticView: true path.
  1. TestStreamBricklensTerminalWithRecordsDoesNotFallBack: the guard against over-eager fallback: a terminal run whose Bricklens stream does have a record prints it ("line":"hello"), returns true, no error — confirming the fallback only fires on a genuinely empty stream.
  2. TestFetchLogsFallsBackToMLflowWhenBricklensEmpty: the end-to-end reproduction of your bug: one fake server serving empty Bricklens /logs plus a populated MLflow path (runs/get → get-output → artifacts/list → credentials-for-read → presigned chunk download). Asserts fetchLogs returns success and the two MLflow log lines (line 1, line 2) reach stdout — i.e. the print path now behaves like --download-to

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 9d8e92a

Run: 31642925843

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 291 1104 4:08
💚​ aws windows 4 4 293 1102 2:58
💚​ azure linux 4 4 290 1104 4:27
💚​ azure windows 4 4 292 1102 3:51
💚​ gcp linux 1 5 291 1104 4:42
💚​ gcp windows 1 5 293 1102 4:30
8 interesting tests: 4 RECOVERED, 4 SKIP
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R
Top 1 slowest tests (at least 2 minutes):
duration env testname
2:34 azure windows TestAccept

Comment thread experimental/air/cmd/logstream.go Outdated
Comment on lines +306 to +310
// Bricklens served every request but never returned a record. That is
// not proof the run has no logs: Bricklens ingestion can lag or miss a
// run whose logs are fully present in MLflow (the path --download-to
// uses). Defer to the MLflow fallback rather than declaring "no logs"
// here; it owns the real no-logs report, with the same exit code.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shorten comment

@riddhibhagwat-db
riddhibhagwat-db force-pushed the air-logs-empty-fallback branch from 00e0aee to a505c70 Compare August 12, 2026 21:08
@riddhibhagwat-db
riddhibhagwat-db changed the base branch from main to air-cli August 12, 2026 21:10
…l run

`air logs <run>` printed "No logs available for run <id>. Run terminated
in state SUCCESS" and exited 0 for runs whose logs were fully retrievable:
`air logs --download-to DIR` on the same run returned the complete log.
This happens when Bricklens is enabled for the workspace but never ingested
the run — it answers every request successfully with zero records, yet the
logs are present in MLflow.

The streaming (print) path only fell back to MLflow on errBricklensFeatureDisabled
(gated off / not deployed / persistent failure); an empty-but-successful
Bricklens response was treated as the final answer. The download path already
reads from MLflow, which is why it worked.

Treat "Bricklens served every request but never returned a record" the same
as feature-disabled: hand off to the MLflow fallback, which owns the real
no-logs report and preserves the run-derived exit code. This mirrors the
Python CLI fix (databricks-eng/universe#2366012). Applies to both the terminal
tail and the static (past-retry) view.

Co-authored-by: Isaac
@riddhibhagwat-db
riddhibhagwat-db force-pushed the air-logs-empty-fallback branch from a505c70 to 9d8e92a Compare August 12, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants