Add smoke-test gate and auto-rollback to the ECS deploy - #111
Merged
saqibmanan merged 2 commits intoAug 18, 2026
Conversation
Adds a smoke-tests job calling CivicDataSpace-test's run-smoke.yml, passing deployed_sha (github.sha) so the gate verifies /health/'s git_sha field matches, and min_passed to catch a fully-skipped run looking green. Requires a new repo variable DEV_API_BASE_URL and the same secrets DataSpaceFrontend's own pipeline already passes to this workflow (HOME_URL_DEV, TEST_EMAIL_1/2, TEST_PASSWORD_1/2). Also captures the currently-running task definition ARN before the service update (deploy-app now has an output for it) -- needed by the rollback job that follows in the next commit.
Restores the task definition ARN captured before this deploy's service update (previous commit), waits for the rolled-back service to stabilize, then exits 1 -- the run stays red even after successful mitigation, matching the policy that a rollback is damage control, not a pass. Migrations applied by the bad deploy are never auto-reverted; the error message points at where to find what ran. OTel collector rollback is explicitly out of scope here -- separate service, independent risk, keeps this change's blast radius to the app service only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phases 4+5 of the ECS CD-pipeline plan — the last piece. Stacked on #110 — merge #108 → #109 → #110 → this, in order. Also depends on CivicDataSpace-test#16 landing first (the
api_base_url/deployed_sha/min_passedinputs this PR passes don't exist onrun-smoke.ymluntil that merges).Before this,
wait-for-service-stability: truewas the only signal a deploy worked — it only proves ECS tasks started and passed the container healthcheck (which, before #108, always returned 200 regardless of actual health). Nothing verified the app actually works, and nothing rolled back a bad deploy.smoke-testsjob callsCivicDataSpace-test'srun-smoke.yml@CIafterdeploy-appsucceeds, passingdeployed_sha: github.sha(verified against/health/'s newgit_shafield — proves the live code is the code that was just pushed, not just that a container answered) andmin_passed: 1.deploy-appnow captures the task definition ARN that was running before this deploy's service update (new job output), so a rollback has something concrete to restore.rollback-on-smoke-failurerestores that ARN, waits for the service to stabilize, then deliberately exits 1 — the run stays red even after successful mitigation, so a rollback never quietly looks like success. Migrations are never auto-reverted (Django reverse migrations are frequently unsafe); the error message points at the deploy job's migration-task logs.One real design tradeoff, worth being upfront about rather than silently deviating from what I'd planned:
run-smoke.ymlbundles three jobs in one workflow file (consumer-smoke,api-smoke,provider-smoke) — a reusable workflow caller can't select a subset, so this gate runs the full suite (~60-80 min), not just the fast, browserlessapi-smokesubset I'd originally scoped. That matchesDataSpaceFrontend's own existing precedent for calling this same workflow, and arguably has real value here — DataSpaceBackend backs the whole site, so re-validating the full user-facing surface after a backend change isn't unreasonable. But it's slower than the narrow API-only gate ParakhAPI got, and splittingapi-smokeinto its own dedicated reusable workflow is a clean follow-up if the runtime becomes a problem.Configuration needed before this can run
DEV_API_BASE_URL(new) — the backend's own base URL, soapi-smoke's tests actually run instead of skipping (see CivicDataSpace-test#16's description for the pre-existing bug this fixes).HOME_URL_DEV,TEST_EMAIL_1,TEST_PASSWORD_1(required) andTEST_EMAIL_2,TEST_PASSWORD_2(optional) — same onesDataSpaceFrontend's pipeline already has configured; need to be added to this repo too.Test plan
actionlintclean on every line this PR touches (one new pre-existing-category finding: the rollback job's "Configure AWS credentials" step reuses the same outdated@v1action pin already used everywhere else in this file — not a new issue class, out of scope to fix repo-wide here)uses:/with:/secrets:, noenvironment:on auses:job) against the known gotchas from the frontend pipeline work