Skip to content

Add smoke-test gate and auto-rollback to the ECS deploy - #111

Merged
saqibmanan merged 2 commits into
feat/ecs-explicit-migration-taskfrom
feat/ecs-smoke-gate
Aug 18, 2026
Merged

Add smoke-test gate and auto-rollback to the ECS deploy#111
saqibmanan merged 2 commits into
feat/ecs-explicit-migration-taskfrom
feat/ecs-smoke-gate

Conversation

@saqibmanan

Copy link
Copy Markdown
Contributor

Summary

Phases 4+5 of the ECS CD-pipeline plan — the last piece. Stacked on #110 — merge #108#109#110 → this, in order. Also depends on CivicDataSpace-test#16 landing first (the api_base_url/deployed_sha/min_passed inputs this PR passes don't exist on run-smoke.yml until that merges).

Before this, wait-for-service-stability: true was the only signal a deploy worked — it only proves ECS tasks started and passed the container healthcheck (which, before #108, always returned 200 regardless of actual health). Nothing verified the app actually works, and nothing rolled back a bad deploy.

  • smoke-tests job calls CivicDataSpace-test's run-smoke.yml@CI after deploy-app succeeds, passing deployed_sha: github.sha (verified against /health/'s new git_sha field — proves the live code is the code that was just pushed, not just that a container answered) and min_passed: 1.
  • deploy-app now captures the task definition ARN that was running before this deploy's service update (new job output), so a rollback has something concrete to restore.
  • rollback-on-smoke-failure restores that ARN, waits for the service to stabilize, then deliberately exits 1 — the run stays red even after successful mitigation, so a rollback never quietly looks like success. Migrations are never auto-reverted (Django reverse migrations are frequently unsafe); the error message points at the deploy job's migration-task logs.

One real design tradeoff, worth being upfront about rather than silently deviating from what I'd planned: run-smoke.yml bundles three jobs in one workflow file (consumer-smoke, api-smoke, provider-smoke) — a reusable workflow caller can't select a subset, so this gate runs the full suite (~60-80 min), not just the fast, browserless api-smoke subset I'd originally scoped. That matches DataSpaceFrontend's own existing precedent for calling this same workflow, and arguably has real value here — DataSpaceBackend backs the whole site, so re-validating the full user-facing surface after a backend change isn't unreasonable. But it's slower than the narrow API-only gate ParakhAPI got, and splitting api-smoke into its own dedicated reusable workflow is a clean follow-up if the runtime becomes a problem.

Configuration needed before this can run

  • Repo variable DEV_API_BASE_URL (new) — the backend's own base URL, so api-smoke's tests actually run instead of skipping (see CivicDataSpace-test#16's description for the pre-existing bug this fixes).
  • Secrets HOME_URL_DEV, TEST_EMAIL_1, TEST_PASSWORD_1 (required) and TEST_EMAIL_2, TEST_PASSWORD_2 (optional) — same ones DataSpaceFrontend's pipeline already has configured; need to be added to this repo too.

Test plan

  • actionlint clean on every line this PR touches (one new pre-existing-category finding: the rollback job's "Configure AWS credentials" step reuses the same outdated @v1 action pin already used everywhere else in this file — not a new issue class, out of scope to fix repo-wide here)
  • Traced through the reusable-workflow call syntax (uses:/with:/secrets:, no environment: on a uses: job) against the known gotchas from the frontend pipeline work
  • Not verifiable locally: the actual smoke-test run and rollback against real AWS/ECS — no credentials in my environment. Phase 6 (staged rollout: build-only, then one supervised deploy, then a deliberate forced-failure to prove rollback) is where this gets proven for real.

Adds a smoke-tests job calling CivicDataSpace-test's run-smoke.yml,
passing deployed_sha (github.sha) so the gate verifies /health/'s
git_sha field matches, and min_passed to catch a fully-skipped run
looking green. Requires a new repo variable DEV_API_BASE_URL and the
same secrets DataSpaceFrontend's own pipeline already passes to this
workflow (HOME_URL_DEV, TEST_EMAIL_1/2, TEST_PASSWORD_1/2).

Also captures the currently-running task definition ARN before the
service update (deploy-app now has an output for it) -- needed by the
rollback job that follows in the next commit.
Restores the task definition ARN captured before this deploy's service
update (previous commit), waits for the rolled-back service to
stabilize, then exits 1 -- the run stays red even after successful
mitigation, matching the policy that a rollback is damage control, not
a pass. Migrations applied by the bad deploy are never auto-reverted;
the error message points at where to find what ran.

OTel collector rollback is explicitly out of scope here -- separate
service, independent risk, keeps this change's blast radius to the
app service only.
@saqibmanan
saqibmanan merged commit 7cff213 into feat/ecs-explicit-migration-task Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant