Fix health-check false-green and unsafe runtime migrations - #108
Merged
Conversation
health_check previously returned JsonResponse(data) unconditionally, so the ECS container healthcheck (curl -f) and any future smoke gate would false-green a container with a dead DB/ES/Redis/telemetry connection. Also exposes git_sha so a deploy can verify the running code matches what was just pushed.
Follows the same ARG->ENV pattern the container already uses for other build-time config. Feeds health_check's new git_sha field.
Generating migration files at deploy time instead of using committed ones is unsafe under a rolling deployment (briefly 2 tasks live) and means the schema that lands in prod was never reviewed. migrate itself stays here for now; moving it to an explicit one-off step is next.
This was referenced Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First slice of the ECS CD-pipeline work (Phase 0+1 of the plan). Independent value on its own — fixes two real bugs found while auditing
deploy-to-ecs.yml.api/views/health.py:health_checkreturned HTTP 200 unconditionally (JsonResponse(data), nostatus=), even whenoverall_statuswasFalse. The ECS task definition's container healthcheck (curl -f http://localhost:8000/health/) and any future smoke gate hitting this endpoint would both false-green a container with a dead DB/ES/Redis/telemetry connection. Now returns 503 when unhealthy.git_shafield to the same response, sourced from a newGIT_COMMIT_SHAenv var (plumbed viaARG/ENVin the Dockerfile, same pattern already used for other build-time config). This is the primitive a later smoke gate needs to verify "the code that's live is the code that was just pushed."docker-entrypoint.sh: droppedpython manage.py makemigrations --noinput, which ran on every container boot. Generating migration files at deploy time instead of using committed ones is unsafe under Fargate's rolling deployment (briefly 2 tasks live) and means the schema landing in prod was never reviewed.migrate --noinputstays in the entrypoint for now — moving it to an explicit one-off pipeline step is a separate follow-up PR.deploy-backend.ymldraft (EC2 + GHCR +docker compose) that had no matching secrets/infra behind it — confirmed the real deploy target is ECS Fargate (CloudFormation-provisioned RDS/ElastiCache/managed-ES, already-configureddevelopmentenvironment secrets), not EC2.Test plan
tests/test_health.py(new, 5 tests): 200 when all deps healthy, 503 when ES down, 503 when Redis down,git_shadefaults to"unknown",git_shareflectsGIT_COMMIT_SHApytest tests/): 100 passed vs. the pre-existing baseline's 95 passed — no new failures. The 22 failed / 7 errored tests incharts/test_dataset_schema/test_geography_hierarchyare pre-existing ondev(confirmed by stashing this branch's changes and re-running against unmodifieddev— identical failure set)bash -n+shellcheckclean ondocker-entrypoint.shpython -m py_compileclean onhealth.pyNext in this series
Moving
migrateout of the entrypoint into an explicit ECS one-off task, fixingdeploy-to-ecs.yml'sif: always()and fragile infra-change condition, wiring a smoke-test gate, and adding rollback-on-failure — tracked as follow-up PRs.