Batch triggerer unused trigger cleanup deletes - #70961
Draft
shivaam wants to merge 1 commit into
Draft
Conversation
Large trigger tables make synchronous cleanup hold locks until every eligible row is deleted, delaying triggerer progress and database writers.
shivaam
commented
Aug 3, 2026
| update(TaskInstance) | ||
| .where( | ||
| TaskInstance.state != TaskInstanceState.DEFERRED, | ||
| TaskInstance.trigger_id.is_not(None), |
Contributor
Author
There was a problem hiding this comment.
The existing TaskInstance.trigger_id cleanup update remains unbounded. I’m keeping that separate from this PR because batching it safely requires selecting TaskInstances by their id, locking the triggers, and validating the approach across supported databases. This PR stays focused on #68243’s Trigger deletion problem
shivaam
commented
Aug 3, 2026
| select(cls.id) | ||
| .where( | ||
| ~cls.assets.any(), | ||
| ~cls.callback.has(), |
Contributor
Author
There was a problem hiding this comment.
Noticed that callback.trigger_id has no explicit index, which could make this lookup slower.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trigger.clean_unused()runs synchronously in every triggerer supervisor loop. It currently removes every eligible trigger in one transaction, so busy metadata databases can hold many row locks and delay the triggerer heartbeat and concurrent writers.This change selects and deletes unused trigger IDs in bounded transactions, committing between batches. It keeps the current
NOT EXISTSpredicates andSKIP LOCKEDcandidate locking from #68244, uses a portable bounded select-ID/delete-ID shape on every dialect, and retries a failed batch in a fresh transaction. A new positive[triggerer] unreferenced_triggers_cleanup_batch_sizeoption defaults to 500; non-positive values fail fast rather than restoring an unbounded path.Focused coverage exercises empty, below/exactly/above-threshold and multi-batch cleanup, protected deferred-task, asset, and callback references, invalid configuration, committed progress across a later failed batch, fresh retry transactions, and eventual drain.
Prior work: #68269 proposed the original batching direction. This implementation is a clean current-main rewrite because that draft predates #68244 and committed inside a caller-injected session.
Scope note: this PR leaves the pre-existing unbounded TaskInstance reference update and portable
callback.trigger_idindexing unchanged; those require separate batching/index and migration designs.Validation:
[500, 500]deletes while 5 rows were concurrently locked, eventual drain after lock release, and protected asset/callback rows preserved.upstream/mainat66444504c4; the rebased patch ID is identical to the runtime-tested patch.closes: #68243
Was generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-5) following the guidelines