Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 49 additions & 27 deletions cuda_core/cuda/core/_cpp/GRAPH_ATTACHMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,13 @@ The CUDA user-object reference count controls the attachment lifetime.
`GraphAttachmentMap` only lets cuda.core find the attachment currently
associated with a node.

Each `NodeAttachment` contains two type-erased `OpaqueHandle` owners:
Each `NodeAttachment` has two type-erased `OpaqueHandle` slots, allowing it to
hold up to two node-specific resource owners. These are:

- kernel: kernel and argument storage
- host callback: callback and copied user data
- memcpy: destination and source
- memset or event: destination or event in the first owner
- kernel node: kernel and argument storage
- host callback node: callback and copied user data
- memcpy node: destination and source
- memset or event node: destination or event

`OpaqueHandle` is `shared_ptr<const void>`. Existing cuda.core handles reuse
their shared ownership when converted to it. Python objects and copied callback
Expand All @@ -93,23 +94,47 @@ published attachment. The resources those owners keep alive, including Python
objects, may remain mutable, but they must not be modified in a way that
releases resources still referenced by an installed parameter version.

## Executable graph attachments

Executable graphs can be modified after instantiation. Those updates may
introduce new resources (events, memory, kernels, kernel parameters, and host
callbacks) that must outlive in-flight launches. CUDA provides no way to attach
user objects to an executable graph (`cuGraphRetainUserObject` accepts a
`CUgraph` only), so cuda.core emulates that lifetime tracking.

Before instantiation or whole-graph update, cuda.core retains one
`ExecAttachments` accumulator on the source graph as a CUDA user object.
Instantiation or `cuGraphExecUpdate` propagates that reference into the
executable; cuda.core then releases the source graph's temporary reference so
the executable (and its in-flight launches) own the accumulator. Individual
node updates append owners to that accumulator through the same prepare/commit
transaction used for definition attachments.
Comment thread
juenglin marked this conversation as resolved.

Appended owners are never removed: each update can only grow the accumulator.
A successful whole-graph update replaces the accumulator entirely, so the
previous owners are dropped once their last launch finishes. Enable/disable
attaches nothing. A child-graph update relies on CUDA cloning the replacement
graph's user-object references, which carry the child definition's attachments.

## Deferred cleanup

CUDA invokes a user-object destructor on an internal thread where CUDA API
calls are forbidden. Destroying an attachment there could release handles whose
deleters call CUDA or run Python finalizers.

`NodeAttachment` therefore inherits from `DeferredCleanupItem`. The CUDA
`NodeAttachment` and `ExecAttachments` therefore inherit from
`DeferredCleanupItem`. The CUDA
destructor callback only adds the attachment to the process-lifetime
`DeferredCleanupQueue` and requests a `Py_AddPendingCall`.

One pending call drains all queued attachments from Python's main thread. The
queue coalesces work because CPython's pending-call queue is bounded. If
scheduling fails, attachments stay queued and a later enqueue or safe cuda.core
entry retries. Graph and executable-graph destruction and explicit close paths
provide additional retry points. During Python finalization, scheduling stops
and unreclaimable attachments are intentionally leaked rather than destroyed
in an unsafe context.
queue coalesces work because CPython's pending-call queue is bounded, and there
could be many more deferred cleanup items than allowed pending calls. If
`Py_AddPendingCall` fails, the attachments remain queued. A later successful
`Py_AddPendingCall` will safely clean them up. Graph and executable-graph
destruction and explicit close paths provide additional retry points. During
Python finalization, scheduling stops and unreclaimable attachments are
intentionally leaked rather than destroyed in an unsafe context.

## Graph hierarchy state

Expand Down Expand Up @@ -157,30 +182,27 @@ be invalidated when CUDA destroys that graph. They use separate

## Invariants

1. The owner bundle of a published `NodeAttachment` is never modified in place.
1. The owner bundle of a published `NodeAttachment` is never modified in place;
replace the whole bundle.
2. CUDA user-object references, not metadata pointers, own attachments.
3. Metadata is removed or replaced before its graph reference is released.
4. Fallible attachment setup and metadata allocation happen before the CUDA
graph mutation they support.
5. Every live cuda.core `CUgraph` has one canonical `GraphBox` and registry
entry.
6. Graph boxes remain in parent-before-child order.
7. Destroyed child boxes remain at stable addresses in the graveyard.
8. A raw graph handle is unregistered before its box becomes a tombstone.
9. CUDA callbacks only enqueue attachments; they never release owners or call
3. Fallible attachment setup and metadata allocation happen before the CUDA
graph mutation they support; metadata is removed or replaced before its
graph reference is released.
4. CUDA callbacks only enqueue attachments; they never release owners or call
CUDA.
10. Graph mutations and their metadata updates require the same external
5. Graph mutations and their metadata updates require the same external
synchronization as the underlying CUDA graph.

## Scope

- Attachment metadata tracks graph mutations performed through cuda.core.
- Raw driver clones receive the CUDA user-object references needed for safe
execution, but cuda.core cannot reconstruct their node-to-attachment map.
- Executable graphs rely on CUDA's inherited user-object references; they do
not use `GraphAttachmentMap`.
- Direct executable-node updates require separate executable ownership and are
not tracked by definition attachment metadata.
- Executable graphs keep one append-only accumulator instead of a
`GraphAttachmentMap`. cuda.core cannot map an executable node back to its
owners, so it can neither report nor release them individually.
- Executable-node updates do not change definition attachment metadata, and
definition updates do not change an executable's accumulator.
- Stream capture explicitly retains host callbacks. Other captured operations
keep their documented caller-owned lifetime contract.
- CPython's cyclic garbage collector cannot follow the ownership path from a
Expand Down
Loading
Loading