Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
85e52c1
[python][ray] Add time-based resumable commits to write_paimon
XiaoHongbo-Hope Aug 4, 2026
d1f9d60
[python][ray] Simplify resumable write checkpoints
XiaoHongbo-Hope Aug 4, 2026
86a5d68
[python][ray] Keep cross-partition write guard
XiaoHongbo-Hope Aug 4, 2026
a1337a4
[python][ray] Harden and batch resumable write checkpoints
XiaoHongbo-Hope Aug 4, 2026
709c954
[python][ray] Allow resumable writes across target compaction
XiaoHongbo-Hope Aug 5, 2026
7306371
[python][ray] Stabilize transform fingerprints across drivers
XiaoHongbo-Hope Aug 5, 2026
20d713f
[python][ray] Stabilize native source split offsets
XiaoHongbo-Hope Aug 5, 2026
4a91e64
[python] Limit schema checks to protected commits
XiaoHongbo-Hope Aug 5, 2026
891cac8
Merge remote-tracking branch 'apache/master' into codex/write-paimon-…
XiaoHongbo-Hope Aug 5, 2026
e32911e
[python][ray] Periodically commit Ray Dataset writes
XiaoHongbo-Hope Aug 5, 2026
160850c
[python][ray] Clarify incremental write recovery
XiaoHongbo-Hope Aug 5, 2026
8f2a067
[python][ray] Simplify incremental source dispatch
XiaoHongbo-Hope Aug 5, 2026
d9f91da
[python][ray] Isolate incremental target preparation
XiaoHongbo-Hope Aug 5, 2026
22b43d5
[python][ray] Split incremental write source paths
XiaoHongbo-Hope Aug 5, 2026
24088e0
[python][ray] Tighten incremental write comments
XiaoHongbo-Hope Aug 5, 2026
939381e
[python][ray] Consolidate incremental write test helpers
XiaoHongbo-Hope Aug 5, 2026
27dc746
[python][ray] Move incremental target preparation to sink
XiaoHongbo-Hope Aug 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions docs/docs/pypaimon/ray-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,60 @@ overlapping buckets or sequence numbers for those modes.
HASH_FIXED grouping. This option does not enable HASH_DYNAMIC or
CROSS_PARTITION primary-key writes.

### Incremental write

An existing Ray Dataset can commit target groups periodically:

```python
write_paimon(
updates,
"database_name.target",
{"warehouse": "/path/to/warehouse"},
commit_mode="incremental",
update_cols=["feature"],
commit_interval_seconds=600,
)
```

Committed groups survive failure, but retrying recomputes the Dataset lineage.
A plain Dataset does not accept `operation_id`. Pin the starting snapshot when
it reads from the target table itself.

Use `PaimonOffsetSource` to resume source progress. Put the replayable
transformation, including inference, in `transform`:

```python
from pypaimon.ray import PaimonOffsetSource, write_paimon

def build_updates(window):
return window.map_batches(infer, batch_format="pyarrow")

source = PaimonOffsetSource(
"database_name.source",
projection=["id", "payload"],
transform=build_updates,
)
write_paimon(
source,
"database_name.target",
{"warehouse": "/path/to/warehouse"},
commit_mode="incremental",
update_cols=["feature"],
operation_id="feature-backfill-2026-07",
commit_interval_seconds=600,
)
```

Retry with the same `operation_id` skips completed source windows; only an
unfinished window may run again.

Use a fixed-bucket, partial-update target. Source rows need unique primary keys
and all `update_cols`; missing keys are inserted. Source splits run in bounded
windows, so the actual interval may be longer. Concurrent data changes or schema
changes fail; compaction is allowed. After success,
`delete_write_paimon_checkpoint` releases retained snapshots. An `operation_id`
must not be reused after changing the source transformation.

### `TableWrite.write_ray()` (lower-level)

If you have already constructed a `table_write` from a write builder, you can
Expand Down
4 changes: 4 additions & 0 deletions paimon-python/pypaimon/ray/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@
)
from pypaimon.ray.update_by_row_id import update_by_row_id
from pypaimon.ray.read_by_row_id import read_by_row_id
from pypaimon.ray.offset_source import PaimonOffsetSource
from pypaimon.ray.incremental_write import delete_write_paimon_checkpoint

__all__ = [
"read_paimon",
Expand All @@ -40,6 +42,8 @@
"merge_into",
"update_by_row_id",
"read_by_row_id",
"PaimonOffsetSource",
"delete_write_paimon_checkpoint",
"WhenMatched",
"WhenNotMatched",
"source_col",
Expand Down
Loading
Loading