Skip to content

[python][ray] Add time-based resumable commits to write_paimon - #9033

Closed
XiaoHongbo-Hope wants to merge 17 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/write-paimon-time-checkpoint
Closed

[python][ray] Add time-based resumable commits to write_paimon#9033
XiaoHongbo-Hope wants to merge 17 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/write-paimon-time-checkpoint

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Allow long-running Ray primary-key updates to commit periodically and resume after a driver restart.

Changes

  • add PaimonOffsetSource with a pinned source snapshot and stable offsets;
  • add commit_mode="incremental", operation_id, commit_interval_seconds, and update_cols to write_paimon;
  • store source progress in the same target snapshot as each data commit;
  • retain the source snapshot and latest checkpoint with internal tags;
  • reject concurrent target commits and schema changes;
  • add delete_write_paimon_checkpoint for completed operations.

The default atomic path is unchanged. Incremental mode currently supports fixed-bucket primary-key targets using merge-engine=partial-update; sequence fields and cross-partition updates are not supported.

Tests

  • real Ray E2E: time-triggered commits, restart recovery, empty results, inserts, concurrent commits, and schema changes;
  • existing Ray write and commit regression suites;
  • project flake8 and git diff --check.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 4, 2026 14:22
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft August 5, 2026 13:38
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 5, 2026 16:52
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft August 6, 2026 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant