HDDS-15946. [Ozone versioning] [T2] VersionId generation strategy framework - #10845
Open
symious wants to merge 11 commits into
Open
HDDS-15946. [Ozone versioning] [T2] VersionId generation strategy framework#10845symious wants to merge 11 commits into
symious wants to merge 11 commits into
Conversation
symious
changed the title
symious
changed the title
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…equest Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Key names in OBJECT_STORE buckets contain '/' verbatim, so a '/' separator interleaves a key's versions with those of keys nested under it, breaking the single-seek promotion and the merged ListObjectVersions order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
symious
force-pushed
the
HDDS-15946
branch
2 times, most recently
from
August 20, 2026 01:52
7358ce5 to
82a79e7
Compare
Adds VersionIdGenerator, the pluggable source of the id an object version is numbered with, and UniqueIdVersionIdGenerator as the cluster default. The id is proposed on the OM that received the request, in preExecute, so it travels in the replicated request and every OM applies a version that is already numbered. Nothing about it depends on the transaction carrying the write, or on OM being replicated by Ratis. The default numbers a version with the time it was written, through the scheme Ozone already uses for block local IDs: currentTimeMillis << 16 with a 16-bit counter separating ids proposed inside one millisecond. It needs no allocator state and no coordination, which is what makes it safe to read on any OM. The interface has one abstract method, generateVersionId(), plus a default versionIdFor(proposed, hasCurrentVersion) that lets a generator number some versions specially at apply time. The implementation is selected cluster-wide by ozone.om.versioning.version-id-generator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds VersionIdAllocator, which turns the id proposed for a version into the id it is applied with. versionedKeyTable orders a key's versions by Long.MAX_VALUE - versionId, so the ids of one key have to increase in the order the versions were written. A proposal is a clock reading and cannot promise that: ids proposed inside one millisecond can exhaust the counter separating them, and a leader change onto a lagging clock proposes a lower value. So a proposal is a floor. The applied id is the later of it and the id after the key's current version, which the write path already holds - no read of its own, no global state, and identical on every OM. Under a clock regression an affected key's ids climb by one until proposals overtake them again: the versions stay ordered and only the id's reading as a time degrades. propose() runs in preExecute on the OM that received the request; allocate() runs under the write's lock on every OM. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds PinnedFirstVersionIdGenerator, which numbers versions like the default except that a key's first version takes FIRST_VERSION_ID, so it can be referenced without listing the key's versions first. Whether the key already has a version is not known when the id is proposed, so the generator decides it in versionIdFor, under the write's lock, from the current version the allocator was handed. The sentinel is 1: below every proposed id, so a pinned version sorts at the old end of the key in versionedKeyTable, and above the unset value a pre-versioning record carries. It says nothing about the null version, which carries a proposed id like any other and is marked by isNullVersion. Known trade-off: once every version of a key has been permanently deleted, a recreated key takes the sentinel again, so an external reference to the first version resolves to the new content. The generator is off by default and selected by ozone.om.versioning.version-id-generator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
First 6 commits belong to HDDS-15879, can ignore them.
This ticket includes the follows tasks:
VersionIdGeneratorinterface;TransactionIndexVersionIdGenerator; theozone.om.versioning.version-id-generatorclass-name config, loaded reflectivelyVersionIdAllocator: refuse a commit whose id does not come after the key's current version; fall back to a versionedKeyTable lookup for records predating versioningINVALID_REQUEST; taken id on a pre-versioning record rejected withKEY_ALREADY_EXISTS; steady state performs no lookup (unit test asserts versionedKeyTable is never touched)PinnedFirstVersionIdGenerator: first-version detection (no current in keyTable) +FIRST_VERSION_ID = 1sentinelWhat is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15946
How was this patch tested?
Unit test