[server] Fix orphan segments after full truncation - #4013
Conversation
Delete the previous active segment after opening its replacement at a different offset. Strengthen deletion assertions for all segment files and cover truncation below the first segment across restart so a higher-offset orphan cannot advance the recovered LEO. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 Co-Authored-By: Qoder <noreply@qoder.com> AI-Contributed/Feature: 2/2 AI-Contributed/UT: 54/54
3e3b599 to
b9b2f03
Compare
Rebuild writer state from persisted batches without applying online sequence validation. Warn about discontinuities for observability and verify that the recovered state survives another restart. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 5/5 AI-Contributed/UT: 0/0
b9b2f03 to
2efe41f
Compare
| @@ -328,14 +328,14 @@ LogSegment createAndDeleteSegment( | |||
| if (newOffset == segmentToDelete.getBaseOffset()) { | |||
| deleteSegmentFiles(Collections.singletonList(segmentToDelete), reason); | |||
There was a problem hiding this comment.
When newOffset equals the segment's base offset, this deletes the old segment before its replacement is opened. deleteIfExists() closes the log channel, but the old object remains the active entry in segments; an unsynchronized read can therefore select a closed segment, and if deletion or LogSegment.open() fails this LocalLog instance is left pointing at an unusable active segment. Please rename the old files to the .deleted suffix to free the original paths, install the replacement, and only then physically delete the old segment.
| for (LogRecordBatch batch : records.batches()) { | ||
| if (batch.hasWriterId()) { | ||
| updateWriterAppendInfo(writerStateManager, batch, loadedWriters, false); | ||
| long writerId = batch.writerId(); |
There was a problem hiding this comment.
Could we keep recovery on the existing updateWriterAppendInfo/WriterAppendInfo.append path and parameterize only the sequence-validation policy (for example, ENFORCE versus WARN_AND_ACCEPT)? This block duplicates the writer lookup and aggregation logic, while appendForRecovery duplicates sequence handling and writer-state advancement. Future fixes to batch metadata or writer-state updates would then need to be maintained in both paths and could easily be missed.
2 participants
Purpose
Linked issue: close #4012
[server] Fix orphan segments after full truncation:Delete the previous active segment after creating its replacement. Add regression coverage for upward and downward truncation to ensure stale segment files cannot affect the recovered log end offset.
[server] Tolerate sequence gaps during writer recovery:Rebuild writer state from persisted batches without applying online sequence validation. Warn about discontinuities for observability and verify that the recovered state survives another restart.
Brief change log
Tests
API and Format
Documentation