Skip to content

Keep posts created in the same second they are ingested - #48

Open
huxint wants to merge 1 commit into
xai-org:mainfrom
huxint:fix/thunder-same-second-post-drop
Open

Keep posts created in the same second they are ingested#48
huxint wants to merge 1 commit into
xai-org:mainfrom
huxint:fix/thunder-same-second-post-drop

Conversation

@huxint

@huxint huxint commented Aug 14, 2026

Copy link
Copy Markdown

Summary

PostStore::insert_posts uses a strict < when comparing created_at against the current wall-clock second, so a post whose create event is applied in the same second it was created is silently discarded and never enters the in-network store.

Root cause

thunder/posts/post_store.rs:134:

posts.retain(|p| {
    p.created_at < current_time
        && current_time - p.created_at <= (self.retention_seconds as i64)
});

Both timestamps are epoch seconds, so for a low-latency pipeline created_at == current_time is a normal case, not a future-dated timestamp. The event is consumed once and not revisited for the lifetime of the process — only a restart, which re-reads the topic under a fresh consumer group, would recover it — so the freshest in-network content is simply missing from that instance's store.

Fix

Use <= so only genuinely future-dated posts are rejected. Boundary arithmetic is unaffected: at equality current_time - created_at == 0, which passes the retention check here, and the u64 subtraction in trim_old_posts still cannot underflow since created_at <= current_time continues to hold at insert time.

Verification

  • One-character change. thunder/ ships no Cargo manifest, so the crate cannot be compiled from this snapshot; rustfmt parses the file cleanly.
  • The existing unit tests in this file only use created_at values of current_time - 1 and older, so they do not exercise this boundary and are unaffected by the change.

insert_posts drops any post whose created_at is not strictly less than
the current wall-clock second. A post whose create event is applied in
the same second it was created — normal for a low-latency Kafka
pipeline — fails the strict comparison and is silently discarded. The
event is not revisited for the lifetime of that serving process: only a
restart, which re-reads the topic under a fresh consumer group, would
pick the post up again. Until then the freshest in-network content is
simply missing from that instance's store.

The check exists to reject future-dated timestamps; created_at equal to
the current second is not future-dated. Use <= so only genuinely
future-dated posts are dropped. The retention arithmetic is unaffected:
current_time - created_at is 0 at the boundary, which passes the
retention check and cannot underflow in trim_old_posts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant