HDDS-15997. Fix QuotaRepairTask scan hang and partial counts on worker failure - #10884
HDDS-15997. Fix QuotaRepairTask scan hang and partial counts on worker failure#10884chihsuan wants to merge 3 commits into
Conversation
chungen0126
left a comment
chungen0126
left a comment
There was a problem hiding this comment.
Thanks @chihsuan for working on this.
| for (Future<?> f : tasks) { | ||
| if (f.isDone()) { | ||
| f.get(); | ||
| throw new IllegalStateException("quota repair scan worker exited prematurely"); |
There was a problem hiding this comment.
Could you help clarify when this scenario occurs?
There was a problem hiding this comment.
Good question. It's a safety guard, not a case we expect to hit. In theory, it should be unreachable today, since a worker can't finish normally while the producer is still feeding it, so a missing worker always comes with an exception we fail on.
The reason it's still there is that this PR changes the producer to poll instead of blocking forever, so the loop needs a guaranteed exit. If workers ever went away without an exception, nobody would drain the queue, and the producer would just keep polling. Does that make sense to you?
There was a problem hiding this comment.
@chungen0126 Do you have any feedback on this? Or do you prefer to remove this?
chungen0126
left a comment
chungen0126
left a comment
There was a problem hiding this comment.
Overall, LGTM. Just one minor.
| kvList = new ArrayList<>(BATCH_SIZE); | ||
| } | ||
| } | ||
| q.put(kvList); | ||
| putBatch(q, kvList, tasks); |
There was a problem hiding this comment.
If the total number of keys is exactly a multiple of the batch size, this putBatch will put an empty kvList. I think it would be better to add a condition here to check if kvList is not empty before putting it.
There was a problem hiding this comment.
Fixed, the trailing batch is now only queued when non-empty. Thanks!
What changes were proposed in this pull request?
QuotaRepairTaskscans each table with one producer thread feeding batches to worker threads through a bounded queue. If all workers die with an exception, the producer blocks forever on the full queue and the repair hangs withIN_PROGRESSheld. If the scan is interrupted, both sides return silently and the partially scanned counts are submitted through Ratis as a successful repair.This change makes scan failures fail the repair instead:
scanTableInBatcheswith an injectable per-record consumer.putBatch, which fails fast when a worker has already exited.New unit tests cover worker failure, producer interrupt, and awaiting in-flight workers. This is a prerequisite for HDDS-15835, which adds the first per-record consumer that reads RocksDB and makes worker failure realistic.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15997
How was this patch tested?
Focused unit tests passed (7 tests, no failures):
mvn -pl :ozone-manager test -Dtest=TestQuotaRepairTask -DskipShade -DskipRecon -DskipDocsRepository checkstyle also passed:
Generated-by: Codex (GPT-5)