perf: reuse Arrow IPC compression context across shuffle blocks

[peterxcli]

Which issue does this PR close?

Closes #5008.

Rationale for this change

Avoid recreating Arrow IPC compression state for every shuffle block.

What changes are included in this PR?

Reuse one CompressionContext per buffered partition writer and across batches in the JVM row conversion path. Update benchmarks and callers accordingly.

How are these changes tested?

Existing shuffle IPC round-trip, buffered writer, and shuffle scan tests pass. The shuffle writer benchmark also compiles successfully.

[andygrove]

Thanks @peterxcli. Can you share any benchmark results?

[peterxcli]

@andygrove this would take effect only after #4888 is merged.
and the reason is:

---

with arrow 58.4:

Production path	Baseline / patch blocks	FlatBuffer allocations per block	FlatBuffer bytes per block	Context allocations
JVM sort → row.rs	2,688 / 2,441	11 / 11	4,136 / 4,136	0 / 0
Native → BufBatchWriter	5,857 / 4,298	11 / 11	4,136 / 4,136	0 / 0

with arrow 59.1:

1,000 blocks	Fresh context	Reused context
Allocation calls	212,000	201,011
Requested bytes	8,508,496,000	8,504,364,136

• In Arrow 58.4, CompressionContext contains only an optional zstd compressor—no FlatBufferBuilder: compression.rs (lines 25–34)
• Comet does not enable Arrow IPC compression, so that compressor remains uninitialized:

  datafusion-comet/native/shuffle/src/writers/shuffle_block_writer.rs

  Lines 107 to 110 in 8c50487

  	// Shuffle blocks are always written with metadata version V5. Pin it explicitly rather than
  	// relying on `IpcWriteOptions::default`, because IPC_EOS is only the correct end-of-stream
  	// marker for V5. Alignment 64 and non-legacy framing match the arrow defaults.
  	let write_options = IpcWriteOptions::try_new(64, false, MetadataVersion::V5)?;

• Arrow 58.4 independently constructs a new FlatBuffer builder for every batch: writer.rs (lines 520–526).
• The review description in perf: encode shuffle IPC schema once per writer instead of per block #5006 (comment) matches Arrow 59.1’s newer IpcWriteContext, which owns the builder: compression.rs (lines 27–38).

[peterxcli]

@andygrove I've publish the benchmark, would appreciate it if you could take a look at this and see if we can get this merged.

[sunchao]

LGTM. The mutable context stays with the buffered writer or row-conversion loop, while dictionary tracking and outer compression frames remain independent for each block.

I also compared the base and head writers using a focused Arrow 58.4 probe covering all four codecs, dictionary and schema changes, repeated flushes, and output failures. All seven tests passed. The Arrow-version dependency of the performance benefit is already covered in the benchmark discussion.