fix: canonicalize NaN in flat arrays_overlap float keys

[sunchao]

Why are the changes needed?

arrays_overlap answers a simple question: do two arrays contain at least one common value? For floating-point arrays, Spark treats every representation of NaN as the same value, even when their underlying IEEE 754 sign bits or payloads differ. Comet's native implementation compared those raw bit patterns instead. As a result, enabling Comet could silently change the answer to an otherwise ordinary Spark SQL query.

Consider a Parquet table containing a NaN:

CREATE TABLE t_flat(x DOUBLE) USING parquet;
INSERT INTO t_flat VALUES (DOUBLE('NaN')), (0.0D), (1.0D);

SELECT x,
       arrays_overlap(array(x), array(-x)) AS overlap
FROM t_flat;

For the first row, -x is still a NaN, but negating it at execution time changes its sign bit. Spark correctly considers that value equal to the original NaN. Before this change, Comet incorrectly treated the two representations as different:

Input x	Spark	Comet before	Comet after
NaN	true	false	true
+0.0	false	false	false
1.0	false	false	false

The +0.0 row doubles as an important compatibility control: -x becomes -0.0, and Spark deliberately considers the two zeros different for flat arrays. Fixing NaN comparisons must not change that result.

The mismatch also affects SQL null semantics. If the first array additionally contains NULL, Spark returns true because the two NaN values overlap, while the previous Comet implementation returns NULL because it misses that definite match:

SELECT arrays_overlap(
  array(x, CAST(NULL AS DOUBLE)),
  array(-x)
)
FROM t_flat
WHERE isnan(x);

-- Spark: true
-- Comet before: NULL
-- Comet after: true

Runtime negation in these examples is intentional. Parquet canonicalizes floating-point NaN values when writing them, so storing a negative or custom-payload NaN in the input file is not enough to reproduce the bug. A distinct representation must be produced after the scan.

There is one important boundary: Spark's flat-array equality considers all NaN representations equal, but still treats positive and negative zero as different values:

SELECT arrays_overlap(array(0.0D), array(-0.0D));

-- Spark: false
-- Comet before and after: false

A general-purpose floating-point normalizer that also merges signed zero would therefore replace the original bug with a different Spark compatibility regression.

Closes #5270.

What changes were proposed in this PR?

The change aligns Comet's existing flat-array comparison with the equality contract Spark already uses for boxed floating-point values. When Comet derives a comparison key for a FLOAT or DOUBLE, every NaN representation is mapped to one canonical key. All other values retain their original bit patterns, so +0.0 and -0.0 remain distinct.

Applying the correction at the shared comparison-key boundary fixes both ways the existing implementation checks for overlap: direct comparisons for small arrays and hash-based lookups for larger arrays. The execution plan, fast-path structure, null propagation, and behavior for non-floating-point values are otherwise unchanged.

This PR intentionally does not change nested-array comparisons. Spark uses a different equality contract for nested values, and the separate nested signed-zero issue is addressed in #5235.

How was this PR tested?

The native Rust regressions construct positive, negative, and signaling NaN bit patterns directly for both floating-point widths. They verify that those values overlap, that signed zeros remain distinct, that null handling remains correct, and that the same behavior holds on both sides of the small-array/hash-lookup threshold. All 23 focused arrays_overlap tests pass.

The Spark regression exercises the actual native execution path against a Parquet-backed table. It first confirms that Parquet canonicalized the stored NaN values, then uses runtime negation to produce a noncanonical representation after the scan. Spark and Comet results are compared for FLOAT and DOUBLE, both argument orders, nullable arrays, signed-zero controls, and arrays large enough to use hash lookup. The focused regression passes on both Spark 3.5 and Spark 4.0.

Native formatting, compilation, and focused tests:

cd native
cargo fmt --all -- --check
cargo build
cargo test -p datafusion-comet-spark-expr --lib \
  array_funcs::arrays_overlap::tests -- --nocapture

Spark 3.5, from the repository root with JDK 17 configured:

./mvnw test -Pspark-3.5 -Dtest=none \
  '-Dsuites=org.apache.comet.CometArrayExpressionSuite floating-point NaN payloads and signed zeros' \
  -Dscalastyle.skip=true

Spark 4.0, using a clean reactor build to avoid mixing Scala versions:

./mvnw clean test -Pspark-4.0 -Dtest=none \
  '-Dsuites=org.apache.comet.CometArrayExpressionSuite floating-point NaN payloads and signed zeros' \
  -Dscalastyle.skip=true

Maven Spotless checks also pass for both Spark profiles.