[core][flink][spark] Refactor global index building and support incremental refresh

[leaves12138]

Purpose

This PR has two related goals:

1. Refactor the global-index build pipeline so scan planning is shared by Core and execution-specific Flink/Spark builders consume an explicit scan result.
2. Add opt-in incremental refresh for data-evolution indexes when indexed values in already covered row ranges change.

Previously, generic and sorted index builders mixed scan planning, mutable build state, and index-file writing. Incremental builds also considered only uncovered row ranges, so an index could remain readable but become stale after partial updates to indexed columns.

Global-index build refactoring

• Introduce ScanResult, which carries the scan snapshot ID, selected row ranges, scan entries, and index entries to delete.
• Move generic index scan planning into Core as GenericGlobalIndexScanner and remove the Flink-specific GenericGlobalIndexBuilder.
• Split the sorted builder into SortedGlobalIndexScanner and SortedGlobalIndexWriter so scan state is returned explicitly instead of retained in mutable builder fields.
• Move shared row-range calculation, split grouping, and shard helpers into GlobalIndexBuilderUtils.
• Make generic and sorted scanners expose consistent scan and incrementalScan flows.
• Adapt the Flink and Spark topology builders to consume ScanResult and keep index writing separate from scan planning.
• Remove test-only overloads and duplicate incremental-range planning from execution modules.

Data-evolution index refresh

• Record the data snapshot scanned by an index build as scanSnapshotId in GlobalIndexMeta.sourceMeta, encoded with a DEIX magic number and version.
• Record the scan snapshot rather than the later index-commit snapshot, so updates committed while a build is running remain newer than the watermark and are found by the next build.
• When global-index.detect-datafile-change=true, select an existing index for refresh when an active data file:
  • belongs to the same partition and bucket,
  • intersects the index row range,
  • physically contains at least one indexed field, resolved by stable field ID, and
  • has maxSequenceNumber greater than the index's scanSnapshotId.
• Merge affected indexed ranges with newly uncovered ranges for generic and sorted indexes, including vector, BTree, bitmap, and full-text implementations.
• Commit deletion of old index files and addition of their replacements atomically, so readers never observe an index-unavailable gap.
• Reject deletion of a missing global-index file by default to detect concurrent replacement. global-index.ignore-missing-delete=true enables permissive behavior when explicitly required.

Data-file change detection is disabled by default because it requires scanning manifest entries. Existing uncovered-range incremental builds retain their previous behavior unless the option is enabled.

Compatibility and scope

Indexes created before DEIX source metadata was introduced do not have a trustworthy scan watermark. This PR intentionally does not perform automatic migration or a one-time refresh for those indexes. They remain treated as covered and must be explicitly rebuilt once to populate sourceMeta before participating in change-based refresh.

User impact

For data-evolution tables with change detection enabled, users can continuously update indexed columns and run incremental index builds. Unaffected ranges remain available, while affected ranges are rebuilt and atomically replace their previous index files. Updates to unrelated physical columns do not trigger a refresh.

Tests

• Source-metadata serialization, magic/version validation, and scan-snapshot propagation.
• Generic and sorted scanner behavior, writer lifecycle, row-range splitting, and shard planning.
• Refresh planning across sequence watermarks, physical field projections, schema renames, multiple indexed fields, partitions, buckets, and overlapping ranges.
• Legacy-index exclusion and no-op incremental builds when no indexed value changed.
• Atomic replacement, strict missing-delete handling, and configurable permissive behavior.
• Flink and Spark coverage for vector, BTree, bitmap, and full-text index build paths.
• Lumina native integration coverage on Linux.

[JingsongLi]

Older versions of the vector index are skipped by the refresh planner when DEIX source metadata is missing; at the same time, the row ranges covered by the index are not added to unindexedRowRanges, resulting in the first automatic refresh—as described in the PR—never being triggered after the upgrade.

[JingsongLi]

The feature “automatically refresh the old vector index once after an upgrade” has not yet been implemented**. The old index lacks sourceMeta, so the current planner skips it entirely; however, the range planner treats the old index as overwritten, so it is neither rebuilt nor included in the incremental refresh. New tests even explicitly assert that the old index “does not automatically refresh,” which contradicts the PR description and previous feedback. It is recommended that this change not be merged for now.

[leaves12138]

@JingsongLi Thanks for pointing this out. The previous wording was inconsistent with the implemented behavior.

After discussion, automatic migration of indexes created before DEIX is intentionally out of scope for this PR. Such indexes do not have a trustworthy scanSnapshotId; they remain treated as covered and must be explicitly rebuilt once before change-based refresh can apply. I have updated the PR description to state this scope clearly.

I also updated the title and description to cover the broader global-index build refactoring included here: ScanResult, Core-level scanners, the sorted Scanner/Writer split, shared row-range planning, and the corresponding Flink/Spark adaptations.

[JingsongLi]

Remove this option, just use IGNORE.

[leaves12138]

OK～

[JingsongLi]

Do we need this option?

[leaves12138]

Fixed

[JingsongLi]

Just one comment left

[JingsongLi]

+1