Add spark-gluten-clickhouse entry (Spark + Gluten with the CH backend)

.gitignore

@@ -29,3 +29,9 @@ hits.vortex
 
 # Python venvs created by install scripts
 myenv
+__pycache__/
+
+# spark-gluten-clickhouse: from-source build tree and symlinked artifacts
+gluten/
+gluten.jar
+libch.so

spark-gluten-clickhouse/README.md

@@ -0,0 +1,34 @@
+This entry runs Apache Spark with the [Apache Gluten](https://gluten.apache.org/) plugin configured to use the **ClickHouse backend** ('ch'). Gluten loads `libch.so` (a fork of ClickHouse v23.1) into the Spark executor JVM and runs the columnar physical plan natively through it. See also [`spark-gluten/`](../spark-gluten/) (Velox backend), [`spark-velox/`](../spark-velox/), and the [accelerators README](../spark/README-accelerators.md).
+
+### Run
+
+`./benchmark.sh` sets a few env vars and delegates to the shared driver
+[`../lib/benchmark-common.sh`](../lib/benchmark-common.sh), which runs the
+per-system scripts (`install`, `load`, `query`, ...) and prints the results in
+the format collected by play.clickhouse.com. `./install` builds everything from
+source (no pre-built bundle is published for the CH backend).
+
+## Notes
+
+### Build
+
+The CH backend is not part of Apache Gluten's release tarball — only the Velox bundle is published. As a result `install` builds two things from source:
+
+1. **`libch.so`** — built from [Kyligence/ClickHouse](https://github.com/Kyligence/ClickHouse) at the org/branch/commit pinned in `gluten/cpp-ch/clickhouse.version`. The build uses Clang 19 / cmake / ninja (Gluten v1.4.0's CH backend requires Clang 19). Its `extern-local-engine` module links against JNI **including AWT**, so `install` uses the full `openjdk-17-jdk` (not `-headless`, which omits `jawt.h`/`libjawt.so` and makes cmake fail with `Could NOT find JNI (missing: AWT)`).
+2. **The Gluten Spark plugin** — built via Maven with `-Pbackends-clickhouse -Pspark-3.5 -Pscala-2.12 -Pdelta`. The `delta` profile is required by the `backends-clickhouse` module's enforcer; `spark-3.5` pins the matching Delta version (3.2.0). JDK 8 is required at compile time (Gluten's POM); Spark itself runs under JDK 17 (see `./query`).
+
+Building libch.so essentially compiles ClickHouse from source: it is **memory-hungry** (Gluten's docs note that 64 GB RAM is recommended). On a c6a.4xlarge (32 GB RAM) the compile may OOM; use c6a.8xlarge or larger for a clean run.
+
+### Configuration
+
+- `spark.gluten.sql.columnar.backend.lib=ch` selects the ClickHouse backend over Velox.
+- `spark.gluten.sql.columnar.libpath=<libch.so>` points to the native library. Gluten's wrapper cmakes into `gluten/cpp-ch/build_ch`, which drives an inner ClickHouse cmake that builds `libch.so` under `gluten/cpp-ch/build/.../extern-local-engine/`; `install` globs for it under `cpp-ch/` and symlinks it as `libch.so` in the entry directory.
+- Memory is split 50/50 between Spark heap and Gluten off-heap, identical to the Velox entry — the CH backend also runs off-heap via JNI.
+- `libch.so` is preloaded into the JVM via `LD_PRELOAD` (set in `query.py`). Because the library carries initial-exec-model TLS, a lazy `dlopen` from the running JVM otherwise fails with `cannot allocate memory in static TLS block`; preloading it at JVM startup — while the static TLS block still has room — avoids this. Gluten's docs use `spark.executorEnv.LD_PRELOAD` for this, but in `local[*]` mode the driver JVM is the executor and launches before that config is read, so the preload is done through the JVM's environment instead.
+- Queries use ClickHouse-style regex backreferences (`\1`) rather than Spark's `$1`, since the regex evaluation happens inside libch.so. See the discussion in [`spark-gluten/README.md`](../spark-gluten/README.md) and [Gluten issue #7545](https://github.com/apache/incubator-gluten/issues/7545).
+
+### Links
+
+- [Gluten ClickHouse-backend getting started](https://gluten.apache.org/docs/get-started/ClickHouse/).
+- [Gluten release page](https://gluten.apache.org/downloads/) (Velox bundles only).
+- [Kyligence/ClickHouse fork](https://github.com/Kyligence/ClickHouse) (the source of libch.so).

spark-gluten-clickhouse/benchmark.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-single"
+export BENCH_RESTARTABLE=no
+# Single-process engine: each query forks a fresh full-machine process with no
+# shared scheduler across connections, so the concurrent-QPS test only
+# oversubscribes RAM rather than measuring throughput. Skip it by default;
+# override BENCH_CONCURRENT_DURATION to re-enable. See issue #946.
+export BENCH_CONCURRENT_DURATION="${BENCH_CONCURRENT_DURATION:-0}"
+exec ../lib/benchmark-common.sh

spark-gluten-clickhouse/check

@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+
+# shellcheck disable=SC1091
+source myenv/bin/activate
+python3 -c 'import pyspark' >/dev/null 2>&1
+[ -f gluten.jar ]
+[ -f libch.so ]

spark-gluten-clickhouse/data-size

@@ -0,0 +1,4 @@
+#!/bin/bash
+set -e
+
+du -b hits.parquet | cut -f1

spark-gluten-clickhouse/install

@@ -0,0 +1,216 @@
+#!/bin/bash
+set -e
+
+# Spark + Apache Gluten with the ClickHouse backend ('ch'). Unlike the Velox
+# backend, no pre-built bundle is published for the CH backend, so we build
+# both libch.so (a ClickHouse fork) and the Gluten Spark plugin from source.
+#
+# The libch.so compile is essentially a ClickHouse build: RAM-hungry (Gluten's
+# docs recommend >= 64 GB) and slow. Run on c6a.8xlarge or larger; c6a.4xlarge
+# (32 GB) may OOM.
+#
+# Note: Keep in sync with spark-*/install (see README-accelerators.md).
+
+# Track Gluten main, not a release tag: the CH backend's pinned ClickHouse fork
+# is only rebased forward on main, so release tags carry an old ClickHouse. This
+# commit pulls ClickHouse 25.12.10.7 (cpp-ch/clickhouse.version =>
+# Kyligence rebase_ch/20260425-25.12.10.7); v1.4.0 was on a ~2-year-old fork.
+GLUTEN_COMMIT=8b701473f465e4d7eb05335b0529bad9327db8b3  # apache/gluten main @ 2026-07-24
+SPARK_PROFILE=spark-3.5
+
+# Build prerequisites:
+#  - Java 8 (headless) to build Gluten via Maven (Gluten's pom requires JDK 8).
+#  - Java 17, the FULL jdk (not -headless), to run Spark AND to satisfy the
+#    libch.so build's JNI/AWT dependency: ClickHouse's extern-local-engine
+#    links against jawt.h/libjawt.so, which the -headless packages omit, so
+#    its cmake fails with `Could NOT find JNI (missing: AWT)` /
+#    `JAVA_AWT_INCLUDE_PATH-NOTFOUND`. openjdk-17-jdk ships jawt.h and pulls
+#    in the non-headless JRE that ships libjawt.so.
+#  - Clang 19, cmake, ninja, etc. to build libch.so (Gluten's CH backend
+#    requires Clang 19; see get-started/ClickHouse.md).
+sudo apt-get update -y
+sudo apt-get install -y python3-pip python3-venv \
+    openjdk-8-jdk-headless openjdk-17-jdk \
+    maven git cmake ccache ninja-build nasm yasm gawk \
+    lsb-release wget software-properties-common gnupg
+
+# ClickHouse (libch.so) starts core-scaled thread pools at init; on a 192-core
+# box the process needs far more threads and memory-map areas than the defaults
+# allow, and JavaSparkContext init otherwise dies with `OutOfMemoryError:
+# unable to create native thread`. Raise the kernel ceilings once here
+# (best-effort; ./query also raises the matching per-process ulimits).
+sudo sysctl -w kernel.threads-max=4000000 kernel.pid_max=4000000 \
+    vm.max_map_count=2000000 >/dev/null 2>&1 || true
+
+# Install Clang 19 via apt.llvm.org.
+if ! command -v clang-19 >/dev/null 2>&1; then
+    wget -O - https://apt.llvm.org/llvm.sh | sudo bash -s -- 19
+fi
+
+export CC=clang-19
+export CXX=clang++-19
+
+ARCH=$(dpkg --print-architecture)
+
+# pyspark venv.
+if [ ! -d myenv ]; then
+    python3 -m venv myenv
+fi
+# shellcheck disable=SC1091
+source myenv/bin/activate
+# pyspark matches the spark-3.5 profile Gluten is built against below (Spark
+# 3.5.5), so the plugin jar's Spark APIs line up with the runtime.
+pip install -q pyspark==3.5.5 psutil
+
+# Clone Gluten and the Kyligence ClickHouse fork that the CH backend wraps.
+# The CH fork org/branch/commit are pinned in Gluten's cpp-ch/clickhouse.version.
+GLUTEN_DIR="$PWD/gluten"
+if [ ! -d "$GLUTEN_DIR" ]; then
+    # Shallow-fetch the pinned commit. `git clone --branch` accepts only refs,
+    # not SHAs, so init + fetch-by-SHA (GitHub allows reachable-SHA fetches).
+    git init -q "$GLUTEN_DIR"
+    git -C "$GLUTEN_DIR" remote add origin https://github.com/apache/gluten.git
+    git -C "$GLUTEN_DIR" fetch -q --depth 1 origin "$GLUTEN_COMMIT"
+    git -C "$GLUTEN_DIR" checkout -q FETCH_HEAD
+fi
+
+CH_VERSION_FILE="$GLUTEN_DIR/cpp-ch/clickhouse.version"
+CH_ORG=$(grep '^CH_ORG=' "$CH_VERSION_FILE" | cut -d= -f2)
+CH_BRANCH=$(grep '^CH_BRANCH=' "$CH_VERSION_FILE" | cut -d= -f2)
+CH_DIR="$GLUTEN_DIR/cpp-ch/ClickHouse"
+if [ ! -d "$CH_DIR" ]; then
+    git clone --recursive --shallow-submodules \
+        --branch "$CH_BRANCH" \
+        "https://github.com/${CH_ORG}/ClickHouse.git" "$CH_DIR"
+fi
+
+# Build libch.so WITHOUT jemalloc. ClickHouse enables jemalloc by default and
+# Gluten hardcodes -DENABLE_JEMALLOC=ON for the inner ClickHouse cmake in
+# cpp-ch/CMakeLists.txt (there is no env hook to override it). jemalloc is only
+# safe when it interposes malloc/free process-wide, which requires preloading
+# libch.so. We load it lazily (System.load in query.py, via the static-TLS
+# tunable) rather than LD_PRELOAD, so jemalloc does NOT interpose: the first
+# time ClickHouse frees a pointer the system/JVM allocator made, jemalloc's
+# radix tree has no metadata for it and SIGSEGVs in rtree_metadata_read during
+# the CH backend's native init (CHListenerApi.initialize -> nativeInitNative).
+# Building without jemalloc makes ClickHouse use the system allocator uniformly,
+# removing the cross-allocator boundary. Patch + verify so a future Gluten bump
+# that changes the flag fails loudly instead of silently re-enabling jemalloc.
+CH_CMAKELISTS="$GLUTEN_DIR/cpp-ch/CMakeLists.txt"
+sed -i 's/-DENABLE_JEMALLOC=ON/-DENABLE_JEMALLOC=OFF/g' "$CH_CMAKELISTS"
+if grep -q -- '-DENABLE_JEMALLOC=ON' "$CH_CMAKELISTS" \
+   || ! grep -q -- '-DENABLE_JEMALLOC=OFF' "$CH_CMAKELISTS"; then
+    echo "ERROR: could not disable jemalloc in $CH_CMAKELISTS" >&2
+    exit 1
+fi
+
+# Scala 2.12 compat for the Delta 3.3 backend sources. On Gluten main,
+# backends-clickhouse/src-delta33/.../delta/Snapshot.scala calls Option.when,
+# which only exists in Scala 2.13's stdlib -- but scala-2.12 is Gluten's default
+# profile and matches the pyspark wheel we run, so the CH backend module fails
+# to compile ("value when is not a member of object Option"). Rather than pull
+# in a whole Scala-2.13 Spark stack, drop in Option.when/unless as implicit
+# extensions on the Option companion (compiled from src-delta33, so it is on the
+# module's own source path) and import it into Snapshot.scala. Under a scala-2.13
+# build the real Option.when takes precedence and this shim is simply unused, so
+# it is safe across versions. Only Snapshot.scala uses Option.when in
+# backends-clickhouse (verified by code search), so this one file is enough.
+DELTA33_SCALA="$GLUTEN_DIR/backends-clickhouse/src-delta33/main/scala"
+SNAPSHOT_SCALA="$DELTA33_SCALA/org/apache/spark/sql/delta/Snapshot.scala"
+if [ -f "$SNAPSHOT_SCALA" ]; then
+    mkdir -p "$DELTA33_SCALA/org/apache/gluten/compat"
+    cat > "$DELTA33_SCALA/org/apache/gluten/compat/OptionCompat.scala" <<'SCALA'
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.compat
+
+// Scala 2.12 lacks Option.when/Option.unless (added in 2.13). Provide them as
+// implicit extensions on the Option companion so Delta 3.3 sources compile
+// under scala-2.12 (Gluten's default profile). Under scala-2.13 the real
+// stdlib methods take precedence and this shim is unused.
+object OptionCompat {
+  implicit class OptionCompanionOps(companion: Option.type) {
+    def when[A](cond: Boolean)(a: => A): Option[A] = if (cond) Some(a) else None
+    def unless[A](cond: Boolean)(a: => A): Option[A] = if (cond) None else Some(a)
+  }
+}
+SCALA
+    if ! grep -q 'org.apache.gluten.compat.OptionCompat' "$SNAPSHOT_SCALA"; then
+        sed -i '/^package /a import org.apache.gluten.compat.OptionCompat._' "$SNAPSHOT_SCALA"
+    fi
+    if ! grep -q 'org.apache.gluten.compat.OptionCompat._' "$SNAPSHOT_SCALA"; then
+        echo "ERROR: failed to inject the Option.when shim into Snapshot.scala" >&2
+        exit 1
+    fi
+fi
+
+# Build libch.so via Gluten's own wrapper. It cmakes cpp-ch into
+# cpp-ch/build_ch, whose build_ch target drives an inner ClickHouse cmake that
+# builds `--target libch` into cpp-ch/build/, so the artifact lands under
+# cpp-ch/build/.../extern-local-engine/. Glob for it rather than hardcoding —
+# the two build dirs (build_ch wrapper vs build inner) are easy to confuse.
+# Honors the CC/CXX exported above; JAVA_HOME points cmake's FindJNI at the
+# full JDK 17 so it locates jawt.h/libjawt.so (see the JDK note above).
+# Note: the wrapper was renamed build_clickhouse.sh -> build-clickhouse.sh
+# after v1.4.0; glob so either name works across Gluten versions.
+export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-${ARCH}"
+LIBCH_SO=$(find "$GLUTEN_DIR/cpp-ch" -name libch.so -type f 2>/dev/null | head -n1)
+if [ -z "$LIBCH_SO" ]; then
+    BUILD_CH_SCRIPT=$(find "$GLUTEN_DIR/ep/build-clickhouse" \
+        -name 'build[-_]clickhouse.sh' -type f 2>/dev/null | head -n1)
+    if [ -z "$BUILD_CH_SCRIPT" ]; then
+        echo "ERROR: could not find the build-clickhouse wrapper script" >&2
+        exit 1
+    fi
+    bash "$BUILD_CH_SCRIPT"
+    LIBCH_SO=$(find "$GLUTEN_DIR/cpp-ch" -name libch.so -type f 2>/dev/null | head -n1)
+fi
+if [ -z "$LIBCH_SO" ]; then
+    echo "ERROR: libch.so not found after build" >&2
+    exit 1
+fi
+
+# Build the Gluten Spark plugin against the CH backend. JDK 8 is required at
+# compile time per Gluten's pom; Spark itself runs under JDK 17 in ./query.
+# pyspark wheels ship Scala 2.12 jars, so build with scala-2.12 to match.
+# The delta profile is mandatory: backends-clickhouse's pom carries an
+# `enforce-delta-profile` enforcer rule that fails the build with
+# `"-P delta" must be set when building Gluten with ClickHouse backend` unless
+# it is active. The spark-3.5 profile pins the matching delta.version (3.3.2 on
+# current main), so -Pdelta needs no extra version flag.
+JAVA_HOME_8="/usr/lib/jvm/java-8-openjdk-${ARCH}"
+GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1)
+if [ -z "$GLUTEN_JAR" ]; then
+    (
+        cd "$GLUTEN_DIR"
+        export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g"
+        JAVA_HOME="$JAVA_HOME_8" PATH="$JAVA_HOME_8/bin:$PATH" \
+            mvn -B clean package \
+                -Pbackends-clickhouse -P"$SPARK_PROFILE" -Pscala-2.12 -Pdelta \
+                -DskipTests -Dcheckstyle.skip -Dspotless.check.skip=true
+    )
+    GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1)
+fi
+if [ -z "$GLUTEN_JAR" ]; then
+    echo "ERROR: could not locate built Gluten CH-backend jar" >&2
+    ls "$GLUTEN_DIR/backends-clickhouse/target/" >&2 || true
+    exit 1
+fi
+
+# query.py expects the jar and native library as ./gluten.jar and ./libch.so.
+ln -sf "$GLUTEN_JAR" gluten.jar
+ln -sf "$LIBCH_SO" libch.so

spark-gluten-clickhouse/load

@@ -0,0 +1,6 @@
+#!/bin/bash
+set -e
+
+# Nothing to load: query.py reads hits.parquet directly. Just flush writeback
+# so the parquet the harness downloaded is durably on disk before queries run.
+sync