From e7589adcc5f7fb8072434de51718860e7a651346 Mon Sep 17 00:00:00 2001 From: shoumikhin Date: Tue, 11 Aug 2026 23:08:08 -0700 Subject: [PATCH 1/6] Split the runtime out of the Python bindings extension ## The problem `pip install executorch` gives you the Python half of ExecuTorch and nothing a C++ program can link. Everything is fused into one large Python extension file, so a C++ developer has to clone the repository, sync submodules, and build from source. There is a correctness problem underneath the packaging one. Because the runtime is fused into the extension, anything else that needs it gets its own private copy, and two copies mean two registries. A backend registered in one is invisible to the other. ## The change Build the runtime and the pieces around it as separate shared libraries, and make the Python extension link them instead of embedding them. A shared library is a file a program loads at run time, so several programs can use one copy instead of each carrying its own. ``` executorch/ lib/libexecutorch.so the runtime lib/libexecutorch_kernels_optimized.so CPU operator kernels lib/libexecutorch_backend_xnnpack.so the XNNPACK delegate lib/libexecutorch_threadpool.so one thread pool per process lib/libexecutorch_etdump.so the profiler extension/pybindings/_portable_lib.so now under a megabyte, links the above ``` The extension is much smaller than before, because it no longer contains what it now links. Linux only, and only when the CUDA backend is off. macOS and Windows keep the fused extension because the split relies on ELF sonames, the `$ORIGIN` search-path token and GNU linker options, none of which apply there, and enabling it elsewhere now fails while configuring rather than much later. A CUDA build also keeps the fused extension for now, because the CUDA libraries are not yet shipped alongside the others. ## Test plan Built the wheel from source, installed it into a clean environment, and checked: - exactly one library defines each component, and it is the library that should own it. Counting owners alone would also pass on the old fused layout, which has exactly one too. - the Python extension defines none of them and resolves all of them from outside. - every shipped library loads with no unresolved dependency, and none of them searches a directory from the machine that built the wheel. The build-directory patterns are matched as whole path components, because a bare substring also matched an unrelated directory a user could really have, such as `/home/user/cmake-outputs/torchlibs`, and stripping that breaks a dependency the library legitimately resolves there. - a read-only build output is no longer archived world-writable. The previous code granted write to the group and to everyone via an absolute chmod mode; the fix only grants owner-write, so a `0555` file lands as `0755` instead of `0777`. - a custom operator library compiles and links against the shipped Python extension, which is the existing contract this must not break. Linking these libraries from a standalone C++ application additionally needs an installed CMake package, which the wheel does not carry yet. - the ahead-of-time quantized library records its route to the runtime through the same helper the other targets use. It was written by hand in two blocks that between them covered only the wheel layout and only when the wheel flag was set, so a plain `-DEXECUTORCH_BUILD_SHARED=ON` build left it with no route at all. Checked all four combinations of the shared build and the presence of the Python extension. Ran on Linux x86_64 and aarch64, including a Jetson device. Not fixed here: these libraries bundle third-party code that torch also links, and both keep it visible, so a process holds two definitions of symbols like `pthreadpool_create`. A caller reaches whichever the loader found first. Fixing it means hiding or dropping the bundled copies, which is a larger change. ghstack-source-id: b7977f7b24091ed1948635588bfb9841780c4303 ghstack-comment-id: 5200527760 Pull-Request: https://github.com/pytorch/executorch/pull/21610 --- .ci/scripts/tests/test_wheel_platform_tag.py | 77 + .ci/scripts/wheel/test_linux.py | 9 + .ci/scripts/wheel/test_linux_aarch64.py | 10 + .ci/scripts/wheel/test_shared_libraries.py | 1501 +++++++++++++++++ .../workflows/build-wheels-aarch64-linux.yml | 2 + .github/workflows/build-wheels-linux.yml | 2 + .github/workflows/build-wheels-macos.yml | 2 + .github/workflows/build-wheels-windows.yml | 2 + CMakeLists.txt | 290 +++- backends/qualcomm/CMakeLists.txt | 33 +- backends/xnnpack/CMakeLists.txt | 38 +- codegen/tools/CMakeLists.txt | 15 +- configurations/CMakeLists.txt | 19 + devtools/bundled_program/CMakeLists.txt | 9 +- devtools/etdump/CMakeLists.txt | 37 +- extension/llm/custom_ops/CMakeLists.txt | 36 +- extension/llm/runner/CMakeLists.txt | 4 + extension/threadpool/CMakeLists.txt | 35 +- extension/training/CMakeLists.txt | 57 +- install_utils.py | 43 +- kernels/portable/CMakeLists.txt | 4 +- kernels/quantized/CMakeLists.txt | 41 +- pyproject.toml | 1 + .../exec_aten/util/tensor_dimension_limit.h | 2 + setup.py | 210 ++- tools/cmake/Codegen.cmake | 40 +- tools/cmake/Utils.cmake | 252 ++- tools/cmake/cmake_cache.py | 16 +- tools/cmake/executorch-wheel-config.cmake | 56 +- tools/cmake/preset/default.cmake | 4 +- tools/cmake/preset/pybind.cmake | 13 + 31 files changed, 2719 insertions(+), 141 deletions(-) create mode 100644 .ci/scripts/tests/test_wheel_platform_tag.py create mode 100644 .ci/scripts/wheel/test_shared_libraries.py diff --git a/.ci/scripts/tests/test_wheel_platform_tag.py b/.ci/scripts/tests/test_wheel_platform_tag.py new file mode 100644 index 00000000000..415ef29582c --- /dev/null +++ b/.ci/scripts/tests/test_wheel_platform_tag.py @@ -0,0 +1,77 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +"""Unit tests for the wheel platform tag comparison. + +Here rather than in the wheel checks, because the decision under test is a pure +function of two strings. Running it as part of the wheel checks meant it needed eleven +built wheels to exercise one comparison, and it still could not run on a machine that +had not built one. + +The comparison had a real defect that this covers. The release pipeline builds in a +manylinux image and rewrites the wheel's file name, so the tag on the file and the tag +auditwheel reports never agree in spelling, and comparing them as text rejected every +correct wheel. No local build reproduces that rewrite, so nothing short of a unit test +catches it before CI. +""" + +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "wheel")) + +from test_shared_libraries import ( # noqa: E402 + _tag_architectures_match, + _wheel_architecture, +) + + +@pytest.mark.parametrize( + "claimed,supported", + [ + # What the release pipeline actually produces: it builds in a manylinux image + # and rewrites the file name, while auditwheel reports a plain linux tag + # because the wheel depends on torch without vendoring torch's libraries. + ("manylinux_2_28_x86_64", "linux_x86_64"), + ("manylinux_2_28_aarch64", "linux_aarch64"), + # The legacy spelling, which has no underscore before its version. + ("manylinux2014_x86_64", "linux_x86_64"), + ("manylinux2014_aarch64", "linux_aarch64"), + # A local build, where nothing rewrites the name. + ("linux_x86_64", "linux_x86_64"), + ("linux_aarch64", "linux_aarch64"), + ], +) +def test_accepts_tags_the_release_pipeline_produces(claimed, supported): + assert _tag_architectures_match(claimed, supported) is True + + +@pytest.mark.parametrize( + "claimed,supported", + [ + ("manylinux_2_28_aarch64", "linux_x86_64"), + ("manylinux_2_28_x86_64", "linux_aarch64"), + ("linux_aarch64", "linux_x86_64"), + ], +) +def test_rejects_an_architecture_mismatch(claimed, supported): + """A wheel labelled for the wrong architecture installs where it cannot run.""" + assert _tag_architectures_match(claimed, supported) is False + + +@pytest.mark.parametrize("tag", ["win_amd64", "macosx_11_0_arm64", "any", "", "linux"]) +def test_reports_a_tag_it_cannot_read(tag): + """None, not False, so an unreadable tag is not mistaken for a mismatch.""" + assert _wheel_architecture(tag) is None + assert _tag_architectures_match(tag, "linux_x86_64") is None + + +def test_reads_every_architecture_the_project_builds_for(): + for architecture in ("x86_64", "aarch64", "i686", "ppc64le", "s390x", "armv7l"): + assert _wheel_architecture(f"linux_{architecture}") == architecture + assert _wheel_architecture(f"manylinux_2_28_{architecture}") == architecture diff --git a/.ci/scripts/wheel/test_linux.py b/.ci/scripts/wheel/test_linux.py index 812eec89215..d76ed6b2462 100644 --- a/.ci/scripts/wheel/test_linux.py +++ b/.ci/scripts/wheel/test_linux.py @@ -7,8 +7,11 @@ # LICENSE file in the root directory of this source tree. import platform +import tempfile +from pathlib import Path import test_base +import test_shared_libraries from examples.models import Backend, Model if __name__ == "__main__": @@ -41,6 +44,12 @@ test_base.test_cmsis_nn_install() + # The wheel ships the runtime, the kernels, the delegate, the thread + # pool and the profiler as separate shared libraries now, so check that + # each has exactly one owner and that all of them are loadable. + with tempfile.TemporaryDirectory() as work_dir: + test_shared_libraries.run_tests(Path(work_dir)) + test_base.run_tests( model_tests=[ test_base.ModelTest( diff --git a/.ci/scripts/wheel/test_linux_aarch64.py b/.ci/scripts/wheel/test_linux_aarch64.py index c0cca95b3fb..b268c72cea3 100644 --- a/.ci/scripts/wheel/test_linux_aarch64.py +++ b/.ci/scripts/wheel/test_linux_aarch64.py @@ -5,7 +5,11 @@ # This source code is licensed under the BSD-style license found in the # LICENSE file in the root directory of this source tree. +import tempfile +from pathlib import Path + import test_base +import test_shared_libraries from examples.models import Backend, Model if __name__ == "__main__": @@ -26,6 +30,12 @@ ), f"OpenvinoBackend not found in registered backends: {registered}" print("✓ OpenvinoBackend is registered") + # The wheel ships the runtime, the kernels, the delegate, the thread pool and + # the profiler as separate shared libraries now, so check that each has + # exactly one owner and that all of them are loadable. + with tempfile.TemporaryDirectory() as work_dir: + test_shared_libraries.run_tests(Path(work_dir)) + test_base.run_tests( model_tests=[ test_base.ModelTest( diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py new file mode 100644 index 00000000000..ad69e432862 --- /dev/null +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -0,0 +1,1501 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +"""Checks that the wheel ships its runtime as separate shared libraries. + +The Python bindings extension used to contain the runtime, the registries, the +CPU kernels, the XNNPACK delegate and the profiler, all statically linked into +one file. It now links them as shared libraries the wheel ships alongside it. +These checks run against the installed wheel only; they never look at the source +tree's build directory, because a checkout on the module search path makes every +check below pass while inspecting the wrong thing. + +The properties verified here are the ones the split exists to create: + +1. Each component has exactly one definer, and it is the library that is meant + to own it. Counting definers alone is not enough: the monolithic layout also + had exactly one of each, inside the Python extension. +2. The extension contains none of those components and depends on every shipped + library instead. +3. Every shipped library loads with no absolute runtime search path, including + after being moved, so the wheel is relocatable rather than only working on + the machine that built it. +""" + +import importlib.metadata +import importlib.util +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +# Registry entry points. A second definer of any of these means a second +# process-wide registry. +_REGISTRY_SYMBOLS = ( + "executorch::runtime::register_backend", + "executorch::runtime::get_num_registered_backends", + "executorch::runtime::get_backend_class", +) + +# The thread pool accessor. A second definer means a second pool, which +# oversubscribes the CPU because each pool sizes itself to all cores. +_THREADPOOL_SYMBOLS = ("executorch::extension::threadpool::get_threadpool",) + +# A representative operator from the merged CPU kernels. A second definer means +# the operators are registered twice, which aborts at startup. +_KERNEL_SYMBOLS = ("torch::executor::native::abs_out",) + +# The registry entry points, kept separate from the kernel implementations above. +# A library that carries its own copy of these has its own registration code, which +# is what this split is meant to prevent: one owner of the operator table. Checking +# only a kernel implementation would miss that entirely. +_KERNEL_REGISTRY_SYMBOLS = ( + "executorch::runtime::register_kernels", + "executorch::runtime::get_registered_kernels", +) + +# A representative symbol from the XNNPACK delegate. A second definer means the +# process carries two copies of the delegate. +_XNNPACK_SYMBOLS = ( + "executorch::backends::xnnpack::XnnpackBackendOptions::workspace_manager", +) + +# Third-party code the shipped libraries bundle rather than depend on. These are C +# symbols with default visibility, so a second copy in the same process is not a +# duplicate of ExecuTorch's own code but it is still two thread pools or two +# XNNPACK runtimes, and which one a caller reaches depends on load order. +# +# Checked separately from the wrapper symbols above because the wrappers can each +# have exactly one owner while the bundled code underneath them does not. That is +# the same failure the split exists to prevent, reached by a different route. +_BUNDLED_THREADPOOL_SYMBOLS = ("pthreadpool_create", "cpuinfo_initialize") +_BUNDLED_XNNPACK_SYMBOLS = ("xnn_create_runtime_v4",) + +# A representative symbol from the profiler. A second definer means two event +# tracers, so a trace records only part of what ran. +_ETDUMP_SYMBOLS = ("executorch::etdump::ETDumpGen::ETDumpGen",) + +# `nm -DC` prints " " for a definition and +# " U " for an undefined reference. +_DEFINED = re.compile(r"^[0-9a-fA-F]+\s+(?P[A-Za-z])\s+(?P.+)$") + +# Symbol kinds that mean the object owns the code or storage. +_OWNING_KINDS = frozenset("TtBbDdGgSsRrWV") + + +def _declared_requirements() -> set: + """Import names the installed wheel declares a requirement for. + + Used to tell a package this environment simply does not have from one the wheel + said it needed, because only the second is a packaging defect. + + Three sources, because none alone is sufficient. importlib's reverse map is + authoritative where a distribution is INSTALLED, which is what makes + PyYAML -> yaml, ruamel.yaml -> ruamel and hydra-core -> hydra resolve correctly. + But it enumerates installed distributions only, so a declared dependency that is + MISSING can never appear in it, which is precisely the case this function exists + to catch. The transformed distribution name covers most of the rest. + + Some distributions import under a name no transformation produces, so those are + listed. Measured against the wheel's own declared list: py-cpuinfo -> cpuinfo, + PyYAML -> yaml and hydra-core -> hydra are all missed by the transformation, and + each would turn a dependency the wheel failed to install into a quiet skip. + """ + # Distribution name to import name, where the two are unrelated. Keyed on the + # normalised distribution name so a change in case or separator still matches. + unrelated_import_names = { + "py_cpuinfo": ("cpuinfo",), + "pyyaml": ("yaml", "_yaml"), + "hydra_core": ("hydra",), + "scikit_learn": ("sklearn",), + "typing_extensions": ("typing_extensions",), + "pillow": ("PIL",), + "protobuf": ("google",), + "opencv_python": ("cv2",), + } + try: + from importlib.metadata import packages_distributions, requires + except ImportError: # pragma: no cover + return set() + try: + declared = requires("executorch") or [] + except Exception: + return set() + + wanted, names = set(), set() + for requirement in declared: + # Only the distribution name, dropping any version specifier, extra or + # environment marker. + name = re.split(r"[\s;\[<>=!~(]", requirement.strip(), maxsplit=1)[0] + if not name: + continue + normalised = name.lower().replace("-", "_").replace(".", "_") + wanted.add(normalised) + # The likely import name, so a MISSING declared dependency is still + # recognised as declared rather than silently skipped. + names.add(name) + names.add(name.replace("-", "_")) + names.add(name.split(".")[0]) + names.update(unrelated_import_names.get(normalised, ())) + + for import_name, distributions in packages_distributions().items(): + for distribution in distributions: + if distribution.lower().replace("-", "_").replace(".", "_") in wanted: + names.add(import_name) + return names + + +def _nm_defined_args(): + """The nm flags that list what a library defines.""" + return ["-DC"] + + +def _installed_package_dir() -> Path: + """The installed executorch package, never the source checkout. + + Enforced rather than assumed. Python puts the working directory on the module + search path, so running from a checkout resolves `executorch` to the source + tree, where there are no shipped libraries and every check below passes while + testing nothing. That is worse than a failure, because it looks like a pass. + """ + import executorch + + paths = [Path(entry).resolve() for entry in executorch.__path__] + # Every entry, not just the first. This is a namespace package, so a checkout on + # the module search path adds a second entry, and a module can then resolve from + # the checkout while the first entry still looks like a clean install. + outside = [ + path + for path in paths + if "site-packages" not in path.parts and "dist-packages" not in path.parts + ] + assert not outside, ( + f"executorch also resolves through {outside}, which is not an installed " + "package. Run this from a directory that contains no executorch checkout, " + "or the checks silently inspect the source tree instead of the wheel." + ) + assert len(paths) == 1, ( + f"executorch resolves through {len(paths)} paths ({paths}). Even when all of " + "them are installs, a module could come from either, so which artifact is " + "under test is ambiguous." + ) + return paths[0] + + +def _tool(name: str): + """Locate a build tool, including one pip installed beside this interpreter. + + `shutil.which` searches PATH only, and a virtual environment's `bin` is on PATH + only when the environment is activated. These tests are normally run by invoking + the interpreter directly, so a tool pip installed into that environment is present + on disk and invisible to a PATH search. + """ + found = shutil.which(name) + if found: + return found + beside = Path(sys.executable).parent / name + return str(beside) if beside.is_file() else None + + +def _shipped_shared_objects(package_dir: Path): + """Every shared object the wheel installed. + + Asserts it found some, because a wheel that installed none would let every check that walks this list + report success having examined nothing. + """ + found = [ + path + for path in sorted(package_dir.rglob("*.so*")) + if path.is_file() and not path.is_symlink() + ] + assert ( + found + ), f"no shared objects found under {package_dir}, so nothing below would be checking them" + return found + + +def _shipped_runtime_libraries(package_dir: Path): + """The libraries the wheel ships under lib/, whatever it names them. + + One place, because three checks previously spelled the pattern themselves as + `*.so.*` and every one of them silently stopped matching when the build moved to + unversioned names. The failure was invisible: a check that finds nothing reports + that the component is absent, which each of those treats as acceptable. + + Matches a versioned name too, so a build that does set SOVERSION is still found. + """ + lib_dir = package_dir / "lib" + if not lib_dir.is_dir(): + return [] + return [ + path + for path in sorted(lib_dir.glob("lib*.so*")) + if path.is_file() and not path.is_symlink() + ] + + +def _defines_symbol(library: Path, symbol: str) -> bool: + """Whether `library` owns a definition of `symbol`, read from the dynamic table. + + Limited to exported definitions on purpose, because that is all the shipped + artifacts carry: every library the wheel ships is stripped, so the static symbol + table `nm -C` would read is gone. Measured on a real wheel, `nm -C` finds zero of + these sentinels while `nm -DC` finds them, so widening the reader would turn this + check off rather than strengthen it. + + A duplicate hidden behind non-default visibility would therefore not be seen here. + That is a real gap in this check: it reads what a library exports, so a second copy + compiled with hidden visibility is invisible to it. Catching that needs a running + process, which counts what actually registered rather than what is visible. + """ + result = subprocess.run( + [_tool("nm"), "-DC", str(library)], capture_output=True, text=True, check=False + ) + if result.returncode != 0: + # A file that is not an object file at all is not this check's concern: something whose + # name merely ends in .so must not abort the run. A shipped library the reader cannot + # parse is different, because a real definition could be hiding inside it, and reporting + # "defines nothing" would let a duplicate pass. The ELF magic bytes tell them apart + # without depending on the reader's wording. + with library.open("rb") as handle: + is_object_file = handle.read(4) == b"\x7fELF" + assert not is_object_file, ( + f"nm could not read {library.name}, which is a shipped object file, so the symbol " + f"checks cannot be trusted: {result.stderr.strip()[:200]}" + ) + return False + for line in result.stdout.splitlines(): + if symbol not in line: + continue + match = _DEFINED.match(line) + if ( + match + and match.group("name").startswith(symbol) + and match.group("kind") in _OWNING_KINDS + ): + return True + return False + + +def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None: + """At most one shipped library may define each of `symbols`. + + The owner is named where one is expected, because counting definers alone does + not prove the split happened: the monolithic layout has exactly one definer too, + the Python extension. Requiring the symbol to live in the library that is + supposed to own it is what distinguishes the two. + + A component the wheel does not ship at all is a valid configuration, not a + fault. Delegates and kernel sets are build options, so a wheel built without one + has zero definers and is reported as such. What must never happen is two. + """ + assert _tool("nm") is not None, "nm is required to inspect the wheel" + + package_dir = _installed_package_dir() + libraries = _shipped_shared_objects(package_dir) + assert libraries, f"no shared libraries found under {package_dir}" + + # Every symbol is resolved before anything is reported, so a component that is only + # half present is described as such rather than looking like one that is absent. + found = { + symbol: [lib for lib in libraries if _defines_symbol(lib, symbol)] + for symbol in symbols + } + # A component is either wholly present or wholly absent. Some symbols defined and + # others not means a partial build, which is neither of those and is a fault. + present = {symbol for symbol, definers in found.items() if definers} + if not present: + # When the caller has already established that the owner library ships, finding none of its + # symbols is a fault rather than an absence. Returning success here made this a no-op the moment + # a sentinel symbol was renamed or inlined, which turns the ownership check off without anyone + # noticing, and one of these sentinels is a two-line accessor. + assert owner is None, ( + f"the wheel ships {owner}, which owns the {what}, but none of its symbols " + f"{sorted(found)} are defined anywhere. Either the sentinel symbols were renamed or " + "inlined, in which case this check needs updating, or the library is empty." + ) + print(f"- this wheel ships no {what}, nothing to check") + return + assert len(present) == len(found), ( + f"the wheel defines only part of the {what}: {sorted(present)} are present " + f"and {sorted(set(found) - present)} are not, so it is neither shipped nor " + "absent" + ) + + for symbol, definers in found.items(): + pretty = [str(lib.relative_to(package_dir)) for lib in definers] + assert len(definers) == 1, ( + f"expected at most one library to define {symbol}, found " + f"{len(definers)}: {pretty}. More than one definition means the " + f"process has more than one {what}." + ) + if owner is not None: + assert definers[0].name.startswith(owner), ( + f"{symbol} is defined by {pretty[0]}, but it belongs in {owner}. One " + "definer is not enough on its own: the monolithic layout this change " + "replaces also had exactly one, inside the Python extension." + ) + where = f" owned by {owner}" if owner else "" + print(f"✓ single {what}{where} across {len(libraries)} shipped libraries") + + +# Each component the wheel ships as its own library, the symbols that identify it, +# and the library that must own them. `required` says whether the owner has to be +# present: the optimized kernels are optional, because a wheel built without them +# deliberately links the portable ops into the Python extension instead, which is a +# supported configuration rather than a duplicate. +# +# A table rather than one function per component, because the per-function form let +# one of them drift: it looked up its library with its own glob, which silently +# stopped matching when the libraries were renamed while the others kept working. +_OWNED_COMPONENTS = ( + ("backend registry", _REGISTRY_SYMBOLS, "libexecutorch.so", True), + ("operator registry", _KERNEL_REGISTRY_SYMBOLS, "libexecutorch.so", True), + ("thread pool", _THREADPOOL_SYMBOLS, "libexecutorch_threadpool.so", True), + ("profiler", _ETDUMP_SYMBOLS, "libexecutorch_etdump.so", True), + ( + "XNNPACK delegate", + _XNNPACK_SYMBOLS, + "libexecutorch_backend_xnnpack.so", + True, + ), + ( + "set of CPU kernels", + _KERNEL_SYMBOLS, + "libexecutorch_kernels_optimized.so", + False, + ), + # The third-party code these libraries bundle, checked separately from the + # wrappers above. A wrapper can have a single owner while the implementation + # underneath it is bundled into two of these, which is two real thread pools or + # two XNNPACK runtimes. + # + # One copy among the libraries this wheel ships, which is what this change + # controls. torch links the same projects and exports the same symbols, so the + # process still holds two definitions and which one a caller reaches depends on + # load order. Fixing that needs an explicit export list: hiding them wholesale + # with --exclude-libs,ALL breaks aarch64, where the optimized kernels resolve + # cpuinfo_initialize from the thread pool across a library boundary. + ( + "bundled thread pool implementation", + _BUNDLED_THREADPOOL_SYMBOLS, + "libexecutorch_threadpool.so", + True, + ), + ( + "bundled XNNPACK runtime", + _BUNDLED_XNNPACK_SYMBOLS, + "libexecutorch_backend_xnnpack.so", + True, + ), +) + + +def test_each_component_has_one_owner() -> None: + """No component may be defined by more than one library the wheel ships. + + This is the property the split exists to create. Two copies of a component mean + two registries or two thread pools in one process, and a static initializer that + registers into a table nothing else reads shows up as an operator missing at run + time rather than as a link error. + """ + shipped = { + path.name for path in _shipped_runtime_libraries(_installed_package_dir()) + } + for what, symbols, owner, required in _OWNED_COMPONENTS: + present = any(name.startswith(owner) for name in shipped) + assert present or not required, ( + f"the wheel ships no {owner}, which owns the {what}. Either packaging " + "dropped it or the build did not produce it." + ) + _assert_single_definer(symbols, what, owner if present else None) + + +def test_python_extensions_import() -> None: + """Every shipped Python extension must import from a clean environment. + + The symbol and dependency checks work on the files. This covers the other + half: an extension can be packaged correctly and still fail to load because a + runtime path does not reach one of its dependencies. Run in a subprocess with + `LD_LIBRARY_PATH` removed so a value from the build environment cannot supply + a path the shipped library is missing. + + The list is discovered from the installed package rather than written here, so + an extension added later is covered without anyone remembering to add it. A + hardcoded list is how the ones this change relinked went untested. + """ + package_dir = _installed_package_dir() + modules = [] + for extension in sorted(package_dir.rglob("*.so")): + # Only Python extensions, which carry the interpreter's suffix. The plain + # shared libraries under lib/ are checked by the load test instead. + if ".cpython-" not in extension.name: + continue + relative = extension.relative_to(package_dir).parent + module = extension.name.split(".", 1)[0] + modules.append(".".join(["executorch", *relative.parts, module])) + assert modules, "the wheel ships no Python extension, which cannot be right" + + # Torch has to be installed, the same as for the dependency check: these + # extensions link it, so without it they cannot import for a reason that says + # nothing about packaging. + if importlib.util.find_spec("torch") is None: + print("- torch is not installed, skipping the extension import check") + return + environment = { + key: value for key, value in os.environ.items() if key != "LD_LIBRARY_PATH" + } + for module in modules: + result = subprocess.run( + [sys.executable, "-c", f"import {module}"], + capture_output=True, + text=True, + check=False, + env=environment, + ) + if result.returncode == 0: + print(f"✓ {module} imports from a clean environment") + continue + # A Python dependency that is simply not installed here, including torch, + # says nothing about how the wheel was built. Only a failure to load a + # native library does. + # A Python package this environment simply does not have says nothing about + # how the wheel was built, PROVIDED the wheel does not claim to require it. + # Match only that shape, so a native load failure reported as + # ModuleNotFoundError is still caught below. + missing_python_package = re.search( + r"ModuleNotFoundError: No module named '(?!executorch)([\w.]+)", + result.stderr, + ) + if missing_python_package: + absent = missing_python_package.group(1).split(".")[0] + # A package the extension needs must appear in the wheel's declared + # requirements. Without this the test was silent when a required package + # was omitted from Requires-Dist: the environment did not have it, so + # the import failed, and the previous rule skipped anything not already + # declared. That treats a missing declaration as coverage rather than + # as the bug it is. + assert absent in _declared_requirements(), ( + f"{module} cannot import because {absent} is missing, and the wheel " + "does not declare it. Add it to install_requires, or the extension " + "silently needs a package a user is not asked to install." + ) + print(f"- {module} needs {absent}, which this environment lacks, skipping") + continue + # Anything else is a real failure to load what the wheel ships: a missing + # native library, an unresolved symbol, or an ABI mismatch. + raise AssertionError( + f"{module} ships in the wheel but does not import: " + f"{result.stderr.strip()[-500:]}" + ) + + +_CUSTOM_OP_SOURCE = """\ +// A custom operator, built the way an out-of-tree project builds one: against the +// shipped Python extension rather than an ExecuTorch source tree. +#include +#include + +namespace { + +executorch::aten::Tensor& custom_double_out( + executorch::runtime::KernelRuntimeContext& context, + const executorch::aten::Tensor& input, + executorch::aten::Tensor& out) { + (void)context; + const float* in = input.const_data_ptr(); + float* dst = out.mutable_data_ptr(); + for (ssize_t i = 0; i < input.numel(); ++i) { + dst[i] = in[i] * 2.0f; + } + return out; +} + +} // namespace + +// The registration macro is the point of the check: it has to compile and resolve +// against the registry the shipped extension provides. +EXECUTORCH_LIBRARY(wheel_check, "custom_double.out", custom_double_out); +""" + + +_CUSTOM_OP_CMAKE = """\ +cmake_minimum_required(VERSION 3.24) +project(custom_op_check CXX) + +find_package(executorch REQUIRED) + +add_library(custom_op_check SHARED custom_op.cpp) +# The legacy contract: a custom-op library links the shipped Python extension, +# which owns the operator registry it registers into. +target_link_libraries(custom_op_check PRIVATE _portable_lib) +# The runtime headers include c10 headers, which belong to torch rather than to +# this wheel, so an out-of-tree operator project supplies them the same way it +# supplies torch itself. The package config does not and should not ship them. +# +# The include directory is passed in rather than found with find_package(Torch), +# because that enables the CUDA language and fails on a machine with a CUDA +# toolkit it cannot probe, which has nothing to do with compiling an operator. +target_include_directories(custom_op_check PRIVATE ${TORCH_INCLUDE_DIR}) +# Deliberately no target_compile_features here. These headers need C++20, and the +# package config is what has to say so. Setting it here would compile the check +# correctly while leaving a real consumer to fail. +""" + + +# Libraries that belong to torch rather than to this wheel. A library here resolves when the +# Python package that owns it is imported, so it is not something this wheel can or should ship. +_TORCH_LIBRARY_PREFIXES = ( + "libtorch", + "libc10", + "libshm", + "libgomp", + "libcudnn", + "libcublas", +) + + +def _is_torch_library(name: str) -> bool: + return name.startswith(_TORCH_LIBRARY_PREFIXES) + + +def test_shipped_libraries_load() -> None: + """Every shipped library must depend only on things that exist. + + The symbol checks prove each component is defined exactly once, but a library + can still be unloadable if it needs something nothing provides, which is a + packaging bug rather than a duplication bug. + + A dependency the wheel ships elsewhere is fine even when `ldd` cannot resolve + it: some extensions are loaded after `import torch` has already brought their + dependencies into the process, so they intentionally carry no path to them. + Only a name nothing in the wheel provides is a real problem. + """ + if _tool("ldd") is None: + print("- ldd not available, skipping the load check") + return + # Torch has to be installed for this to mean anything: several shipped libraries + # depend on it and resolve once it is imported. Without it every one of them looks + # broken, which would report a packaging fault that does not exist. + if importlib.util.find_spec("torch") is None: + print("- torch is not installed, skipping the load check") + return + + package_dir = _installed_package_dir() + libraries = _shipped_shared_objects(package_dir) + shipped = {library.name for library in libraries} + + # A dependency is only excusable when the wheel ships it AND the loader can + # actually reach it from the library that needs it. Loaded-later extensions + # such as the Torch libraries are the real exception: they resolve once the + # Python package that owns them is imported. Anything the wheel itself ships + # must resolve here, because a RUNPATH applies to the library carrying it and + # is not inherited on behalf of a dependency's own dependencies. + broken = {} + unreachable = {} + unresolved = {} + for library in libraries: + resolved = subprocess.run( + # -r resolves data and function symbols too, not just the NEEDED + # entries. A SHARED link does not error on undefined symbols, so + # without this an under-linked library passes here and fails at first + # use instead. + [_tool("ldd"), "-r", str(library)], + capture_output=True, + text=True, + check=False, + # Any LD_LIBRARY_PATH in the build environment would paper over a + # RUNPATH the shipped library is actually missing. + env={ + key: value + for key, value in os.environ.items() + if key != "LD_LIBRARY_PATH" + }, + ) + # ldd reports missing libraries on stdout but undefined symbols on stderr, + # so both streams matter. + combined = resolved.stdout + resolved.stderr + # A non-zero exit with none of the expected text means ldd could not inspect + # the file at all, which a text-only search reads as "nothing wrong". A file + # under lib/ that is not a loadable object is a packaging defect, so treat it + # as one rather than passing it. + if ( + resolved.returncode != 0 + and "not found" not in combined + and "undefined symbol" not in combined + ): + unresolved[str(library.relative_to(package_dir))] = [ + f"ldd could not inspect this file: {combined.strip()[:160]}" + ] + continue + missing = [ + line.split("=>")[0].strip() + for line in combined.splitlines() + if "not found" in line + ] + # Interpreter symbols are excluded rather than whole files. A library that + # is loaded by Python, whether a extension module or an ahead-of-time + # plugin, resolves those only once an interpreter is running, so ldd can + # never resolve them and their absence says nothing about packaging. + # Filtering the symbols rather than guessing from the file name keeps the + # check active for everything else those libraries need. + undefined = [ + line.strip() + for line in combined.splitlines() + if "undefined symbol" in line + and not re.search(r"undefined symbol:\s+_?Py", line) + ] + if undefined: + unresolved[str(library.relative_to(package_dir))] = undefined[:5] + # Torch's own libraries are the documented exception. They are not in this wheel, and a + # library that needs them resolves once the Python package owning them is imported, which + # is how every accelerator and AOT library in this package is used. Treating them as + # missing fails a wheel that works, and it fires only where torch installs its libraries + # somewhere the plain loader search does not reach. + absent = [ + name + for name in missing + if name not in shipped and not _is_torch_library(name) + ] + present_but_unreachable = [name for name in missing if name in shipped] + if absent: + broken[str(library.relative_to(package_dir))] = absent + if present_but_unreachable: + unreachable[str(library.relative_to(package_dir))] = present_but_unreachable + + assert not broken, ( + "shipped libraries need dependencies that nothing provides, so they will " + f"fail to load: {broken}" + ) + assert not unreachable, ( + "shipped libraries need dependencies the wheel ships but the loader " + "cannot reach from them, which usually means a missing RUNPATH entry: " + f"{unreachable}" + ) + assert not unresolved, ( + "shipped libraries reference symbols nothing provides, so they will fail " + f"at first use rather than at load: {unresolved}" + ) + print("✓ every shipped library resolves in an environment with torch present") + + +def test_shipped_libraries_resolve_without_build_tree() -> None: + """A shipped library must resolve using only its relative runtime paths. + + Packaging copies binaries out of the build directory, so they still carry the + absolute paths they were linked with. On the machine that produced the wheel + those paths exist, which means a library whose relative path is wrong can still + resolve and look correct. Anywhere else it would fail. + + Copy each library and its wheel-provided dependencies into a fresh tree that + mirrors the wheel layout, drop every absolute runtime path, and check what is + left is enough. + """ + if _tool("ldd") is None or _tool("patchelf") is None: + print("- ldd or patchelf unavailable, skipping the relocated load check") + return + + package_dir = _installed_package_dir() + libraries = _shipped_shared_objects(package_dir) + environment = { + key: value for key, value in os.environ.items() if key != "LD_LIBRARY_PATH" + } + + with tempfile.TemporaryDirectory() as work_dir: + root = Path(work_dir) / package_dir.name + # Mirror the layout so a relative path such as $ORIGIN/../../lib still + # points where it would in a real install. + for library in libraries: + target = root / library.relative_to(package_dir) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(library, target) + + broken = {} + for library in libraries: + target = root / library.relative_to(package_dir) + current = subprocess.run( + [_tool("patchelf"), "--print-rpath", str(target)], + capture_output=True, + text=True, + check=False, + ).stdout.strip() + relative = [ + entry for entry in current.split(":") if entry.startswith("$ORIGIN") + ] + subprocess.run( + [_tool("patchelf"), "--set-rpath", ":".join(relative), str(target)], + # A failure here would leave the original absolute build paths in + # place, and the check below would then pass by resolving through + # them, which is exactly what this test exists to rule out. + check=True, + ) + resolved = subprocess.run( + [_tool("ldd"), str(target)], + capture_output=True, + text=True, + check=False, + env=environment, + ).stdout + shipped = {item.name for item in libraries} + all_missing = [ + line.split("=>")[0].strip() + for line in resolved.splitlines() + if "not found" in line + ] + # Only wheel-provided dependencies are asserted on, because an external + # one is expected to come from the environment. They are still reported, + # since silently dropping them would hide a library that resolves only + # through an absolute build path. + missing = [name for name in all_missing if name in shipped] + external = [name for name in all_missing if name not in shipped] + if external: + print( + f"- {library.relative_to(package_dir)} also needs " + f"{external} from the environment" + ) + if missing: + broken[str(library.relative_to(package_dir))] = missing + + assert not broken, ( + "shipped libraries only resolve their wheel-provided dependencies " + "through absolute build paths, so they would fail on any other " + f"machine: {broken}" + ) + print("✓ every shipped library resolves without the build tree") + + +def test_custom_op_compiles(work_dir: Path) -> None: + """A custom operator compiles and links against the shipped extension. + + This is how an out-of-tree project adds its own kernels, and it points at the + Python extension rather than the runtime, so it is not covered by the consumer + check above. + """ + # Skipped rather than asserted, the same as every other tool this suite needs. + # A missing compiler says nothing about the wheel, and aborting here would take + # the whole run down with it rather than reporting the one check it prevents. + if _tool("cmake") is None: + print("- cmake unavailable, skipping the custom op check") + return + + package_dir = _installed_package_dir() + if not list(package_dir.glob("extension/pybindings/_portable_lib*")): + print("- the wheel ships no Python extension, skipping the custom op check") + return + + source_dir = work_dir / "custom-op" + build_dir = work_dir / "custom-op-build" + source_dir.mkdir(parents=True, exist_ok=True) + (source_dir / "custom_op.cpp").write_text(_CUSTOM_OP_SOURCE) + (source_dir / "CMakeLists.txt").write_text(_CUSTOM_OP_CMAKE) + + # Torch's include directory is handed over directly, because the runtime headers + # include c10 headers that belong to torch. A real out-of-tree project supplies + # them the same way; the package config has no business shipping another + # project's headers. + if importlib.util.find_spec("torch") is None: + print("- torch is not installed, skipping the custom op check") + return + import torch + + torch_include = Path(torch.__path__[0]) / "include" + + configure = subprocess.run( + [ + _tool("cmake"), + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_PREFIX_PATH={package_dir / 'share' / 'cmake'}", + f"-DTORCH_INCLUDE_DIR={torch_include}", + ], + capture_output=True, + text=True, + check=False, + ) + assert configure.returncode == 0, ( + "a custom operator project cannot configure against the wheel: " + f"{(configure.stderr or configure.stdout).strip()[-600:]}" + ) + + compiled = subprocess.run( + [_tool("cmake"), "--build", str(build_dir)], + capture_output=True, + text=True, + check=False, + ) + assert compiled.returncode == 0, ( + "a custom operator does not compile or link against the shipped extension: " + f"{(compiled.stderr or compiled.stdout).strip()[-800:]}" + ) + produced = list(build_dir.rglob("libcustom_op_check.so")) or list( + build_dir.rglob("custom_op_check.dll") + ) + assert produced, "the custom operator library was not produced" + + # Loaded, not just built. A shared library on Linux is allowed to have + # unresolved symbols, so an under-linked custom operator links successfully and + # fails only when something dlopens it and its registration initialiser runs. + # That is exactly the failure this contract exists to prevent. + if importlib.util.find_spec("torch") is None: + print("- torch is not installed, so the custom operator is only built") + return + loaded = subprocess.run( + [ + sys.executable, + "-c", + "import torch\n" + "from executorch.extension.pybindings import portable_lib\n" + f"torch.ops.load_library({str(produced[0])!r})\n" + "print('loaded')", + ], + capture_output=True, + text=True, + check=False, + env={ + key: value for key, value in os.environ.items() if key != "LD_LIBRARY_PATH" + }, + ) + assert loaded.returncode == 0, ( + "a custom operator built against the shipped extension cannot be loaded, so " + "it would fail at first use rather than at link time: " + f"{(loaded.stderr or loaded.stdout).strip()[-800:]}" + ) + print("✓ a custom operator compiles against the shipped Python extension") + + +def _find_wheel_files() -> list: + """The built wheel files, searched where a build actually leaves them. + + WHEEL_DIR is honoured when set, but it is not set in the wheel-build job, so the + usual output directories are searched too. Without this the check has nothing to + inspect and skips. + """ + candidates = [] + configured = os.environ.get("WHEEL_DIR") + if configured: + candidates.append(Path(configured)) + # The build leaves the wheel in dist/ at the repository root, and this file sits at a + # fixed depth below that root, so the location follows from __file__ rather than from + # the current directory. The release job runs the smoke test from the workspace above + # the repository, where a cwd-relative guess finds nothing. + # + # Guarded because a copy of this file can live outside that layout, where indexing + # past the available parents would raise instead of falling through to the other + # candidates. + here = Path(__file__).resolve() + repository_root = here.parents[3] if len(here.parents) > 3 else here.parent + candidates += [ + repository_root / "dist", + Path.cwd() / "dist", + Path.cwd(), + repository_root / "wheelhouse", + ] + for directory in candidates: + try: + found = sorted(directory.glob("executorch-*.whl")) + except OSError: + continue + if found: + return found + return [] + + +# The architectures a Linux wheel tag can name. Matched by suffix rather than parsed +# positionally, because the version part differs between spellings (linux_x86_64, +# manylinux_2_28_x86_64, manylinux2014_x86_64) enough that a positional pattern picks +# it up as part of the name. +# +# Linux only, which is what this check reads. `arm64` deliberately absent: it is the +# macOS spelling, Linux uses aarch64, and including it made a macosx_11_0_arm64 tag +# look like something this could compare. A tag naming none of these is reported as +# unreadable rather than as a mismatch. +_WHEEL_ARCHITECTURES = ("x86_64", "aarch64", "i686", "ppc64le", "s390x", "armv7l") + + +def _wheel_architecture(tag: str): + """The architecture a platform tag names, or None if it names none of ours.""" + for architecture in _WHEEL_ARCHITECTURES: + if tag.endswith("_" + architecture): + return architecture + return None + + +def _tag_architectures_match(claimed: str, supported: str): + """Whether two platform tags name the same architecture. + + Returned rather than asserted so the same decision can be unit tested without a + wheel. The previous arrangement duplicated the comparison in the test, which meant + the test could pass while the shipped check was wrong, and that is exactly what + happened: the original defect was in how the caller compared the two tags, and a + test that re-implemented the comparison could not see it. + + None for either side means the tag names no architecture this project builds for, + which is a different failure from a mismatch and is reported separately. + """ + claimed_arch = _wheel_architecture(claimed) + supported_arch = _wheel_architecture(supported) + if claimed_arch is None or supported_arch is None: + return None + return claimed_arch == supported_arch + + +def test_wheel_platform_tag() -> None: + """The wheel's declared platform tag must name the architecture it was built for. + + Only the architecture. auditwheel cannot certify a glibc baseline for this wheel: + it depends on torch without vendoring torch's libraries, so the contents reference + libtorch.so from outside any manylinux policy and auditwheel reports a plain + linux_. That is the expected answer for a torch-dependent wheel rather than + a defect, and the manylinux tag on the file comes from the build image. Asserting + the baseline here failed every correct wheel. + + The architecture is still worth checking, because a wheel labelled with the wrong + one installs on machines it cannot run on at all, and that is a mistake this can + actually catch. + """ + if importlib.util.find_spec("auditwheel") is None: + # Installed here rather than skipped, because auditwheel is not in any CI + # image and a skip is indistinguishable from a pass in the summary. This + # check is the only thing that compares the wheel's declared tag against + # what its libraries actually need, and this change adds five libraries + # under that tag. + print("- auditwheel not present, installing it so this check can run") + installed = subprocess.run( + [sys.executable, "-m", "pip", "install", "--quiet", "auditwheel"], + capture_output=True, + text=True, + check=False, + ) + if installed.returncode != 0 or importlib.util.find_spec("auditwheel") is None: + raise AssertionError( + "auditwheel is required to check the wheel's platform tag and could not " + "be installed. Skipping instead would report a pass, and this is the only " + "check that compares the declared tag against what the shipped libraries " + f"actually need: {installed.stderr.strip()[-200:]}" + ) + importlib.invalidate_caches() + + wheels = _find_wheel_files() + if not wheels: + print("- no wheel file to inspect, skipping the platform tag check") + return + + result = subprocess.run( + [sys.executable, "-m", "auditwheel", "show", str(wheels[-1])], + capture_output=True, + text=True, + check=False, + ) + # auditwheel wraps its verdict across lines, so compare on collapsed + # whitespace rather than the literal output. + combined = " ".join((result.stdout + result.stderr).split()) + match = re.search( + r'consistent with the following platform tag: "([^"]+)"', combined + ) + assert match, ( + "auditwheel reported no platform tag for the wheel, so its contents could " + f"not be checked against what it claims: {combined[-400:]}" + ) + claimed = wheels[-1].name.split("-")[-1].removesuffix(".whl") + supported = match.group(1) + + claimed_arch = _wheel_architecture(claimed) + supported_arch = _wheel_architecture(supported) + matches = _tag_architectures_match(claimed, supported) + assert matches is not None, ( + f"could not read an architecture from the declared tag {claimed} or from what " + f"auditwheel reported, {supported}" + ) + assert matches, ( + f"the wheel claims architecture {claimed_arch} but its contents are built for " + f"{supported_arch}, so it would install where it cannot run" + ) + print(f"✓ the wheel is tagged for the architecture it contains ({claimed_arch})") + + +def test_no_absolute_runtime_paths() -> None: + """No shipped library may search a directory a user does not have. + + Packaging copies libraries out of the build tree rather than installing them, so + every directory the linker recorded ships as-is. Two kinds are rejected on every + shipped library, not only the lib/ payload: + + - a directory inside the build, which names the machine that produced the wheel + - an empty entry, which the loader reads as the process working directory + + Torch's own directory is accepted. The extensions link torch and resolve it + through the directory the linker recorded, so that entry is load-bearing rather + than leftover. Narrowing this check to lib/ once hid seven extensions carrying + build-tree paths, so the exclusion is by what an entry POINTS AT, never by which + file carries it. + + The check reads the shipped file directly, with nothing stripped, which is what + a user actually receives. + """ + # Fatal, not a skip. Packaging strips these paths best-effort, because it cannot + # guarantee patchelf on PATH, so this is the only place the guarantee can be + # enforced. If both went quiet on the same missing tool, a wheel carrying the + # build machine's directories would ship looking correct. + if _tool("patchelf") is None: + print("- patchelf not present, installing it so this check can run") + subprocess.run( + [sys.executable, "-m", "pip", "install", "--quiet", "patchelf"], + capture_output=True, + text=True, + check=False, + ) + patchelf = _tool("patchelf") + assert patchelf is not None, ( + "patchelf is required to check the shipped runtime paths and could not be " + "installed. Packaging uses it to strip build-tree directories, and without it " + "here neither side would notice that they were left in place." + ) + + package_dir = _installed_package_dir() + + # Directories that only exist inside a build of this project. Compared as whole + # path components, the same way packaging decides what to strip, so the two do + # not describe the build tree differently. + def names_a_build_directory(entry: str) -> bool: + return any( + part in ("pip-out", "cmake-out") or part.startswith("lib.") + for part in entry.split("/") + ) + + # This project's libraries must not name an absolute directory the wheel has a relative route to. The + # one that shipped was a CUDA toolkit prefix recorded on the build machine: it sat ahead of the relative + # hop, so a user with a toolkit at the same prefix resolved the CUDA runtime from there instead of from + # the declared dependency, and the builder always has one, so nothing exercised the hop. + # + # Stated as a property rather than a list of known-bad directories, because a list only catches what + # someone already thought of and that prefix was not on one. + # + # PyTorch's own directory is allowed: the wheel neither declares nor bundles PyTorch, so an absolute + # path is the only way to reach it. The maths library directories are allowed too, because they come + # from PyTorch's build flags and reach everything that links PyTorch, including this project's own + # extensions, naming a location on whichever machine built PyTorch that nothing here can change. + allowed_absolute = ("/torch/lib", "/lib/intel64", "/lib/win-x64") + # PyTorch's own libraries are vendored into the wheel and also record a CUDA toolkit directory. That is + # the one path this check exists to reject on our libraries, so it is allowed only on theirs. + vendored_prefixes = ( + "libtorch", + "libc10", + "libshm", + "libcaffe2", + "libgomp", + "libiomp", + ) + + offenders = {} + checked = 0 + for library in sorted(package_dir.rglob("*.so*")): + if not library.is_file() or library.is_symlink(): + continue + result = subprocess.run( + [patchelf, "--print-rpath", str(library)], + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + continue + # An absent RPATH and one containing a single empty entry both print as an + # empty string, so treat empty output as "no runtime path" rather than as an + # empty entry. A library with nothing to search is fine; the defect is + # searching somewhere unusable. + raw = result.stdout.strip() + if not raw: + continue + checked += 1 + bad = [] + for entry in raw.split(":"): + if not entry: + bad.append("") + elif ( + entry.startswith("/") + and not any(allowed in entry for allowed in allowed_absolute) + and not library.name.startswith(vendored_prefixes) + ): + # Named separately so the message says which kind it is: a build directory and a + # toolkit prefix are the same defect with different causes. + kind = ( + "inside a build of this project" + if names_a_build_directory(entry) + else "an absolute directory the wheel has a relative route to" + ) + bad.append(f"{entry} ({kind})") + if bad: + offenders[str(library.relative_to(package_dir))] = bad + + assert not offenders, ( + "shipped libraries search directories a user does not have, either inside " + "the build tree that produced the wheel or, for an empty entry, the process " + f"working directory: {offenders}" + ) + assert checked, ( + f"no shipped library under {package_dir} carries a runtime search path, so this check examined " + "nothing and would pass on a wheel that shipped no libraries at all" + ) + print( + f"✓ none of the {checked} shipped libraries searches a build-tree or empty " + "runtime path" + ) + + +def test_extension_contains_no_component() -> None: + """The Python extension must link the components, not contain them. + + This is the property the change exists to create, and no count of definers + proves it: the monolithic layout has exactly one definer of every symbol too, + inside the extension. The direct statement is that the extension defines none of + what the shipped libraries own, and records a dependency on each instead. + """ + assert _tool("nm") is not None, "nm is required to inspect the wheel" + if _tool("readelf") is None: + print("- readelf unavailable, skipping the extension composition check") + return + + package_dir = _installed_package_dir() + extensions = sorted( + (package_dir / "extension" / "pybindings").glob("_portable_lib.*.so") + ) + assert len(extensions) == 1, f"expected one _portable_lib, found {extensions}" + extension = extensions[0] + + lib_dir = package_dir / "lib" + if not lib_dir.is_dir(): + print("- this wheel ships no lib directory, nothing to check") + return + + # Every symbol group a shipped library owns. The extension holding any of these + # means it still carries its own copy of that component. + # Derived from the ownership table rather than restated. A hand-written copy drifts: this once + # listed five symbol groups while the table covered eleven, so the components added later, including + # both CUDA ones, were never checked here. + # + # The bundled third-party groups are left out on purpose. That code is also linked by torch, and the + # extension links torch, so seeing those symbols there says nothing about this split. + # Only components whose owning library is actually in this wheel. One of them is optional, so a build + # with it turned off ships no owner, and asserting the extension does not define its symbols would + # reject a configuration the table itself marks as supported. + shipped = { + path.name for path in _shipped_runtime_libraries(_installed_package_dir()) + } + owned = tuple( + symbol + for _, symbols, owner, required in _OWNED_COMPONENTS + if symbols not in (_BUNDLED_THREADPOOL_SYMBOLS, _BUNDLED_XNNPACK_SYMBOLS) + and (required or any(name.startswith(owner) for name in shipped)) + for symbol in symbols + ) + contained = [symbol for symbol in owned if _defines_symbol(extension, symbol)] + assert not contained, ( + f"{extension.name} defines {contained}, which the shipped libraries own. The " + "extension is supposed to link them rather than contain them, so this is the " + "monolithic layout the split removes." + ) + + # And it has to actually depend on each shipped library. Defining nothing while + # depending on nothing would be an extension that cannot work at all. + needed = { + line.split("[", 1)[1].rstrip("]").strip() + for line in subprocess.run( + [_tool("readelf"), "-d", str(extension)], + capture_output=True, + text=True, + check=True, + ).stdout.splitlines() + if "NEEDED" in line + } + shipped = {path.name for path in _shipped_runtime_libraries(package_dir)} + assert shipped, ( + f"the wheel installed no runtime libraries under {package_dir / 'lib'}, so this check would " + "compare the extension against nothing and pass" + ) + unused = sorted(shipped - needed) + assert not unused, ( + f"the wheel ships {unused} but {extension.name} does not depend on them, so " + "either they are dead weight or a retention option did not hold" + ) + + # Positive proof that the extension resolves these from elsewhere, rather than + # only the absence of a visible definition. A hidden or local copy would not + # appear in the dynamic symbol table at all, so "defines nothing" on its own is + # satisfiable by an extension that still carries its own private runtime. An + # UNDEFINED reference cannot be faked that way: it says the definition is not + # here and has to come from a dependency. + undefined = subprocess.run( + [_tool("nm"), "-DC", "--undefined-only", str(extension)], + capture_output=True, + text=True, + check=False, + ).stdout + defined = subprocess.run( + [_tool("nm"), *_nm_defined_args(), str(extension)], + capture_output=True, + text=True, + check=False, + ).stdout + candidates = (*_REGISTRY_SYMBOLS, *_THREADPOOL_SYMBOLS) + imported = [symbol for symbol in candidates if symbol in undefined] + # A symbol the extension defines itself is the hidden copy this check exists to catch. A + # symbol it neither imports nor defines is simply unused, which happens for the backend + # registry when the wheel is built with the optional delegates off. + carried = [ + symbol for symbol in candidates if symbol not in undefined and symbol in defined + ] + assert not carried, ( + f"{extension.name} defines {carried} itself rather than importing it, so it carries a " + "private copy of a component the wheel also ships as a library" + ) + print( + f"✓ {extension.name} ({extension.stat().st_size // 1024} KiB) contains no " + f"component, imports the runtime symbols it uses, and depends on all " + f"{len(shipped)} shipped libraries" + ) + + +def test_shipped_library_names_are_expected() -> None: + """Every library in lib/ must be one this build could have produced. + + Packaging copies binaries out of a staging directory rather than running an + install step, so anything left there from an earlier build ships too. That + really happened: a wheel picked up three libraries from a different revision + and still passed every symbol check, because those checks only ask how many + definers a symbol has, never whether a file belongs in the wheel at all. + + Two properties catch it. A library's recorded soname matches its file name, or a + consumer records a dependency the wheel does not contain. And its name is one + packaging knows how to produce, which is what a leftover from an older layout + fails. A wheel ships unversioned names on purpose, so the name itself carries no + version to check. + """ + package_dir = _installed_package_dir() + lib_dir = package_dir / "lib" + if not lib_dir.is_dir(): + print("- this wheel ships no lib directory, nothing to check") + return + + # Regular files only. A symlink here would be a deliberate alias rather than the + # stale-artifact case this check is about, and a leftover from an earlier build is + # a real file, so it is still caught. + shipped = sorted( + p for p in lib_dir.glob("*.so*") if p.is_file() and not p.is_symlink() + ) + assert shipped, f"the wheel ships a lib directory with no libraries: {lib_dir}" + + # The names packaging can put here. Listed rather than derived because setup.py + # names each one literally, and a file with any other name did not come from + # this build. Which of them are present depends on the build options, so + # absence is fine and an unknown name is not. + # + # Matched in full rather than by taking the part before the first ".so", because + # that prefix is satisfied by a name like libexecutorch.so.old.so.1, which is + # exactly the shape a leftover file takes. + known = ( + "libexecutorch", + "libexecutorch_kernels_optimized", + "libexecutorch_backend_xnnpack", + "libexecutorch_threadpool", + "libexecutorch_etdump", + ) + # A plain .so, because the wheel build does not version these. A trailing + # .so. would also be a name packaging did not produce here. + permitted = re.compile(rf"(?:{'|'.join(known)})\.so") + unknown = sorted(p.name for p in shipped if not permitted.fullmatch(p.name)) + assert not unknown, ( + f"the wheel ships {unknown} under lib/, which packaging does not produce. " + "A file packaging did not put there came from a stale staging directory, " + "and it ships while looking correct to every other check." + ) + + if _tool("readelf") is None: + print(f"✓ {len(shipped)} shipped libraries have expected names") + return + + # The recorded soname has to match the file name, or a consumer records a + # dependency on a name the wheel does not contain. + mismatched = {} + for library in shipped: + dynamic = subprocess.run( + [_tool("readelf"), "-d", str(library)], + capture_output=True, + text=True, + check=False, + ).stdout + soname = next( + ( + line.split("[", 1)[1].rstrip("]").strip() + for line in dynamic.splitlines() + if "SONAME" in line + ), + None, + ) + if soname != library.name: + mismatched[library.name] = soname + assert not mismatched, ( + "shipped libraries record a soname that is not their file name, so a " + f"consumer would look for a file the wheel does not ship: {mismatched}" + ) + print(f"✓ {len(shipped)} shipped libraries have expected names and sonames") + + +_PARITY_MODEL = ''' +import json +import sys + +import torch +from executorch.exir import to_edge_transform_and_lower +from executorch.extension.pybindings.portable_lib import ( + _load_for_executorch_from_buffer, +) + + +class Net(torch.nn.Module): + """Several operator kinds rather than one, so the run exercises the merged CPU + kernels rather than a single add.""" + + def __init__(self): + super().__init__() + self.linear = torch.nn.Linear(8, 16) + self.conv = torch.nn.Conv2d(1, 4, 3, padding=1) + + def forward(self, x, image): + a = torch.relu(self.linear(x)) + b = self.conv(image).flatten(1) + return a.sum(dim=1, keepdim=True) + b.mean(dim=1, keepdim=True) + + +delegate = sys.argv[1] == "delegate" +torch.manual_seed(0) +model = Net().eval() +example = (torch.randn(2, 8), torch.randn(2, 1, 6, 6)) +with torch.no_grad(): + expected = model(*example) + +partitioners = [] +if delegate: + from executorch.backends.xnnpack.partition.xnnpack_partitioner import ( + XnnpackPartitioner, + ) + + partitioners = [XnnpackPartitioner()] + +program = to_edge_transform_and_lower( + torch.export.export(model, example), partitioner=partitioners +).to_executorch() +buffer = program.buffer + +actual = _load_for_executorch_from_buffer(buffer).forward(list(example))[0] +# Compared rather than merely run. The point of the split is that behaviour does +# not change, and only a numeric comparison shows that; a model that returns +# wrong values without erroring passes everything else. +torch.testing.assert_close(actual, expected, rtol=1e-4, atol=1e-4) + +print(json.dumps({"delegated": delegate, "has_xnnpack": b"XnnpackBackend" in bytes(buffer)})) +''' + + +def test_model_matches_eager_pytorch(work_dir: Path) -> None: + """A model exported and run through the bindings must match eager PyTorch. + + Twice: once plain, so the CPU kernels resolve from the shared library, and once + delegated to XNNPACK, so the delegate does. The delegated program is also + checked for the delegate's own identity, because a partitioner that claimed + nothing would silently fall back to the CPU kernels and still match. + + Separate processes, because a fault in one export leaves state that makes the + next look broken when it is not. + """ + if importlib.util.find_spec("torch") is None: + print("- torch is not installed, skipping the eager comparison") + return + + work_dir.mkdir(parents=True, exist_ok=True) + script = work_dir / "parity.py" + script.write_text(_PARITY_MODEL) + + # Both halves run. The delegate is a required component in the ownership table + # above, so a wheel reaching here without it has already failed that check, and + # tolerating its absence here would only hide a second symptom of the same fault. + for mode in ["plain", "delegate"]: + result = subprocess.run( + [sys.executable, str(script), mode], + capture_output=True, + text=True, + check=False, + cwd=str(work_dir), + ) + assert result.returncode == 0, ( + f"the {mode} model does not export, run, and match eager PyTorch: " + f"{(result.stderr or result.stdout).strip()[-1500:]}" + ) + report = json.loads(result.stdout.strip().splitlines()[-1]) + if mode == "delegate": + assert report["has_xnnpack"], ( + "the delegated export produced a program with no XNNPACK partition, " + "so the delegate was never exercised and the comparison only proves " + "the CPU kernels work" + ) + print(f"✓ the {mode} model matches eager PyTorch") + + +def test_declared_dependencies_match_the_wheel_tag() -> None: + """A CPU wheel must not declare the CUDA runtime, and a CUDA wheel must declare it. + + The tag is what a user resolves against, so a mismatch is a promise the wheel cannot keep in + either direction: a CPU wheel that pulls the CUDA packages costs a user hundreds of megabytes it + never loads, and a CUDA wheel that declares nothing leaves the runtime unresolvable. + + This is metadata only, so no library check can see it. A CPU wheel that wrongly declared the CUDA + runtime passed every other check in this file. + """ + requirements = importlib.metadata.requires("executorch") or [] + cuda = sorted(r.split()[0] for r in requirements if r.lower().startswith("nvidia")) + + # The local version segment of the installed version states what the wheel was built for. + version = importlib.metadata.version("executorch") + local = version.partition("+")[2] + is_cuda_wheel = local.startswith("cu") + + if is_cuda_wheel: + assert cuda, ( + f"version {version} says this is a CUDA wheel, but it declares no CUDA runtime " + "packages, so nothing resolves the runtime it links" + ) + print(f"✓ this CUDA wheel declares the runtime ({len(cuda)} packages)") + else: + assert not cuda, ( + f"version {version} is not a CUDA wheel, yet it declares {cuda}. A user installing it " + "would download the CUDA runtime this wheel never loads." + ) + print("✓ this non-CUDA wheel declares no CUDA runtime") + + +def run_tests(work_dir: Path) -> None: + # Ordered by what a failure tells you, because these run in sequence and the + # first failure stops the rest. The checks that prove the split behaves + # correctly come first; packaging metadata comes last, so a weak check cannot + # hide a strong one. + test_each_component_has_one_owner() + test_python_extensions_import() + test_declared_dependencies_match_the_wheel_tag() + test_extension_contains_no_component() + test_shipped_library_names_are_expected() + test_shipped_libraries_load() + test_shipped_libraries_resolve_without_build_tree() + test_custom_op_compiles(work_dir) + test_no_absolute_runtime_paths() + test_model_matches_eager_pytorch(work_dir) + # Last: a wrong answer here says the wheel is labelled wrong, not that the + # split is broken. + test_wheel_platform_tag() diff --git a/.github/workflows/build-wheels-aarch64-linux.yml b/.github/workflows/build-wheels-aarch64-linux.yml index b0b9a9c0fee..8adf4268228 100644 --- a/.github/workflows/build-wheels-aarch64-linux.yml +++ b/.github/workflows/build-wheels-aarch64-linux.yml @@ -6,9 +6,11 @@ on: paths: - .ci/**/* - .github/workflows/build-wheels-aarch64-linux.yml + - '**/CMakeLists.txt' - examples/**/* - pyproject.toml - setup.py + - tools/cmake/**/* push: branches: - nightly diff --git a/.github/workflows/build-wheels-linux.yml b/.github/workflows/build-wheels-linux.yml index 1a89079e428..7428b68a773 100644 --- a/.github/workflows/build-wheels-linux.yml +++ b/.github/workflows/build-wheels-linux.yml @@ -6,9 +6,11 @@ on: paths: - .ci/**/* - .github/workflows/build-wheels-linux.yml + - '**/CMakeLists.txt' - examples/**/* - pyproject.toml - setup.py + - tools/cmake/**/* push: branches: - nightly diff --git a/.github/workflows/build-wheels-macos.yml b/.github/workflows/build-wheels-macos.yml index 3fddb8e6d26..6ace109edf7 100644 --- a/.github/workflows/build-wheels-macos.yml +++ b/.github/workflows/build-wheels-macos.yml @@ -6,9 +6,11 @@ on: paths: - .ci/**/* - .github/workflows/build-wheels-macos.yml + - '**/CMakeLists.txt' - examples/**/* - pyproject.toml - setup.py + - tools/cmake/**/* push: branches: - nightly diff --git a/.github/workflows/build-wheels-windows.yml b/.github/workflows/build-wheels-windows.yml index 9b1f8663bd2..60c6520b944 100644 --- a/.github/workflows/build-wheels-windows.yml +++ b/.github/workflows/build-wheels-windows.yml @@ -5,9 +5,11 @@ on: paths: - .ci/**/* - .github/workflows/build-wheels-windows.yml + - '**/CMakeLists.txt' - examples/**/* - pyproject.toml - setup.py + - tools/cmake/**/* push: branches: - nightly diff --git a/CMakeLists.txt b/CMakeLists.txt index ff3b9e86f7e..5511fab231e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -193,6 +193,19 @@ if(DEFINED EXECUTORCH_BAREMETAL_SKIP_INSTALL endif() if(EXECUTORCH_BUILD_SHARED) + # Linux only, and said here rather than left to fail somewhere downstream. The + # shared build names libraries with an ELF soname, records $ORIGIN runtime + # paths, and uses GNU linker options to keep a registration-only library on a + # link line. None of that applies on Apple, which is served by the Swift + # package distribution, or on Windows, where the runtime carries no export + # annotations for a DLL. Enabling it elsewhere failed much later and less + # clearly, when packaging looked for a .so the build never emitted. + if(NOT CMAKE_SYSTEM_NAME STREQUAL "Linux") + message( + FATAL_ERROR "EXECUTORCH_BUILD_SHARED is supported on Linux only, not " + "${CMAKE_SYSTEM_NAME}." + ) + endif() set(CMAKE_POSITION_INDEPENDENT_CODE ON) endif() @@ -284,18 +297,54 @@ add_subdirectory(third-party) # state crash at import time. This function replaces pybind11::embed with # pybind11::module (which links Python::Module instead of Python::Python — # headers and ABI only, no libpython) and adds -undefined dynamic_lookup for -# symbol resolution. No-op on non-Apple platforms. +# symbol resolution. +# +# Linux needs the same treatment for a different reason. An extension is loaded +# by an interpreter that already has the Python runtime in the process, so it +# must not carry its own dependency on libpython: the interpreter's library +# directory is not on any search path a wheel can predict, and an interpreter +# built with a shared libpython leaves the extension unable to resolve it. The +# undefined symbols are supplied by the loading process, which is how a Python +# extension normally works. No-op on Windows, where extensions link the import +# library by design. function(strip_python_lib target) - if(NOT APPLE) + if(MSVC) return() endif() + # The module helper links the embedding form of Python for anything that is + # not a MODULE library, and that form brings a hard dependency on the + # interpreter's shared library plus an absolute path to wherever it was found + # on the build machine. Neither survives being shipped: the loader cannot + # satisfy the dependency from an installed wheel, and the absolute path names + # the build machine. + # + # These targets cannot simply become MODULE libraries, because other targets + # link them and CMake refuses to link a MODULE. So keep the type and replace + # the Python library instead. An extension does not need it: the interpreter + # already provides those symbols to anything it loads. + # + # The interfaces arrive as PRIVATE links, so they appear in LINK_LIBRARIES + # wrapped rather than as bare target names. Rewrite the property from the + # filtered list instead of relying on a name match against the wrapped form. get_target_property(_libs ${target} LINK_LIBRARIES) if(_libs) - list(REMOVE_ITEM _libs Python::Python pybind11::embed) - list(APPEND _libs pybind11::module) - set_target_properties(${target} PROPERTIES LINK_LIBRARIES "${_libs}") + set(_kept "") + foreach(_lib IN LISTS _libs) + # Match the embedding interfaces however they are spelled, including + # inside a wrapper. + if(NOT _lib MATCHES "(pybind11::embed|Python3?::Python)") + list(APPEND _kept "${_lib}") + endif() + endforeach() + list(APPEND _kept pybind11::module) + set_target_properties(${target} PROPERTIES LINK_LIBRARIES "${_kept}") + endif() + if(APPLE) + # Apple's linker needs telling that the undefined symbols resolve at load + # time. The Linux loader already permits that for a shared object and does + # not accept the flag. + target_link_options(${target} PRIVATE "LINKER:-undefined,dynamic_lookup") endif() - target_link_options(${target} PRIVATE "LINKER:-undefined,dynamic_lookup") endfunction() # Size-optimized builds disable exceptions, RTTI, and unwind tables. @@ -932,6 +981,55 @@ if(EXECUTORCH_BUILD_PTHREADPOOL AND EXECUTORCH_BUILD_CPUINFO) add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/extension/threadpool) endif() +# Consolidated shared library: bundles executorch_core plus commonly used +# extensions into a single libexecutorch.so. Defined before the pybind and +# kernel targets below so they can link this one runtime instead of embedding a +# private copy of the core, which would give the process a second backend +# registry. +if(EXECUTORCH_BUILD_SHARED) + executorch_add_shared_library(executorch_shared) + set_target_properties( + executorch_shared + PROPERTIES OUTPUT_NAME executorch + ARCHIVE_OUTPUT_NAME executorch_shared + EXPORT_NAME executorch-shared + ) + # Ships in the wheel's lib/ directory beside the libraries that link it. + executorch_target_shipped_runtime_path(executorch_shared) + target_include_directories( + executorch_shared PUBLIC ${_common_include_directories} + ) + target_compile_definitions( + executorch_shared PUBLIC C10_USING_CUSTOM_GENERATED_MACROS + ) + # Link executorch without WHOLE_ARCHIVE because its INTERFACE link options + # (from executorch_target_link_options_shared_lib) already force + # whole-archive. Everything else is pulled in through link options rather than + # the WHOLE_ARCHIVE link feature, because these archives also reference each + # other plainly and CMake before 3.29 refuses to mix a feature with a plain + # reference to the same item. + target_link_libraries(executorch_shared PRIVATE executorch) + set(_executorch_shared_whole_archive executorch_core) + foreach(_ext_target + extension_data_loader extension_flat_tensor extension_named_data_map + extension_module_static extension_tensor + ) + if(TARGET ${_ext_target}) + list(APPEND _executorch_shared_whole_archive ${_ext_target}) + endif() + endforeach() + foreach(_whole_target ${_executorch_shared_whole_archive}) + executorch_target_whole_archive(executorch_shared ${_whole_target}) + endforeach() + configure_file( + tools/cmake/executorch.pc.in ${CMAKE_CURRENT_BINARY_DIR}/executorch.pc + @ONLY + ) + install(FILES ${CMAKE_CURRENT_BINARY_DIR}/executorch.pc + DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig + ) +endif() + if(EXECUTORCH_BUILD_KERNELS_TORCHAO) if(NOT TARGET cpuinfo) message( @@ -992,16 +1090,21 @@ if(EXECUTORCH_BUILD_KERNELS_TORCHAO) endif() +# The shared build ships the profiler as one of its libraries, and the Python +# extension records a hard dependency on it, so the target has to exist whenever +# either of those is being built rather than only when devtools is asked for. +if((EXECUTORCH_BUILD_PYBIND OR EXECUTORCH_BUILD_SHARED) + AND NOT EXECUTORCH_BUILD_DEVTOOLS +) + add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/devtools) +endif() + if(EXECUTORCH_BUILD_PYBIND) if(NOT EXECUTORCH_BUILD_EXTENSION_DATA_LOADER) add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/extension/data_loader) endif() - if(NOT EXECUTORCH_BUILD_DEVTOOLS) - add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/devtools) - endif() - # Add codegen tools subdirectory for selective_build pybind module add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/codegen/tools) @@ -1016,10 +1119,19 @@ if(EXECUTORCH_BUILD_PYBIND) # Ensure bundled_module waits for bundled_program's generated headers add_dependencies(bundled_module bundled_program) - target_link_libraries(bundled_module PRIVATE extension_data_loader) - target_link_libraries( - bundled_module PUBLIC extension_module_static bundled_program - ) + # extension_module_static and the data loader are bundled into + # libexecutorch.so, so link that instead of pulling private static copies in + # through this target's PUBLIC interface. + if(EXECUTORCH_BUILD_SHARED) + target_link_libraries( + bundled_module PUBLIC executorch_shared bundled_program + ) + else() + target_link_libraries(bundled_module PRIVATE extension_data_loader) + target_link_libraries( + bundled_module PUBLIC extension_module_static bundled_program + ) + endif() target_include_directories( bundled_module PUBLIC ${_common_include_directories} @@ -1038,16 +1150,36 @@ if(EXECUTORCH_BUILD_PYBIND) TORCH_PYTHON_LIBRARY torch_python PATHS "${TORCH_INSTALL_PREFIX}/lib" ) - set(_dep_libs - ${TORCH_PYTHON_LIBRARY} - bundled_program - etdump - flatccrt - executorch - extension_data_loader - util - torch - ) + # When the consolidated shared runtime is built, the pybind extension links it + # instead of whole-archiving the static core, so Python and C++ consumers + # share one backend registry. `executorch` and the extensions bundled into + # libexecutorch.so must stay off this list: their INTERFACE link options force + # whole-archive, which would give this module a private second registry. + # executorch_shared is named here for its include directories and compile + # definitions; executorch_target_link_shared_runtime below is what fixes its + # position on the link line. + if(EXECUTORCH_BUILD_SHARED) + set(_dep_libs + ${TORCH_PYTHON_LIBRARY} + bundled_program + etdump + flatccrt + executorch_shared + util + torch + ) + else() + set(_dep_libs + ${TORCH_PYTHON_LIBRARY} + bundled_program + etdump + flatccrt + executorch + extension_data_loader + util + torch + ) + endif() # Build common AOTI functionality if needed by CUDA or Metal backends if(EXECUTORCH_BUILD_CUDA) @@ -1058,13 +1190,19 @@ if(EXECUTORCH_BUILD_PYBIND) list(APPEND _dep_libs aoti_common) endif() - # RPATH for _portable_lib.so + # RPATH for _portable_lib.so. It sits in + # /executorch/extension/pybindings, so torch is three levels up + # and the wheel's own lib/ directory is two. set(_portable_lib_rpath "$ORIGIN/../../../torch/lib") if(EXECUTORCH_BUILD_EXTENSION_MODULE) - # Always use static linking for pybindings to avoid runtime symbol - # resolution issues - list(APPEND _dep_libs extension_module_static) + # extension_module_static is already bundled into libexecutorch.so; linking + # it again here would whole-archive a second copy. + if(NOT EXECUTORCH_BUILD_SHARED) + # Always use static linking for pybindings to avoid runtime symbol + # resolution issues + list(APPEND _dep_libs extension_module_static) + endif() # Add bundled_module if available if(TARGET bundled_module) list(APPEND _dep_libs bundled_module) @@ -1115,9 +1253,15 @@ if(EXECUTORCH_BUILD_PYBIND) endif() if(EXECUTORCH_BUILD_XNNPACK) - # need to explicitly specify XNNPACK and xnnpack-microkernels-prod here - # otherwise uses XNNPACK and microkernel-prod symbols from libtorch_cpu - list(APPEND _dep_libs xnnpack_backend XNNPACK xnnpack-microkernels-prod) + if(EXECUTORCH_BUILD_SHARED) + # The delegate bundles XNNPACK and its microkernels, so naming them again + # here would ship a second copy. + list(APPEND _dep_libs xnnpack_backend) + else() + # need to explicitly specify XNNPACK and xnnpack-microkernels-prod here + # otherwise uses XNNPACK and microkernel-prod symbols from libtorch_cpu + list(APPEND _dep_libs xnnpack_backend XNNPACK xnnpack-microkernels-prod) + endif() endif() if(EXECUTORCH_BUILD_VULKAN) @@ -1150,7 +1294,11 @@ if(EXECUTORCH_BUILD_PYBIND) target_compile_definitions(util PUBLIC C10_USING_CUSTOM_GENERATED_MACROS) target_compile_options(util PUBLIC ${_pybind_compile_options}) - target_link_libraries(util PRIVATE torch c10 executorch extension_tensor) + if(EXECUTORCH_BUILD_SHARED) + target_link_libraries(util PRIVATE torch c10 executorch_shared) + else() + target_link_libraries(util PRIVATE torch c10 executorch extension_tensor) + endif() # pybind portable_lib pybind11_add_module(portable_lib SHARED extension/pybindings/pybindings.cpp) @@ -1167,6 +1315,25 @@ if(EXECUTORCH_BUILD_PYBIND) target_include_directories(portable_lib PRIVATE ${TORCH_INCLUDE_DIRS}) target_compile_options(portable_lib PUBLIC ${_pybind_compile_options}) target_link_libraries(portable_lib PRIVATE ${_dep_libs}) + executorch_target_link_shared_runtime(portable_lib) + # These libraries register their operators or their backend from a static + # initializer, so nothing here references a symbol from them and some linkers + # drop them from DT_NEEDED. That surfaces at runtime as a missing kernel or an + # unregistered backend rather than as a link error. Every shipped library, not + # only the two that register something. A library reached transitively is + # still subject to --as-needed and gets dropped, so the thread pool and the + # profiler need naming here too even though the extension does reference + # symbols from them. A distro toolchain that defaults to --as-needed would + # otherwise leave them out of DT_NEEDED entirely. + foreach(_retained_component optimized_native_cpu_ops_lib xnnpack_backend + extension_threadpool etdump + ) + if(TARGET ${_retained_component}) + executorch_target_retain_shared_library( + portable_lib ${_retained_component} + ) + endif() + endforeach() # Set RPATH to find PyTorch and backend libraries relative to the installation # location. This goes from executorch/extension/pybindings up to @@ -1184,6 +1351,9 @@ if(EXECUTORCH_BUILD_PYBIND) INSTALL_RPATH "${_portable_lib_rpath}" ) endif() + executorch_target_shared_runtime_path( + portable_lib "extension/pybindings" "executorch/extension/pybindings" + ) install( TARGETS portable_lib @@ -1199,7 +1369,18 @@ if(EXECUTORCH_BUILD_PYBIND) strip_python_lib(data_loader) target_include_directories(data_loader PRIVATE ${_common_include_directories}) target_compile_options(data_loader PUBLIC ${_pybind_compile_options}) - target_link_libraries(data_loader PRIVATE executorch) + # This module only exposes a pybind type and calls into no runtime symbols. + # The static target force-links every registration object, which would give + # this module its own operator registry alongside the one in the shared + # runtime, so resolve against the shared runtime instead when there is one. + if(TARGET executorch_shared) + target_link_libraries(data_loader PRIVATE executorch_shared) + executorch_target_shared_runtime_path( + data_loader "extension/pybindings" "executorch/extension/pybindings" + ) + else() + target_link_libraries(data_loader PRIVATE executorch) + endif() install(TARGETS data_loader LIBRARY DESTINATION executorch/extension/pybindings ) @@ -1246,49 +1427,6 @@ if(EXECUTORCH_BUILD_KERNELS_LLM) list(APPEND _executorch_kernels custom_ops_aot_lib) endif() -# Consolidated shared library: bundles executorch_core plus commonly used -# extensions into a single libexecutorch.so. -if(EXECUTORCH_BUILD_SHARED) - executorch_add_shared_library(executorch_shared) - set_target_properties( - executorch_shared - PROPERTIES OUTPUT_NAME executorch - ARCHIVE_OUTPUT_NAME executorch_shared - EXPORT_NAME executorch-shared - ) - target_include_directories( - executorch_shared PUBLIC ${_common_include_directories} - ) - target_compile_definitions( - executorch_shared PUBLIC C10_USING_CUSTOM_GENERATED_MACROS - ) - # Link executorch without WHOLE_ARCHIVE because its INTERFACE link options - # (from executorch_target_link_options_shared_lib) already force - # whole-archive. Link executorch_core explicitly since executorch only has a - # PRIVATE dep on it (symbols wouldn't propagate otherwise). - target_link_libraries( - executorch_shared PRIVATE executorch - $ - ) - foreach(_ext_target - extension_data_loader extension_flat_tensor extension_named_data_map - extension_module_static extension_tensor - ) - if(TARGET ${_ext_target}) - target_link_libraries( - executorch_shared PRIVATE $ - ) - endif() - endforeach() - configure_file( - tools/cmake/executorch.pc.in ${CMAKE_CURRENT_BINARY_DIR}/executorch.pc - @ONLY - ) - install(FILES ${CMAKE_CURRENT_BINARY_DIR}/executorch.pc - DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig - ) -endif() - if(EXECUTORCH_BUILD_KERNELS_QUANTIZED) add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/kernels/quantized) executorch_target_link_options_shared_lib(quantized_ops_lib) diff --git a/backends/qualcomm/CMakeLists.txt b/backends/qualcomm/CMakeLists.txt index 3f4aefa2b76..7f82eb06590 100644 --- a/backends/qualcomm/CMakeLists.txt +++ b/backends/qualcomm/CMakeLists.txt @@ -255,8 +255,20 @@ target_link_libraries( ) target_link_libraries( qnn_executorch_backend PRIVATE qnn_executorch_header qnn_schema qnn_manager - executorch_core qnn_backend_options + qnn_backend_options ) +# Resolve the runtime from the shared library when one is built, so this +# delegate does not carry its own copy of the backend registry. +if(EXECUTORCH_BUILD_SHARED) + target_link_libraries(qnn_executorch_backend PRIVATE executorch_shared) + executorch_target_link_shared_runtime(qnn_executorch_backend) + executorch_target_shared_runtime_path( + qnn_executorch_backend "backends/qualcomm" + "${CMAKE_INSTALL_LIBDIR}/executorch/backends/qualcomm" + ) +else() + target_link_libraries(qnn_executorch_backend PRIVATE executorch_core) +endif() if(${CMAKE_SYSTEM_PROCESSOR} MATCHES Hexagon) # Add macro here so we can dlopen the correct .so library. @@ -359,12 +371,27 @@ if(${CMAKE_SYSTEM_PROCESSOR} MATCHES "x86_64|AMD64") qnn_schema qnn_manager qnn_executorch_header - executorch - extension_tensor qnn_backend_options wrappers qnn_executorch_logging ) + # extension_tensor is bundled into the shared runtime, so naming it again here + # would give this module a second copy of what that library already provides. + if(NOT EXECUTORCH_BUILD_SHARED) + target_link_libraries(PyQnnManagerAdaptor PRIVATE extension_tensor) + endif() + # Same reasoning as the delegate above: take the runtime from the shared + # library when there is one, rather than embedding a second registry. + if(EXECUTORCH_BUILD_SHARED) + target_link_libraries(PyQnnManagerAdaptor PRIVATE executorch_shared) + executorch_target_link_shared_runtime(PyQnnManagerAdaptor) + executorch_target_shared_runtime_path( + PyQnnManagerAdaptor "backends/qualcomm/python" + "${CMAKE_INSTALL_LIBDIR}/executorch/backends/qualcomm/python" + ) + else() + target_link_libraries(PyQnnManagerAdaptor PRIVATE executorch) + endif() pybind11_extension(PyQnnManagerAdaptor) if(NOT MSVC AND NOT ${CMAKE_BUILD_TYPE} MATCHES RelWithDebInfo) diff --git a/backends/xnnpack/CMakeLists.txt b/backends/xnnpack/CMakeLists.txt index cd0d945a84f..abd4bd596fe 100644 --- a/backends/xnnpack/CMakeLists.txt +++ b/backends/xnnpack/CMakeLists.txt @@ -96,16 +96,48 @@ target_include_directories( $ ) -set(xnnpack_third_party pthreadpool extension_threadpool cpuinfo) +if(EXECUTORCH_BUILD_SHARED) + # extension_threadpool is a shared library here and already provides + # pthreadpool and cpuinfo. Naming the static archives as well would give this + # delegate its own second copy of both, so a process would end up with two + # thread pools rather than the one the shared library exists to provide. + set(xnnpack_third_party extension_threadpool) +else() + set(xnnpack_third_party pthreadpool extension_threadpool cpuinfo) +endif() include(cmake/Dependencies.cmake) list(TRANSFORM _xnnpack_backend__srcs PREPEND "${EXECUTORCH_ROOT}/") -add_library(xnnpack_backend ${_xnnpack_backend__srcs}) +# Build the delegate as a shared library for the wheel so a process has one copy +# of it, and keep it static everywhere else so no other build changes. +if(EXECUTORCH_BUILD_SHARED) + set(_xnnpack_backend_library_type SHARED) +else() + set(_xnnpack_backend_library_type STATIC) +endif() +add_library( + xnnpack_backend ${_xnnpack_backend_library_type} ${_xnnpack_backend__srcs} +) target_link_libraries( - xnnpack_backend PUBLIC ${xnnpack_third_party} executorch_core xnnpack_schema + xnnpack_backend PUBLIC ${xnnpack_third_party} xnnpack_schema extension_threadpool ) +if(EXECUTORCH_BUILD_SHARED) + set_target_properties( + xnnpack_backend PROPERTIES OUTPUT_NAME executorch_backend_xnnpack + ) + executorch_target_soname_policy(xnnpack_backend) + # XNNPACK and its microkernels are forced static, so bundle them inside this + # library instead of making every consumer supply them. + executorch_target_whole_archive(xnnpack_backend XNNPACK) + executorch_target_whole_archive(xnnpack_backend xnnpack-microkernels-prod) + target_link_libraries(xnnpack_backend PUBLIC executorch_shared) + # Ships beside the runtime in the wheel's lib/ directory. + executorch_target_shipped_runtime_path(xnnpack_backend) +else() + target_link_libraries(xnnpack_backend PUBLIC executorch_core) +endif() target_include_directories( xnnpack_backend PUBLIC ${_common_include_directories} ) diff --git a/codegen/tools/CMakeLists.txt b/codegen/tools/CMakeLists.txt index b829e83c340..143d2c18c6d 100644 --- a/codegen/tools/CMakeLists.txt +++ b/codegen/tools/CMakeLists.txt @@ -43,7 +43,20 @@ if(TARGET bundled_program) target_compile_definitions(selective_build PRIVATE -DET_BUNDLE_IO) target_link_libraries(selective_build PRIVATE bundled_program) endif() -target_link_libraries(selective_build PRIVATE executorch_core program_schema) +if(EXECUTORCH_BUILD_SHARED) + # This module calls into the runtime, so resolve those symbols from + # libexecutorch.so rather than baking in a second copy of the core. It lands + # in /executorch/codegen/tools, two levels below the wheel's + # lib/. + target_link_libraries( + selective_build PRIVATE executorch_shared program_schema + ) + executorch_target_shared_runtime_path( + selective_build "codegen/tools" "executorch/codegen/tools" + ) +else() + target_link_libraries(selective_build PRIVATE executorch_core program_schema) +endif() # Install the module install(TARGETS selective_build LIBRARY DESTINATION executorch/codegen/tools) diff --git a/configurations/CMakeLists.txt b/configurations/CMakeLists.txt index fb154ff88bc..2379630c00e 100644 --- a/configurations/CMakeLists.txt +++ b/configurations/CMakeLists.txt @@ -50,7 +50,15 @@ if(EXECUTORCH_BUILD_KERNELS_OPTIMIZED) else() set(_optimized_native_cpu_ops_lib_portable_kernels_lib portable_kernels) endif() + # Ship this as a shared library in the wheel so the kernels are registered + # once per process instead of once per component that links them. + if(EXECUTORCH_BUILD_SHARED) + set(_merged_cpu_ops_library_type SHARED) + else() + set(_merged_cpu_ops_library_type "") + endif() gen_operators_lib( + ${_merged_cpu_ops_library_type} LIB_NAME "optimized_native_cpu_ops_lib" KERNEL_LIBS @@ -65,4 +73,15 @@ if(EXECUTORCH_BUILD_KERNELS_OPTIMIZED) EXPORT ExecuTorchTargets DESTINATION ${CMAKE_INSTALL_LIBDIR} ) + if(EXECUTORCH_BUILD_SHARED) + # Named after what the library provides rather than after the code + # generation target that produces it, so the shipped file reads as + # libexecutorch_kernels_optimized.so. The target name stays as it is because + # a source build already refers to it. + set_target_properties( + optimized_native_cpu_ops_lib PROPERTIES OUTPUT_NAME + executorch_kernels_optimized + ) + executorch_target_soname_policy(optimized_native_cpu_ops_lib) + endif() endif() diff --git a/devtools/bundled_program/CMakeLists.txt b/devtools/bundled_program/CMakeLists.txt index 0c213d9a83c..375f7487f06 100644 --- a/devtools/bundled_program/CMakeLists.txt +++ b/devtools/bundled_program/CMakeLists.txt @@ -40,7 +40,14 @@ add_library( bundled_program ${_schema_outputs} ${CMAKE_CURRENT_SOURCE_DIR}/bundled_program.cpp ) -target_link_libraries(bundled_program PUBLIC executorch) +# The `executorch` target forces whole-archive of itself, which would duplicate +# the primitive operator registrations already inside libexecutorch.so and abort +# at load. Resolve them from the shared runtime instead when it is built. +if(EXECUTORCH_BUILD_SHARED) + target_link_libraries(bundled_program PUBLIC executorch_shared) +else() + target_link_libraries(bundled_program PUBLIC executorch) +endif() target_include_directories( bundled_program PUBLIC diff --git a/devtools/etdump/CMakeLists.txt b/devtools/etdump/CMakeLists.txt index 9ef3c8cd6f7..cf331e24416 100644 --- a/devtools/etdump/CMakeLists.txt +++ b/devtools/etdump/CMakeLists.txt @@ -39,8 +39,19 @@ add_custom_command( COMMENT "Generating etdump headers" ) +# The profiler is reachable from both the Python extension and a standalone C++ +# application, and each one linking it statically would keep its own tracing +# state. Build it shared for the wheel, where both are loaded into one process, +# and keep it static everywhere else so no other build changes. +if(EXECUTORCH_BUILD_SHARED) + set(_etdump_library_type SHARED) +else() + set(_etdump_library_type STATIC) +endif() + add_library( etdump + ${_etdump_library_type} ${_schema_outputs} ${CMAKE_CURRENT_SOURCE_DIR}/etdump_flatcc.cpp ${CMAKE_CURRENT_SOURCE_DIR}/emitter.cpp @@ -49,11 +60,27 @@ add_library( ${CMAKE_CURRENT_SOURCE_DIR}/data_sinks/file_data_sink.cpp ${CMAKE_CURRENT_SOURCE_DIR}/data_sinks/file_data_sink.h ) -target_link_libraries( - etdump - PUBLIC flatccrt - PRIVATE executorch -) +# As with bundled_program, avoid the whole-archive of the `executorch` target so +# the primitive operator registrations are not duplicated alongside the copy +# already inside libexecutorch.so. +if(EXECUTORCH_BUILD_SHARED) + # Private, not public: this library bundles the flatbuffer runtime rather than + # depending on it, and the wheel does not ship flatccrt, so exposing it would + # name something a consumer cannot link. Scoped to the shared build so a build + # that does not opt in keeps the parent's public dependency. + target_link_libraries(etdump PRIVATE flatccrt executorch_shared) +else() + target_link_libraries(etdump PUBLIC flatccrt) + target_link_libraries(etdump PRIVATE executorch) +endif() +if(EXECUTORCH_BUILD_SHARED) + set_target_properties(etdump PROPERTIES OUTPUT_NAME executorch_etdump) + executorch_target_soname_policy(etdump) + # Ships beside libexecutorch.so in the wheel's lib/ directory, and needs it, + # so it has to be able to find it from wherever the package is installed. + executorch_target_shipped_runtime_path(etdump) +endif() + target_include_directories( etdump PUBLIC ${DEVTOOLS_INCLUDE_DIR} diff --git a/extension/llm/custom_ops/CMakeLists.txt b/extension/llm/custom_ops/CMakeLists.txt index 8a43a5ddf5c..dc707c7e198 100644 --- a/extension/llm/custom_ops/CMakeLists.txt +++ b/extension/llm/custom_ops/CMakeLists.txt @@ -139,24 +139,50 @@ if(EXECUTORCH_BUILD_KERNELS_LLM_AOT) ${TORCH_INCLUDE_DIRS} ) # TODO: This only works if we install portable_lib.so to - # /executorch/extension/pybindings/. + # /executorch/extension/pybindings/. torch is three directories + # up from here, and this library links it directly, so the hop has to be + # recorded or the only route is the absolute path from the machine that built + # the wheel. if(APPLE) - set(RPATH "@loader_path/../../pybindings") + set(RPATH + "@loader_path/../../pybindings;@loader_path/../../../../torch/lib" + ) + else() + set(RPATH "$ORIGIN/../../pybindings:$ORIGIN/../../../../torch/lib") + endif() + if(EXECUTORCH_BUILD_SHARED) + # Also on the built artifact, not only on install. Packaging copies this + # library out of the build tree rather than running an install step, so + # without this the built file carries whatever the linker recorded. Scoped + # to the shared build so a build that does not opt in keeps the parent's + # install-only behaviour. + set_target_properties( + custom_ops_aot_lib PROPERTIES BUILD_RPATH "${RPATH}" INSTALL_RPATH + "${RPATH}" + ) else() - set(RPATH "$ORIGIN/../../pybindings") + set_target_properties( + custom_ops_aot_lib PROPERTIES INSTALL_RPATH "${RPATH}" + ) endif() - set_target_properties(custom_ops_aot_lib PROPERTIES INSTALL_RPATH ${RPATH}) + executorch_target_shared_runtime_path( + custom_ops_aot_lib "extension/llm/custom_ops" + "executorch/extension/llm/custom_ops" + ) if(TARGET portable_lib) # If we have portable_lib built, custom_ops_aot_lib gives the ability to use # the ops in PyTorch and ExecuTorch through pybind target_link_libraries(custom_ops_aot_lib PUBLIC portable_lib) - else() + elseif(NOT EXECUTORCH_BUILD_SHARED) # If no portable_lib, custom_ops_aot_lib still gives the ability to use the # ops in PyTorch target_link_libraries( custom_ops_aot_lib PUBLIC executorch_core kernels_util_all_deps ) + else() + target_link_libraries(custom_ops_aot_lib PUBLIC kernels_util_all_deps) endif() + executorch_target_link_shared_runtime(custom_ops_aot_lib) target_link_libraries( custom_ops_aot_lib PUBLIC cpublas torch extension_tensor diff --git a/extension/llm/runner/CMakeLists.txt b/extension/llm/runner/CMakeLists.txt index 5247a4ba0a6..20a6e934b35 100644 --- a/extension/llm/runner/CMakeLists.txt +++ b/extension/llm/runner/CMakeLists.txt @@ -123,6 +123,7 @@ if(EXECUTORCH_BUILD_PYBIND) _llm_runner PRIVATE extension_llm_runner tokenizers::tokenizers portable_lib ${TORCH_PYTHON_LIBRARY} ${TORCH_LIBRARIES} ) + executorch_target_link_shared_runtime(_llm_runner) set_target_properties( _llm_runner @@ -141,6 +142,9 @@ if(EXECUTORCH_BUILD_PYBIND) set_target_properties( _llm_runner PROPERTIES BUILD_RPATH "${RPATH}" INSTALL_RPATH "${RPATH}" ) + executorch_target_shared_runtime_path( + _llm_runner "extension/llm/runner" "executorch/extension/llm/runner" + ) # Add include directories target_include_directories( _llm_runner PRIVATE ${_common_include_directories} ${TORCH_INCLUDE_DIRS} diff --git a/extension/threadpool/CMakeLists.txt b/extension/threadpool/CMakeLists.txt index 3b9c7c66ddb..ed16cb169ac 100644 --- a/extension/threadpool/CMakeLists.txt +++ b/extension/threadpool/CMakeLists.txt @@ -30,13 +30,38 @@ else() set(_threadpool_size_flag "EXECUTORCH_THREADPOOL_USE_PERFORMANCE_CORES") endif() +# The thread pool is a process-wide singleton held in a function-local static, +# so every library that links it statically gets its own copy. Build it shared +# for the wheel, where several extensions are loaded into one interpreter, and +# keep it static everywhere else so no other build changes. +if(EXECUTORCH_BUILD_SHARED) + set(_threadpool_library_type SHARED) +else() + set(_threadpool_library_type STATIC) +endif() + add_library( - extension_threadpool threadpool.cpp threadpool_guard.cpp thread_parallel.cpp - cpuinfo_utils.cpp -) -target_link_libraries( - extension_threadpool PUBLIC executorch_core cpuinfo pthreadpool + extension_threadpool + ${_threadpool_library_type} threadpool.cpp threadpool_guard.cpp + thread_parallel.cpp cpuinfo_utils.cpp ) +if(EXECUTORCH_BUILD_SHARED) + set_target_properties( + extension_threadpool PROPERTIES OUTPUT_NAME executorch_threadpool + ) + executorch_target_soname_policy(extension_threadpool) + # Ships beside libexecutorch.so in the wheel's lib/ directory. + executorch_target_shipped_runtime_path(extension_threadpool) + # cpuinfo and pthreadpool are forced static, so bundle them inside this + # library instead of making every consumer supply them. + executorch_target_whole_archive(extension_threadpool cpuinfo) + executorch_target_whole_archive(extension_threadpool pthreadpool) + target_link_libraries(extension_threadpool PUBLIC executorch_shared) +else() + target_link_libraries( + extension_threadpool PUBLIC executorch_core cpuinfo pthreadpool + ) +endif() target_include_directories( extension_threadpool PUBLIC ${_common_include_directories} ) diff --git a/extension/training/CMakeLists.txt b/extension/training/CMakeLists.txt index e835ae0e0a3..04a880a4043 100644 --- a/extension/training/CMakeLists.txt +++ b/extension/training/CMakeLists.txt @@ -49,17 +49,31 @@ target_link_libraries( target_compile_options(train_xor PUBLIC ${_common_compile_options}) if(EXECUTORCH_BUILD_PYBIND) - # Pybind library. - set(_pybind_training_dep_libs ${TORCH_PYTHON_LIBRARY} etdump executorch util - torch extension_training - ) + # Pybind library. When the consolidated shared runtime is built, the runtime + # is resolved from it rather than from the whole-archive-forcing static + # `executorch` target, so this module shares the one backend registry. + if(EXECUTORCH_BUILD_SHARED) + set(_pybind_training_dep_libs ${TORCH_PYTHON_LIBRARY} etdump util torch + extension_training + ) + else() + set(_pybind_training_dep_libs ${TORCH_PYTHON_LIBRARY} etdump executorch + util torch extension_training + ) + endif() if(EXECUTORCH_BUILD_XNNPACK) - # need to explicitly specify XNNPACK and xnnpack-microkernels-prod here - # otherwise uses XNNPACK and microkernel-prod symbols from libtorch_cpu - list(APPEND _pybind_training_dep_libs xnnpack_backend XNNPACK - xnnpack-microkernels-prod - ) + if(EXECUTORCH_BUILD_SHARED) + # The delegate bundles XNNPACK and its microkernels, so naming them again + # here would ask for a second copy of what that library already provides. + list(APPEND _pybind_training_dep_libs xnnpack_backend) + else() + # need to explicitly specify XNNPACK and xnnpack-microkernels-prod here + # otherwise uses XNNPACK and microkernel-prod symbols from libtorch_cpu + list(APPEND _pybind_training_dep_libs xnnpack_backend XNNPACK + xnnpack-microkernels-prod + ) + endif() endif() pybind11_add_module( @@ -82,6 +96,31 @@ if(EXECUTORCH_BUILD_PYBIND) ) target_link_libraries(_training_lib PRIVATE ${_pybind_training_dep_libs}) + if(EXECUTORCH_BUILD_SHARED + AND EXECUTORCH_BUILD_XNNPACK + AND TARGET xnnpack_backend + ) + # The delegate registers itself from a static initializer, so nothing in + # this extension references a symbol from it and a normal link can drop it. + # Keeping it named on the link line is what makes an XNNPACK-delegated + # program usable from here. + executorch_target_retain_shared_library(_training_lib xnnpack_backend) + endif() + executorch_target_link_shared_runtime(_training_lib) + + if(EXECUTORCH_BUILD_SHARED AND NOT APPLE) + # This module links Torch directly, and the only other entry reaching it is + # the absolute build directory CMake adds, which does not exist anywhere + # else, so the Torch path is recorded here rather than left implicit. + set_target_properties( + _training_lib PROPERTIES INSTALL_RPATH "$ORIGIN/../../../../torch/lib" + ) + executorch_target_shared_runtime_path( + _training_lib "extension/training/pybindings" + "executorch/extension/training/pybindings" + ) + endif() + install(TARGETS _training_lib LIBRARY DESTINATION executorch/extension/training/pybindings ) diff --git a/install_utils.py b/install_utils.py index ee4a91aa661..5bbd3eeac73 100644 --- a/install_utils.py +++ b/install_utils.py @@ -142,22 +142,49 @@ def _get_cuda_version(): def _extract_cmake_define(args: List[str], name: str) -> Optional[str]: - prefix = f"-D{name}=" - for arg in args: - if arg.startswith(prefix): - return arg[len(prefix) :] - return None + """The value CMake would use for -D, which is the last one given. + + Repeating a definition is how a caller overrides an earlier one, and CMake keeps the last, so returning + the first would let packaging read one value while the build used another. + + All three spellings CMake accepts are matched, because it treats them identically: -D=, + -D:=, and -D followed by = as a separate argument. Matching only the + first meant a caller who switched an option off in either of the other two forms was read as leaving it + on, so a CPU row could ship a wheel carrying CUDA. + """ + # A bare -D takes its definition from the next argument, so both spellings collapse to one form. + definitions = [] + remaining = iter(args) + for arg in remaining: + if arg == "-D": + definitions.append(next(remaining, "")) + elif arg.startswith("-D"): + definitions.append(arg[2:]) + + # The name may carry a CMake type, as in EXECUTORCH_BUILD_CUDA:BOOL. + pattern = re.compile(rf"{re.escape(name)}(?::\w+)?=(.*)", re.DOTALL) + value = None + for definition in definitions: + match = pattern.fullmatch(definition) + if match: + value = match.group(1) + return value def _normalize_cmake_bool(value: Optional[str], default: bool = False) -> bool: if value is None: return default normalized = value.strip().upper() - if normalized in {"ON", "1", "TRUE", "YES"}: + # Deliberately stricter than CMake. CMake decides false by exclusion, so anything that is not one + # of its false constants is true, including values like "2.0" and "enabled". Here an unrecognised + # spelling reads as off, because this decides whether a component's libraries are packaged and + # shipping a component whose libraries were never built is worse than shipping one fewer. + if normalized in {"ON", "TRUE", "YES", "Y"}: return True - if normalized in {"OFF", "0", "FALSE", "NO"}: + try: + return int(normalized) != 0 + except ValueError: return False - return default def _cuda_version_to_pytorch_suffix(major, minor): diff --git a/kernels/portable/CMakeLists.txt b/kernels/portable/CMakeLists.txt index b1beb25cb92..69fa749f7c2 100644 --- a/kernels/portable/CMakeLists.txt +++ b/kernels/portable/CMakeLists.txt @@ -144,7 +144,7 @@ if(EXECUTORCH_BUILD_KERNELS_CUSTOM_AOT AND NOT EXECUTORCH_BUILD_ARM_BAREMETAL) set(RPATH "\$ORIGIN/../../extensions/pybindings") endif() set_target_properties( - portable_custom_ops_aot_lib PROPERTIES BUILD_RPATH ${RPATH} INSTALL_RPATH - ${RPATH} + portable_custom_ops_aot_lib PROPERTIES BUILD_RPATH "${RPATH}" INSTALL_RPATH + "${RPATH}" ) endif() diff --git a/kernels/quantized/CMakeLists.txt b/kernels/quantized/CMakeLists.txt index 2dac38205b4..938c3bf81db 100644 --- a/kernels/quantized/CMakeLists.txt +++ b/kernels/quantized/CMakeLists.txt @@ -86,6 +86,14 @@ if(NOT CMAKE_GENERATOR STREQUAL "Xcode" gen_custom_ops_aot_lib( LIB_NAME "quantized_ops_aot_lib" KERNEL_SOURCES "${_quantized_sources}" ) + # The route to the runtime, through the same helper every other target uses. + # Written by hand here before, in two separate blocks that between them + # recorded only the wheel layout and only when the wheel flag was set, so a + # plain -DEXECUTORCH_BUILD_SHARED=ON build produced a library with no route + # to the runtime it links. + executorch_target_shared_runtime_path( + quantized_ops_aot_lib "kernels/quantized" "executorch/kernels/quantized" + ) # Register quantized ops to portable_lib, so that they're available via # pybindings. @@ -126,24 +134,47 @@ if(NOT CMAKE_GENERATOR STREQUAL "Xcode" # libraries will look like "@rpath/_portable_lib.cpython-310-darwin.so", # so we can add an LC_RPATH entry to look in a directory relative to the # installed location of our _portable_lib.so file. To see these LC_* - # values, run `otool -l libquantized_ops_lib.dylib`. + # values, run `otool -l libquantized_ops_lib.dylib`. "extension", not + # "extensions": the plural directory does not exist, so the parent's path + # reached nothing and this library could not find the extension it needs. + # torch is three directories up from here, and this library links it + # directly, so the hop has to be recorded or the only route is the + # absolute path from the build machine. if(APPLE) - set(RPATH "@loader_path/../../extensions/pybindings") + set(RPATH + "@loader_path/../../extension/pybindings;@loader_path/../../../torch/lib" + ) else() - set(RPATH "$ORIGIN/../../extensions/pybindings") + set(RPATH + "$ORIGIN/../../extension/pybindings:$ORIGIN/../../../torch/lib" + ) + endif() + # Appended rather than assigned. The helper above already recorded the + # route to the runtime, and overwriting the property here would drop it. + get_target_property(_existing quantized_ops_aot_lib INSTALL_RPATH) + if(_existing) + set(RPATH "${_existing}:${RPATH}") endif() set_target_properties( - quantized_ops_aot_lib PROPERTIES BUILD_RPATH ${RPATH} INSTALL_RPATH - ${RPATH} + quantized_ops_aot_lib PROPERTIES BUILD_RPATH "${RPATH}" INSTALL_RPATH + "${RPATH}" ) endif() endif() endif() add_library(quantized_kernels ${_quantized_kernels__srcs}) +# The thread pool carries the define that switches parallel_for from a serial +# fallback to the real threaded implementation, so without it quantize, +# dequantize and choose_qparams run on one core. Guarded because a bare metal +# target builds these kernels without a thread pool at all, where the serial +# fallback is the only correct choice. target_link_libraries( quantized_kernels PRIVATE executorch_core kernels_util_all_deps ) +if(TARGET extension_threadpool) + target_link_libraries(quantized_kernels PRIVATE extension_threadpool) +endif() target_compile_options(quantized_kernels PUBLIC ${_common_compile_options}) # Build a library for _quantized_kernels_srcs # diff --git a/pyproject.toml b/pyproject.toml index 1bf343cfd5f..c59ded7ed3e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -2,6 +2,7 @@ requires = [ "cmake>=3.24,<4.0.0", # For building binary targets in the wheel. 4.0.0 breaks third-party CMake build so temporarily pin the version. "packaging>=24.2", # Lower bound required by setuptools + "patchelf; sys_platform == 'linux'", # Writes the runtime search paths that let the shipped libraries find each other. "pip>=23", # For building the pip package. "pyyaml", # Imported by the kernel codegen tools. "setuptools>=77.0.3", # For building the pip package contents. diff --git a/runtime/core/exec_aten/util/tensor_dimension_limit.h b/runtime/core/exec_aten/util/tensor_dimension_limit.h index 6e072ab0582..c690d96dc91 100644 --- a/runtime/core/exec_aten/util/tensor_dimension_limit.h +++ b/runtime/core/exec_aten/util/tensor_dimension_limit.h @@ -8,6 +8,8 @@ #pragma once +#include + namespace executorch::runtime { /** * The expected output size may not be the existing size of any inputs and diff --git a/setup.py b/setup.py index e2c053f4d1f..4505cae1bf4 100644 --- a/setup.py +++ b/setup.py @@ -57,6 +57,7 @@ import re import shutil import site +import stat import subprocess import sys from distutils import log # type: ignore[import-not-found] @@ -185,6 +186,13 @@ def _base_dependencies() -> List[str]: "packaging", "pandas>=2.2.2; python_version >= '3.10'", "parameterized", + # backends/qualcomm/__init__.py cannot be imported from a clean install + # without both of these. It reads the CPU vendor to disable an mkldnn path on + # AMD, and the module it imports first does a module-scope `import requests`, + # so declaring only the cpuinfo half leaves the import failing on the line + # before. + "py-cpuinfo", + "requests", "pytorch-tokenizers", "pyyaml", "ruamel.yaml", @@ -671,8 +679,121 @@ def build_extension(self, ext: _BaseExtension) -> None: # but that would clobber the X bit on any executables. TODO(dbort): This # probably won't work on Windows. if not os.access(src_file, os.W_OK): - # Make the file writable. This should respect the umask. - os.chmod(src_file, os.stat(src_file).st_mode | 0o222) + # The owner only. A mode of 0o222 would also grant write to the group + # and to everyone, and chmod takes an absolute mode so no umask + # narrows it, which turned a 0o555 build output into 0o777. + os.chmod(src_file, os.stat(src_file).st_mode | stat.S_IWUSR) + + # The destination too, and before the rewrite below, which opens the file + # for writing. copy_file preserves mode here on purpose, because this path + # also copies flatc and preserve_mode=False would drop its executable bit, + # so a read-only source arrives read-only. + # + # This mode is the one the wheel archives, so widening it here ships a + # world-writable library. + if not os.access(dst_file, os.W_OK): + os.chmod(dst_file, os.stat(dst_file).st_mode | stat.S_IWUSR) + + _strip_absolute_runtime_paths(dst_file) + + +def _strip_absolute_runtime_paths(library: Path) -> None: + """Remove unusable runtime search paths from a library the wheel ships. + + These libraries are copied out of the build tree rather than installed, so they + still carry every directory the linker recorded while resolving their + dependencies. Two kinds of entry are removed: + + - a directory inside this build, which names the machine that produced the + wheel and cannot exist for a user + - an empty entry, which the loader reads as the process working directory + + Other absolute entries are kept. The Python extensions link torch and resolve it + through the directory the linker recorded, so dropping that would stop them + importing in an environment where torch is not beside them. + + Best effort, because the tool cannot be guaranteed on PATH: the pip package does not + reliably provide a binary inside a build venv, so failing the build here would break + building from source on a machine that simply lacks it. It is declared as a build + requirement so a release build gets it. + + A CPU wheel built without it still works, with the absolute paths left in place. A CUDA + wheel does not: the delegate under backends/cuda/ reaches its dependency in lib/ only + through the paths written here, so without the rewrite that edge is missing. + + What must not happen is both this and its check going quiet together, which is how + a wheel carrying build-machine directories could ship unnoticed. So the check in + the release tests treats a missing patchelf as a failure rather than a skip: the + wheel-build environment has it, and that is where the guarantee belongs. + """ + if library.suffix != ".so" and ".so." not in library.name: + return + patchelf = shutil.which("patchelf") + if patchelf is None: + return + result = subprocess.run( + [patchelf, "--print-rpath", os.fspath(library)], + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + return + original = result.stdout.strip() + if not original: + # No runtime search path at all, which is nothing to clean. patchelf prints + # the same empty string for an absent tag and for one holding a single empty + # entry, so there is nothing to distinguish here and nothing to do either way. + return + + def keep(entry: str) -> bool: + if not entry: + # The loader reads an empty entry as the process working directory. + return False + if not entry.startswith("/"): + return True + # Absolute, so decide by what it points at. A directory inside this build + # cannot exist for a user. + # + # A torch lib directory is dropped when a relative route to torch is already + # recorded, since that route is what resolves torch on an installed wheel + # while the absolute one only names a directory from the machine that built + # it. It is kept when no relative route exists, because then it is the only + # way this library finds torch. + # + # Anything else absolute is a dependency the environment provides and the + # wheel has no relative answer for. + # + # Matched as whole path components rather than as substrings. A bare + # "/cmake-out" also matches "/home/user/cmake-outputs/torchlibs", which is + # an unrelated directory a user could really have, and stripping it breaks + # a dependency the library legitimately resolves there. + if entry.rstrip("/").endswith("/torch/lib") and has_relative_torch_route: + return False + parts = entry.split("/") + # The setuptools staging directory is spelled build/lib.-, + # for example lib.linux-x86_64-cpython-312. A bare startswith("lib.") also + # stripped a real user path like /opt/acme/lib.v2, so match the whole shape. + return not any( + part == "pip-out" + or part == "cmake-out" + or re.fullmatch(r"lib\.[^/]+-(cpython-\d+|\d+(?:\.\d+)*)", part) + for part in parts + ) + + # Whether the library can still reach torch without the absolute entry. Read + # inside keep, which closes over this scope. + has_relative_torch_route = any( + not entry.startswith("/") and entry.rstrip("/").endswith("/torch/lib") + for entry in original.split(":") + ) + rewritten = ":".join(entry for entry in original.split(":") if keep(entry)) + if rewritten == original: + return + subprocess.run( + [patchelf, "--set-rpath", rewritten, os.fspath(library)], + check=True, + ) class CustomBuildPy(build_py): @@ -1090,6 +1211,74 @@ def run(self): # noqa C901 [] if _is_minimal_build() else [ + # Install the shared runtime the Python extension links, rather + # than having the extension contain its own copy. Named without a + # version, so a consumer's find_library(executorch) resolves it: that + # matches libexecutorch.so and not libexecutorch.so.1. A version is + # only useful where something upgrades the library independently of + # what links it, which never happens inside a wheel. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/", + src_name="libexecutorch.so", + dst="executorch/lib/libexecutorch.so", + dependent_cmake_flags=["EXECUTORCH_BUILD_SHARED"], + ), + # Install the profiler next to it, as its own library rather than + # code fused into the Python extension, so a process has one copy of + # it however many consumers load. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/devtools/etdump/", + src_name="libexecutorch_etdump.so", + dst="executorch/lib/libexecutorch_etdump.so", + # Not gated on EXECUTORCH_BUILD_DEVTOOLS. The shared build adds + # the devtools subdirectory itself, so the library exists + # whenever the shared build does. The Python extension carries a + # hard dependency on it, so requiring the option here left a + # wheel whose extension could not load at all. + dependent_cmake_flags=["EXECUTORCH_BUILD_SHARED"], + ), + # Install the shared thread pool next to it. It is a separate + # library so that a process has one pool rather than one per + # component that uses it. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/extension/threadpool/", + src_name="libexecutorch_threadpool.so", + dst="executorch/lib/libexecutorch_threadpool.so", + # The target only exists when both of its dependencies are + # enabled, so packaging has to require them too or a shared + # build with either turned off looks for a file that was + # never built. + dependent_cmake_flags=[ + "EXECUTORCH_BUILD_SHARED", + "EXECUTORCH_BUILD_PTHREADPOOL", + "EXECUTORCH_BUILD_CPUINFO", + ], + ), + # Install the merged CPU kernels beside them, so the operators are + # registered once per process rather than once per component. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/configurations/", + src_name="libexecutorch_kernels_optimized.so", + dst="executorch/lib/libexecutorch_kernels_optimized.so", + # The target is only created when the optimized kernels are + # enabled, so packaging has to require that too rather than + # looking for a file a shared build may never have produced. + dependent_cmake_flags=[ + "EXECUTORCH_BUILD_SHARED", + "EXECUTORCH_BUILD_KERNELS_OPTIMIZED", + ], + ), + # Install the XNNPACK delegate beside them, so a process has one + # copy of it instead of one per component that uses it. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/backends/xnnpack/", + src_name="libexecutorch_backend_xnnpack.so", + dst="executorch/lib/libexecutorch_backend_xnnpack.so", + dependent_cmake_flags=[ + "EXECUTORCH_BUILD_SHARED", + "EXECUTORCH_BUILD_XNNPACK", + ], + ), # Install the prebuilt pybindings extension wrapper for the runtime, # portable kernels, and a selection of backends. This lets users # load and execute .pte files from python. @@ -1172,6 +1361,23 @@ def run(self): # noqa C901 is_dynamic_lib=True, dependent_cmake_flags=["EXECUTORCH_BUILD_CUDA"], ), + # The stream helper the library above records as a dependency. It was + # never shipped, and resolved only because the copied library still + # carried the absolute directory it was linked in, which exists on a + # build machine and nowhere else. Stripping that path is what made the + # omission visible as a failed import. + # + # Shipped beside its consumer rather than in lib/, because that + # directory only exists in the shared build and this has to work + # without it. The glob covers both names the target can have: the + # shared build renames it to advertise it as a wheel component. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/extension/cuda/%BUILD_TYPE%/", + src_name="*extension_cuda", + dst="executorch/backends/cuda/", + is_dynamic_lib=True, + dependent_cmake_flags=["EXECUTORCH_BUILD_CUDA"], + ), BuiltFile( src_dir="%CMAKE_CACHE_DIR%/backends/qualcomm/%BUILD_TYPE%/", src_name="qnn_executorch_backend", diff --git a/tools/cmake/Codegen.cmake b/tools/cmake/Codegen.cmake index 4253fa44dc5..aed21c3a495 100644 --- a/tools/cmake/Codegen.cmake +++ b/tools/cmake/Codegen.cmake @@ -261,15 +261,26 @@ function(gen_custom_ops_aot_lib) executorch_target_link_options_shared_lib(${GEN_LIB_NAME}) if(TARGET portable_lib) target_link_libraries(${GEN_LIB_NAME} PRIVATE portable_lib) + elseif(TARGET executorch_shared) + # Named here as well as retained below, because a PRIVATE link does not + # carry the runtime's include directories and compile definitions, and a + # shared build without the pybind extension would then compile against no + # runtime headers. + target_link_libraries(${GEN_LIB_NAME} PRIVATE executorch_shared) else() target_link_libraries(${GEN_LIB_NAME} PRIVATE executorch_core) endif() + executorch_target_link_shared_runtime(${GEN_LIB_NAME}) endfunction() # Generate a runtime lib for registering operators in Executorch +# +# SHARED opts this library into being a shared object. It is opt-in because most +# callers want the default static library, and only the one shipped in the wheel +# needs to be shared so a process has a single copy of the kernels. function(gen_operators_lib) set(multi_arg_names LIB_NAME KERNEL_LIBS DEPS DTYPE_SELECTIVE_BUILD) - cmake_parse_arguments(GEN "" "" "${multi_arg_names}" ${ARGN}) + cmake_parse_arguments(GEN "SHARED" "" "${multi_arg_names}" ${ARGN}) message(STATUS "Generating operator lib:") message(STATUS " LIB_NAME: ${GEN_LIB_NAME}") @@ -282,7 +293,17 @@ function(gen_operators_lib) set(_opvariant_h ${_out_dir}/selected_op_variants.h) endif() - add_library(${GEN_LIB_NAME}) + if(GEN_SHARED) + add_library(${GEN_LIB_NAME} SHARED) + # The caller names the library and sets its version, because the shipped + # name describes what the library provides rather than which generation + # target produced it, and only the caller knows that. + # + # Ships beside the runtime in the wheel's lib/ directory. + executorch_target_shipped_runtime_path(${GEN_LIB_NAME}) + else() + add_library(${GEN_LIB_NAME}) + endif() set(_srcs_list ${_out_dir}/RegisterCodegenUnboxedKernelsEverything.cpp ${_out_dir}/Functions.h ${_out_dir}/NativeFunctions.h @@ -292,6 +313,21 @@ function(gen_operators_lib) endif() target_sources(${GEN_LIB_NAME} PRIVATE ${_srcs_list}) target_link_libraries(${GEN_LIB_NAME} PRIVATE ${GEN_DEPS}) + # Resolve the runtime from the shared library rather than from the static core + # in GEN_DEPS. Linking the static core gives this library its own copy of the + # operator table, so its static initializer registers into a table nothing + # else reads and the operators appear missing at run time. + # + # Only when this target is itself shared. On a static target the retention + # helper cannot work: PRIVATE link options are dropped on a static library, so + # the --no-as-needed scope never reaches whatever links it, and the helper is + # fatal on that rather than pretending. A static operators library is + # extracted whole into its consumer, and the consumer is what retains the + # runtime, so there is nothing to do here. It still needs the runtime's + # headers, which come through GEN_DEPS. + if(GEN_SHARED) + executorch_target_link_shared_runtime(${GEN_LIB_NAME}) + endif() set(portable_kernels_check "portable_kernels") if(GEN_KERNEL_LIBS) diff --git a/tools/cmake/Utils.cmake b/tools/cmake/Utils.cmake index 958b425c47c..e5c5af38c71 100644 --- a/tools/cmake/Utils.cmake +++ b/tools/cmake/Utils.cmake @@ -47,9 +47,64 @@ function(executorch_msvc_kernel_link_options target_name) ) endfunction() +# Bundle a static library's whole contents into a target. +# +# Deliberately a link option rather than the WHOLE_ARCHIVE link feature: that +# feature refuses to coexist with the plain references other targets make to the +# same archive, which the archives bundled here all have, until CMake 3.30. Link +# options are also emitted before the ordered link libraries, which keeps a +# bundled archive ahead of anything that would otherwise satisfy the same +# symbols. +function(executorch_target_whole_archive target_name archive_target) + # One self-contained option per archive. The path sits inside the option so + # its text is unique, which matters because CMake removes a duplicate option + # and that would leave every archive after the first outside the scope, + # silently dropping its registration objects. + # + # The cost, measured rather than assumed: CMake splits a comma-joined LINKER: + # list at every comma, so an archive whose path contains one reaches the + # linker as two broken arguments and the link fails. Accepted, because the + # alternative of giving the path its own option reintroduces the + # de-duplication problem above, and because this file's pre-existing + # SHELL:LINKER: helpers already break on a path containing a space, which is + # the more common case. Both fail loudly at link time rather than producing a + # binary whose registrations are quietly missing. + target_link_options( + ${target_name} + PRIVATE + "LINKER:--push-state,--whole-archive,$,--pop-state" + ) + # Also link it the ordinary way. A link option naming a file is not a build + # prerequisite, so on its own it lets the archive be rebuilt while the library + # bundling it keeps the previous contents, which is a stale registration + # rather than a build error. + target_link_libraries(${target_name} PRIVATE ${archive_target}) +endfunction() + # Ensure that the load-time constructor functions run. By default, the linker # would remove them since there are no other references to them. function(executorch_target_link_options_shared_lib target_name) + # A shared library cannot be retained with --whole-archive: that flag only + # governs how an archive's members are pulled in, so the library is still + # subject to --as-needed and gets dropped along with its registration + # constructor. Export scoped --no-as-needed retention instead, which is what + # actually keeps a registration-only shared library on the link line. + get_target_property(_target_type ${target_name} TYPE) + if(_target_type STREQUAL "SHARED_LIBRARY" AND NOT (APPLE OR MSVC)) + target_link_options( + ${target_name} + INTERFACE + # One option with the library inside it, for two reasons. A SHELL: string + # would split on spaces and break a path containing one, and separate + # options repeat identical text that CMake de-duplicates, which silently + # leaves every library after the first outside any --no-as-needed scope. + "LINKER:--push-state,--no-as-needed,$,--pop-state" + ) + # Retention is fully handled above, and applying whole-archive to a shared + # library below would do nothing: that flag governs archive member + # extraction, and this target is not an archive. + return() + endif() if(APPLE) executorch_macos_kernel_link_options(${target_name}) elseif(MSVC) @@ -212,8 +267,185 @@ function(executorch_target_copy_mlx_metallib target) endif() endfunction() +# Make a target resolve the ExecuTorch runtime from libexecutorch.so. +# +# Naming the shared runtime as an ordinary dependency is not enough. CMake +# orders link libraries so that an archive precedes what it depends on, which +# puts libexecutorch_core.a ahead of libexecutorch.so; the archive then +# satisfies the runtime symbols first and the target ends up with a private copy +# of the backend registry. Link options come before the ordered libraries, so +# naming the runtime there leaves the archive with nothing left to resolve. +# +# On ELF platforms --no-as-needed is needed around it, because a shared library +# with no already-referenced symbol at the point it appears can be dropped, and +# the static archive further along the line would then supply the registry after +# all. Other linkers keep the reference without it. +function(executorch_target_link_shared_runtime target_name) + executorch_target_retain_shared_library(${target_name} executorch_shared) +endfunction() + +# Put a shared library on a consumer's link line and keep it there. +# +# A library whose only purpose is to run a static initializer, such as a backend +# or an operator registration library, has no symbol the consumer references +# directly, so the linker is free to drop it from DT_NEEDED. Some linkers do +# exactly that and the initializer never runs, which shows up at runtime as a +# backend or kernel that is missing rather than as a link error. +function(executorch_target_retain_shared_library target_name library_target) + if(NOT EXECUTORCH_BUILD_SHARED) + return() + endif() + # A target with no link step of its own cannot carry this. PRIVATE link + # options are dropped on both static and object libraries, because neither + # links, while PRIVATE link libraries still propagate to whatever consumes + # them as $. The consumer then gets the shared runtime with no + # --no-as-needed around it, which is the precise condition this function + # exists to prevent, and it fails silently: the build succeeds and the + # registrations land in a table nothing else reads. + # + # Written as the set of types that CAN link rather than a list of types to + # reject, so a target kind added later does not quietly escape. + get_target_property(_target_type ${target_name} TYPE) + if(NOT ${_target_type} MATCHES "^(SHARED_LIBRARY|MODULE_LIBRARY|EXECUTABLE)$") + message( + FATAL_ERROR + "executorch_target_retain_shared_library(${target_name}) cannot work on a " + "${_target_type}: it has no link step, so PRIVATE link options are dropped and " + "${library_target} would reach a consumer without --no-as-needed, leaving its " + "registrations in a private table. Make ${target_name} SHARED, or retain " + "${library_target} from the target that links it." + ) + endif() + # The library being retained has to be one the loader can drop, or the option + # says nothing. Only checked when it is already defined: CMake resolves link + # libraries lazily, and a caller may legitimately name a library created later + # in the configure, which several call sites here do. + if(TARGET ${library_target}) + get_target_property(_library_type ${library_target} TYPE) + if(NOT ${_library_type} STREQUAL "SHARED_LIBRARY") + message( + FATAL_ERROR + "executorch_target_retain_shared_library(${target_name} ${library_target}): " + "${library_target} is a ${_library_type}, and --no-as-needed only affects a shared " + "library. A static or object library is linked by extraction instead, " + "so use executorch_target_whole_archive." + ) + endif() + endif() + # Scoped per library for the same reason as whole-archive above: unique option + # text, so nothing is de-duplicated out of the retention scope. Without this a + # registration-only library is dropped under the default --as-needed and its + # static initializer never runs. + target_link_options( + ${target_name} + PRIVATE + "LINKER:--push-state,--no-as-needed,$,--pop-state" + ) + target_link_libraries(${target_name} PRIVATE ${library_target}) +endfunction() + +# Mark a library as one the wheel ships, so it finds its siblings wherever it +# ends up. In the wheel all of these land in one directory, so "$ORIGIN" is the +# whole answer. +# +# BUILD_WITH_INSTALL_RPATH is deliberately not used. It REPLACES the build-time +# path with the install one, which drops the dependency directories CMake +# records, and in the build tree these libraries are NOT siblings: the runtime +# sits at the top while the others are in their own subdirectories. Those +# recorded directories are what resolves them there, and packaging strips them +# so nothing absolute ships. +function(executorch_target_shipped_runtime_path target_name) + set_target_properties( + ${target_name} PROPERTIES BUILD_RPATH "$ORIGIN" INSTALL_RPATH "$ORIGIN" + ) +endfunction() + +# Give a target a runtime search path that reaches libexecutorch.so in both +# layouts it can end up in. +# +# The two layouts put the runtime in different places. A wheel keeps it in the +# package's own lib/ directory, a fixed number of levels above wherever the +# target lands. A normal install puts it in the prefix library directory +# instead, which that relative path does not reach, so both routes are recorded. +# +# `wheel_subdir` is where the target lands inside the package, below +# executorch/, and gives the wheel route. `install_destination` is the +# DESTINATION its own install() rule uses, and gives the install route. Both are +# needed because the two are not the same shape everywhere: most targets install +# prefix relative while the Qualcomm ones install under the library directory, +# and deriving one from the other put two libraries' search paths a component +# off. +function(executorch_target_shared_runtime_path target_name wheel_subdir + install_destination +) + if(NOT EXECUTORCH_BUILD_SHARED OR APPLE) + return() + endif() + # Up out of the subdirectory, then into the package's lib/. + string(REGEX REPLACE "[^/]+" ".." _up "${wheel_subdir}") + set(_paths "$ORIGIN/${_up}/lib") + # Made absolute lexically, so a destination that is already absolute, as a + # ${CMAKE_INSTALL_LIBDIR} based one becomes, is handled the same as a prefix + # relative one. + # + # Not file(REAL_PATH): it resolves symlinks on this side only, while the + # library directory on the other side of the subtraction stays unresolved, so + # a symlinked prefix produced a path that climbed out of the install tree and + # named the link itself. Measured with a symlinked prefix, where the answer + # should be three hops up: REAL_PATH gave $ORIGIN/../../../../../prefix/lib64 + # and this gives $ORIGIN/../../../. It also dev-warns once per call site on a + # directory that does not exist until install time, which is every call in a + # clean build. + cmake_path( + ABSOLUTE_PATH + install_destination + BASE_DIRECTORY + "${CMAKE_INSTALL_PREFIX}" + NORMALIZE + OUTPUT_VARIABLE + _installed_dir + ) + file(RELATIVE_PATH _to_libdir "${_installed_dir}" + "${CMAKE_INSTALL_FULL_LIBDIR}" + ) + string(APPEND _paths ":$ORIGIN/${_to_libdir}") + get_target_property(_existing ${target_name} INSTALL_RPATH) + if(_existing) + set(_paths "${_existing}:${_paths}") + endif() + set_target_properties( + ${target_name} PROPERTIES BUILD_RPATH "${_paths}" INSTALL_RPATH "${_paths}" + ) +endfunction() + +# Apply the SONAME policy for a library the project ships. +# +# A distribution package needs a versioned SONAME: it installs +# libexecutorch.so.1 into a system directory where independent packages link it, +# and the version is what lets a later major coexist during an upgrade. That is +# why the shared library support carries VERSION and SOVERSION. +# +# A wheel is the opposite case. The library and the only things that link it +# ship in the same archive and are replaced together, so no version needs +# pinning, and a versioned name actively hurts: `find_library(executorch)` +# matches libexecutorch.so and not libexecutorch.so.1, so a consumer's +# find_package could not locate it. The torch wheel ships plain names with +# unversioned SONAMEs for the same reason. Offering an unversioned symlink +# instead is not equivalent, because a wheel is a zip and the format has no +# portable symlink support. +function(executorch_target_soname_policy target_name) + if(EXECUTORCH_BUILD_WHEEL_DO_NOT_USE) + return() + endif() + set_target_properties( + ${target_name} PROPERTIES VERSION "${PROJECT_VERSION}" + SOVERSION "${PROJECT_VERSION_MAJOR}" + ) +endfunction() + # Create and install a shared library composed from dependency libraries. The -# target links the provided dependencies and carries VERSION/SOVERSION. +# target links the provided dependencies and carries the project's SONAME +# policy. function(executorch_add_shared_library target_name) set(empty_source_name "${target_name}_empty.cpp") file( @@ -224,15 +456,21 @@ function(executorch_add_shared_library target_name) add_library( ${target_name} SHARED "${CMAKE_CURRENT_BINARY_DIR}/${empty_source_name}" ) + # The dependencies are linked plainly, without a retention option, because + # each one already carries its own INTERFACE whole-archive option and so pulls + # its registration objects in when linked. The empty source above exists only + # to give this library a translation unit. + # + # Nothing here can verify that invariant: CMake cannot tell at configure time + # whether every transitive dependency carries the option. It is checked where + # it is observable instead, by the release checks asserting exactly one owner + # per component in the shipped artifact, which is what fails if extraction + # stops working. if(ARGN) target_link_libraries(${target_name} PRIVATE ${ARGN}) endif() - set_target_properties( - ${target_name} - PROPERTIES VERSION "${PROJECT_VERSION}" - SOVERSION "${PROJECT_VERSION_MAJOR}" - LINKER_LANGUAGE CXX - ) + set_target_properties(${target_name} PROPERTIES LINKER_LANGUAGE CXX) + executorch_target_soname_policy(${target_name}) install( TARGETS ${target_name} EXPORT ExecuTorchTargets diff --git a/tools/cmake/cmake_cache.py b/tools/cmake/cmake_cache.py index 2b249ea3244..8e7c864db05 100644 --- a/tools/cmake/cmake_cache.py +++ b/tools/cmake/cmake_cache.py @@ -7,7 +7,11 @@ from dataclasses import dataclass from typing import Dict, Optional -_FALSE_VALUES = {"off", "0", "", "no"} +# Deliberately stricter than CMake. CMake decides false by exclusion, so anything that is not one +# of its false constants is true, including values like "2.0" and "enabled". Here an unrecognised +# spelling reads as off, because this decides whether a component's libraries are packaged and +# shipping a component whose libraries were never built is worse than shipping one fewer. +_TRUE_VALUES = {"on", "true", "yes", "y"} @dataclass @@ -35,9 +39,15 @@ def is_enabled(self, var: str, fallback: bool = False) -> bool: @staticmethod def _is_truthy(value: Optional[str]) -> bool: - if (value is None) or (value.lower().strip() in _FALSE_VALUES): + if value is None: + return False + normalized = value.strip().lower() + if normalized in _TRUE_VALUES: + return True + try: + return int(normalized) != 0 + except ValueError: return False - return True @staticmethod def read_cmake_cache(cache_path: str) -> Dict[str, CacheValue]: diff --git a/tools/cmake/executorch-wheel-config.cmake b/tools/cmake/executorch-wheel-config.cmake index 1d6096a2e96..fa789771872 100644 --- a/tools/cmake/executorch-wheel-config.cmake +++ b/tools/cmake/executorch-wheel-config.cmake @@ -68,11 +68,63 @@ if(_portable_lib_LIBRARY) list(APPEND EXECUTORCH_LIBRARIES _portable_lib) add_library(_portable_lib STATIC IMPORTED) set(EXECUTORCH_INCLUDE_DIRS ${CMAKE_CURRENT_LIST_DIR}/../../include) - # PyTorch requires C++20, so pybindings must be compiled with C++20. set_target_properties( _portable_lib PROPERTIES IMPORTED_LOCATION "${_portable_lib_LIBRARY}" INTERFACE_INCLUDE_DIRECTORIES "${EXECUTORCH_INCLUDE_DIRS}" - CXX_STANDARD 20 + # PyTorch requires C++20, so anything linking this must compile + # as + # C++20. An interface requirement rather than CXX_STANDARD, + # because + # an imported target compiles nothing itself and CXX_STANDARD + # does + # not reach consumers, so a custom-op build could still compile + # as + # C++17 and fail against headers that need C++20. + INTERFACE_COMPILE_FEATURES cxx_std_20 + ) + + # The extension links the runtime rather than containing it, so it no longer + # satisfies the runtime symbols a custom-op library references. Put the + # shipped runtime on this target's interface, which is where the definitions + # moved to, so an out-of-tree operator project keeps building and loading + # against the extension exactly as it did before. + find_library( + EXECUTORCH_RUNTIME_LIBRARY executorch + PATHS "${CMAKE_CURRENT_LIST_DIR}/../../lib" + NO_DEFAULT_PATH + ) + if(EXECUTORCH_RUNTIME_LIBRARY) + set_property( + TARGET _portable_lib + APPEND + PROPERTY INTERFACE_LINK_LIBRARIES "${EXECUTORCH_RUNTIME_LIBRARY}" + ) + # CMake adds a linked library's directory to the consumer's build tree + # runtime search path and strips it on install, so an installed consumer + # library reports libexecutorch.so as not found. Publish the directories + # rather than forcing them onto the target: an interface link option reaches + # every consumer and survives install, which would bake this machine's + # package location into a library the consumer ships onward. A consumer that + # installs elsewhere adds these to its own INSTALL_RPATH. + get_filename_component( + EXECUTORCH_RUNTIME_LIBRARY_DIR "${EXECUTORCH_RUNTIME_LIBRARY}" DIRECTORY + ) + get_filename_component( + EXECUTORCH_PYTHON_EXTENSION_DIR "${_portable_lib_LIBRARY}" DIRECTORY + ) + endif() +endif() + +# find_package checks _FOUND, which is case-sensitive and does not +# match the EXECUTORCH_FOUND spelling this file documents. Without this, a +# REQUIRED find_package succeeds even when nothing usable was located, and the +# consumer goes on to link nothing. +set(executorch_FOUND ${EXECUTORCH_FOUND}) +if(NOT executorch_FOUND AND executorch_FIND_REQUIRED) + message( + FATAL_ERROR + "Found the ExecuTorch package but could not locate the Python extension " + "inside it, so there is nothing to link." ) endif() diff --git a/tools/cmake/preset/default.cmake b/tools/cmake/preset/default.cmake index ae5437ea443..fd440a3dc80 100644 --- a/tools/cmake/preset/default.cmake +++ b/tools/cmake/preset/default.cmake @@ -229,8 +229,8 @@ define_overridable_option( ${_default_executorch_build_cpuinfo} ) define_overridable_option( - EXECUTORCH_BUILD_SHARED "Build a consolidated ExecuTorch shared library" BOOL - OFF + EXECUTORCH_BUILD_SHARED + "Build a consolidated ExecuTorch shared library (Linux only)" BOOL OFF ) # Threadpool size options. At most one can be specified. Note that the default diff --git a/tools/cmake/preset/pybind.cmake b/tools/cmake/preset/pybind.cmake index d292c9ed240..f72680836b7 100644 --- a/tools/cmake/preset/pybind.cmake +++ b/tools/cmake/preset/pybind.cmake @@ -104,6 +104,19 @@ elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") endif() endif() set_overridable_option(EXECUTORCH_BUILD_OPENVINO OFF) + # Ship one shared runtime that both the pybind extension and standalone C++ + # consumers link, so a process has a single backend registry. Linux only: + # macOS C++ consumers are served by the Swift package distribution, and the + # runtime has no export annotations for a Windows DLL. + # + # Not with the CUDA backend, whose libraries this build does not ship yet. The + # CUDA libraries currently reach the wheel carrying the absolute path of the + # directory they were linked in, which resolves only on the machine that built + # them. The shared build removes those paths, so enabling it here before the + # CUDA libraries ship would leave the extension unable to load at all. + if(NOT EXECUTORCH_BUILD_CUDA) + set_overridable_option(EXECUTORCH_BUILD_SHARED ON) + endif() elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows" OR CMAKE_SYSTEM_NAME STREQUAL "WIN32" ) From 6e4248a28a822e7ff7fe87cb6d804dfdcb81eb81 Mon Sep 17 00:00:00 2001 From: shoumikhin Date: Tue, 11 Aug 2026 23:08:09 -0700 Subject: [PATCH 2/6] Ship a C++ SDK in the wheel The wheel ships the runtime, kernels, delegate, thread pool and profiler as separate shared libraries, but nothing outside Python can use them, because the installed CMake package names none of them. A C++ application would have to hard-code paths into the wheel's private layout. The headers have the same gap. The wheel installs only the subset a custom-operator build needs, which leaves out `extension/module`, the entry point the documentation tells C++ callers to use. So the wheel ships the libraries to run a model and no way to call them. Name each shipped library as a CMake component, so `find_package` locates them, and ship the headers a caller needs. A component is just a name a consumer can ask for, and CMake reports a missing one while configuring rather than at link time. ```cmake find_package(executorch 1.5 REQUIRED COMPONENTS kernels_optimized) target_link_libraries(my_app PRIVATE executorch::runtime executorch::kernels_optimized) ``` | component | library it resolves to | | --- | --- | | `executorch::runtime` | `libexecutorch.so` | | `executorch::kernels_optimized` | `libexecutorch_kernels_optimized.so` | | `executorch::backend_xnnpack` | `libexecutorch_backend_xnnpack.so` | | `executorch::threadpool` | `libexecutorch_threadpool.so` | | `executorch::etdump` | `libexecutorch_etdump.so` | Each component records where the wheel keeps its libraries, so an application built against it finds them without the caller setting a library search path. Headers include the module and tensor entry points, the CPU kernel helpers, the allocator and data loader concrete classes Module's constructors take, the profiler entry points, and the FlatTensorDataMap and MergedDataMap types plus the .ptd file header a caller writing a .ptd needs. CMake 3.28 or newer gets these targets. Older versions do not, because they write the `$ORIGIN` marker (the "look next to me" token in a library search path) incorrectly: ``` 3.24.3, 3.27.9 Makefiles double the dollar sign, Ninja drops the name 3.28.4, 3.31.8 both write the token correctly ``` That would produce a target that runs where it was built and fails once the application is copied elsewhere, so no target is defined below 3.28. Those versions get plain variables instead: `EXECUTORCH_LIBRARIES` with the runtime and every shipped library by path, plus `EXECUTORCH_INCLUDE_DIRS`, `EXECUTORCH_COMPILE_DEFINITIONS` and `EXECUTORCH_CXX_STANDARD`. All four are needed, because an imported target carries the definitions and the C++ standard along with the library and a plain path carries neither. Linking the libraries alone stops at `#error "You need C++17 to compile ExecuTorch"`. `ET_USE_THREADPOOL` is added to `EXECUTORCH_COMPILE_DEFINITIONS` on the pre-3.28 route when the thread pool library ships. Without it the runtime header supplies a local inline serial fallback for `parallel_for`, so a consumer following the documented recipe linked the thread pool library and still ran serial code with no diagnostic. Built the wheel, installed it into a clean environment, and built a C++ application against the installed wheel alone: - the application links the runtime, runs a model, and matches eager PyTorch, and still runs after being copied away from the wheel. - asking for a component the wheel does not ship fails while configuring, naming the component. - a version request is honoured, including ranges. - shipped headers can be included on their own, and one entry point per shipped component also links against the shipped libraries. A small number are exempt because they need something outside the package: a Windows shim, a test framework, or a header that says in its own text not to include it directly. The exempt list is compiled too, so an entry that starts working is reported rather than left in place. - the thread pool probe compiles with `ET_USE_THREADPOOL`, on both the modern-CMake route (from the runtime target) and the pre-3.28 route (from `EXECUTORCH_COMPILE_DEFINITIONS`). Without it the header supplies a local inline definition and the probe linked identically whether or not the library was on the link line, so it could not detect the component being dropped. Measured both ways. - an application's runtime search path is recorded as `DT_RUNPATH`, not the older `DT_RPATH`. That matters because `DT_RPATH` is searched ahead of `LD_LIBRARY_PATH` and is inherited by dependencies, so a consumer could not point a locally built or instrumented runtime at their application. Verified by shadowing the runtime through `LD_LIBRARY_PATH` and watching the loader pick it up, which `DT_RPATH` ignores. - on real CMake 3.24 and 3.27, an application configures, builds and runs through the variables. Measured what each one contributes, with the consumer pinned to C++14 so its own standard does not hide the package's requirement: linking `EXECUTORCH_LIBRARIES` alone fails on a missing header, adding the include directories and definitions then fails on the C++ standard, and applying `EXECUTORCH_CXX_STANDARD` builds and loads a model. The kernels also need scoped retention there, because a registration-only library exports nothing the application references and the linker drops it, which showed up as "Missing operator" at run time rather than as a link error. The smoke test now runs the same shape automatically when `EXECUTORCH_PRE_328_CMAKE` points at an older cmake binary, so a future change on the fallback path fails a check rather than only showing up on the first user with older cmake. - `find_package` succeeds when the interpreter on PATH is not the one the wheel was built for. The extension's own file name carries its suffix, so asking a different interpreter for it reported a complete install as not found. Ran on Linux x86_64 and aarch64. The macOS wheel keeps the fused extension and ships no separate libraries, so these checks do not apply there and its smoke test does not run them. ghstack-source-id: 490d0661f05ee374413793788a7688c24c4a4835 ghstack-comment-id: 5215967468 Pull-Request: https://github.com/pytorch/executorch/pull/21639 --- .ci/scripts/wheel/test_cpp_sdk.py | 1238 +++++++++++++++++ .ci/scripts/wheel/test_linux.py | 8 + .ci/scripts/wheel/test_linux_aarch64.py | 6 + README-wheel.md | 4 +- devtools/etdump/etdump_flatcc.h | 1 - docs/source/using-executorch-cpp.md | 145 ++ .../memory_allocator/memory_allocator_utils.h | 6 +- runtime/executor/platform_memory_allocator.h | 1 + setup.py | 175 ++- .../executorch-wheel-config-version.cmake.in | 49 + tools/cmake/executorch-wheel-config.cmake | 691 ++++++++- 11 files changed, 2276 insertions(+), 48 deletions(-) create mode 100644 .ci/scripts/wheel/test_cpp_sdk.py create mode 100644 tools/cmake/executorch-wheel-config-version.cmake.in diff --git a/.ci/scripts/wheel/test_cpp_sdk.py b/.ci/scripts/wheel/test_cpp_sdk.py new file mode 100644 index 00000000000..7b4aaf1a917 --- /dev/null +++ b/.ci/scripts/wheel/test_cpp_sdk.py @@ -0,0 +1,1238 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +"""Checks that a standalone C++ application can use the wheel as an SDK. + +The wheel ships prebuilt runtime, kernel, delegate, thread pool and profiler +libraries, plus headers and a CMake package config. A Python test can exercise none +of that: the Python extension links those libraries itself, so it passes whether or +not the package config names them correctly, whether or not the headers are complete, +and whether or not an application that links them can find them at run time. + +So these checks build and run a real application from outside the wheel. Nothing here +uses the source tree, because a user has only the installed package. +""" + +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +# Exports the model to a .pte and prints the reference outputs, so the C++ side can +# be compared against eager PyTorch rather than merely checked for not crashing. +# +# The same network as the Python parity check: several operator kinds, so a run +# exercises the merged CPU kernels rather than a single add. +_EXPORT_SCRIPT = """ +import json +import sys + +import torch +from executorch.exir import to_edge_transform_and_lower + + +class Net(torch.nn.Module): + def __init__(self): + super().__init__() + self.linear = torch.nn.Linear(8, 16) + self.conv = torch.nn.Conv2d(1, 4, 3, padding=1) + + def forward(self, x, image): + a = torch.relu(self.linear(x)) + b = self.conv(image).flatten(1) + return a.sum(dim=1, keepdim=True) + b.mean(dim=1, keepdim=True) + + +destination, mode = sys.argv[1], sys.argv[2] +torch.manual_seed(0) +model = Net().eval() +example = (torch.randn(2, 8), torch.randn(2, 1, 6, 6)) +with torch.no_grad(): + expected = model(*example) + +partitioners = [] +if mode == "delegate": + from executorch.backends.xnnpack.partition.xnnpack_partitioner import ( + XnnpackPartitioner, + ) + + partitioners = [XnnpackPartitioner()] + +program = to_edge_transform_and_lower( + torch.export.export(model, example), partitioner=partitioners +).to_executorch() +buffer = program.buffer +with open(destination, "wb") as handle: + handle.write(buffer) + +# The inputs travel with the model so the C++ side feeds identical values. Written as +# plain text rather than a tensor format, because reading one is not what is under +# test here and a dependency on one would be a second thing that can fail. +print( + json.dumps( + { + "inputs": [ + {"shape": list(t.shape), "data": t.flatten().tolist()} + for t in example + ], + "expected": expected.flatten().tolist(), + "delegated": mode == "delegate", + "has_xnnpack": b"XnnpackBackend" in bytes(buffer), + } + ) +) +""" + + +_CONSUMER_SOURCE = r""" +// A standalone application. It includes only what the wheel installs and links only +// the wheel's imported targets, so it fails if the shipped headers are incomplete or +// the package config does not make the libraries findable at run time. +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +using namespace executorch::extension; + +namespace { + +// A minimal reader for the numbers the export step printed. Deliberately not a JSON +// library: adding a dependency here would mean a failure could come from the parser +// rather than from the SDK. +std::vector read_floats(const std::string& path) { + std::ifstream file(path); + std::vector values; + float value = 0.0f; + while (file >> value) { + values.push_back(value); + } + return values; +} + +std::vector read_ints(const std::string& path) { + std::ifstream file(path); + std::vector values; + int value = 0; + while (file >> value) { + values.push_back(value); + } + return values; +} + +} // namespace + +int main(int argc, char** argv) { + // Seven, because argv[6] is read below: the program name plus six arguments. A guard of six let a + // caller that passed one too few read past the end of argv. + if (argc < 7) { + std::printf("usage: consumer \n"); + return 2; + } + executorch::runtime::runtime_init(); + + const auto shape_a = read_ints(argv[2]); + auto data_a = read_floats(argv[3]); + const auto shape_b = read_ints(argv[4]); + auto data_b = read_floats(argv[5]); + const auto expected = read_floats(argv[6]); + + std::vector sizes_a(shape_a.begin(), shape_a.end()); + std::vector sizes_b(shape_b.begin(), shape_b.end()); + + // The documented entry points, not the lower-level runtime API. Constructing these + // needs real definitions at link time, so it checks that the shipped headers and + // the shipped libraries agree rather than only that the headers parse. + module::Module module(argv[1]); + const auto load_error = module.load(); + if (load_error != executorch::runtime::Error::Ok) { + std::printf("load failed: 0x%x\n", (unsigned)load_error); + return 1; + } + + auto input_a = make_tensor_ptr(sizes_a, data_a.data()); + auto input_b = make_tensor_ptr(sizes_b, data_b.data()); + + const auto result = module.forward({input_a, input_b}); + if (!result.ok()) { + std::printf("forward failed: 0x%x\n", (unsigned)result.error()); + return 1; + } + + const auto output = result->at(0).toTensor(); + if ((size_t)output.numel() != expected.size()) { + std::printf( + "output has %zu values, expected %zu\n", + (size_t)output.numel(), + expected.size()); + return 1; + } + + // Compared against eager PyTorch, not merely produced. A model that returns wrong + // numbers without erroring would satisfy every other check here. + const float* actual = output.const_data_ptr(); + double worst = 0.0; + for (size_t i = 0; i < expected.size(); ++i) { + const double diff = std::fabs((double)actual[i] - (double)expected[i]); + // Rejected here rather than through the comparison below, because fmax treats a + // NaN as a missing value and returns the other operand, so an all-NaN output + // would leave the running maximum at zero and pass at any tolerance. + if (!std::isfinite(diff)) { + std::printf("output value %zu is not comparable: %g\n", i, (double)actual[i]); + return 1; + } + worst = std::fmax(worst, diff); + } + if (worst > 1e-4) { + std::printf("output differs from eager PyTorch by %g\n", worst); + return 1; + } + + std::printf( + "ok backends=%zu maxdiff=%g\n", + (size_t)executorch::runtime::get_num_registered_backends(), + worst); + return 0; +} +""" + + +def _consumer_cmake(components) -> str: + """A consumer project that links the given components by their public names. + + REQUIRED COMPONENTS rather than a bare find_package, because that is the form the + documentation shows and it has to fail loudly when the wheel does not ship what + it advertises. + """ + requested = " ".join(components) + links = "\n".join( + f"target_link_libraries(consumer PRIVATE executorch::{name})" + for name in components + ) + return f"""cmake_minimum_required(VERSION 3.28) +project(consumer CXX) +find_package(executorch REQUIRED COMPONENTS {requested}) +add_executable(consumer consumer.cpp) +{links} +""" + + +def _tool(name: str) -> str: + """Locate a build tool, including one pip installed beside this interpreter. + + `shutil.which` searches PATH only, and a virtual environment's bin is on PATH only + when the environment is activated. These checks run by invoking the interpreter + directly, so a tool installed into that environment is present on disk and + invisible to a PATH search. + """ + found = shutil.which(name) + if found: + return found + beside = Path(sys.executable).parent / name + return str(beside) if beside.is_file() else name + + +def _installed_package_dir() -> Path: + """Where the wheel installed itself, found without importing it. + + Imported from the source tree, `executorch.__file__` points at the checkout rather + than at the installed package, so a check would read the wrong files and pass while + the wheel was broken. + """ + for entry in sys.path: + candidate = Path(entry) / "executorch" + if (candidate / "share" / "cmake").is_dir(): + return candidate + raise AssertionError( + "no installed executorch package with share/cmake on sys.path; these checks " + "must run against an installed wheel, not the source tree" + ) + + +def _write_tensor(directory: Path, stem: str, tensor) -> tuple: + """Write one tensor's shape and values as whitespace-separated text.""" + shape_file = directory / f"{stem}.shape" + data_file = directory / f"{stem}.data" + shape_file.write_text(" ".join(str(n) for n in tensor["shape"])) + data_file.write_text(" ".join(repr(v) for v in tensor["data"])) + return shape_file, data_file + + +def _export(work_dir: Path, mode: str) -> tuple: + """Export the model to a .pte and return it with the reference numbers.""" + script = work_dir / "export.py" + script.write_text(_EXPORT_SCRIPT) + model = work_dir / f"model_{mode}.pte" + result = subprocess.run( + [sys.executable, str(script), str(model), mode], + capture_output=True, + text=True, + check=False, + ) + assert result.returncode == 0, ( + f"exporting the {mode} model failed, so the C++ side cannot be checked " + f"against it:\n{result.stdout[-2000:]}\n{result.stderr[-2000:]}" + ) + reference = json.loads(result.stdout.strip().splitlines()[-1]) + assert model.is_file(), f"the export step produced no {model}" + return model, reference + + +def _build_consumer(work_dir: Path, name: str, components) -> Path: + """Configure and build the consumer application against the installed package.""" + package_dir = _installed_package_dir() + config = package_dir / "share" / "cmake" / "executorch-config.cmake" + assert config.is_file(), f"the wheel ships no CMake package config at {config}" + + source_dir = work_dir / name + build_dir = work_dir / f"{name}-build" + source_dir.mkdir(parents=True, exist_ok=True) + (source_dir / "consumer.cpp").write_text(_CONSUMER_SOURCE) + (source_dir / "CMakeLists.txt").write_text(_consumer_cmake(components)) + + configured = subprocess.run( + [ + _tool("cmake"), + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_PREFIX_PATH={config.parent}", + ], + capture_output=True, + text=True, + check=False, + ) + assert configured.returncode == 0, ( + f"a consumer requesting {list(components)} could not configure against the " + f"installed package:\n{configured.stdout[-2000:]}\n{configured.stderr[-2000:]}" + ) + built = subprocess.run( + [_tool("cmake"), "--build", str(build_dir)], + capture_output=True, + text=True, + check=False, + ) + assert built.returncode == 0, ( + f"a consumer requesting {list(components)} compiled against the shipped " + f"headers but did not link:\n{built.stdout[-3000:]}\n{built.stderr[-3000:]}" + ) + consumer = build_dir / "consumer" + assert consumer.is_file(), f"the build produced no {consumer}" + return consumer + + +def _run_consumer(consumer: Path, model: Path, reference, work_dir: Path) -> str: + """Run the application and require it to match eager PyTorch.""" + inputs = reference["inputs"] + shape_a, data_a = _write_tensor(work_dir, "a", inputs[0]) + shape_b, data_b = _write_tensor(work_dir, "b", inputs[1]) + expected = work_dir / "expected.data" + expected.write_text(" ".join(repr(v) for v in reference["expected"])) + + # No LD_LIBRARY_PATH. Making the shipped libraries findable is the package + # config's job, and inheriting one from the environment would hide a failure to + # do it. + environment = { + key: value for key, value in os.environ.items() if key != "LD_LIBRARY_PATH" + } + result = subprocess.run( + [ + str(consumer), + str(model), + str(shape_a), + str(data_a), + str(shape_b), + str(data_b), + str(expected), + ], + capture_output=True, + text=True, + check=False, + env=environment, + ) + assert result.returncode == 0, ( + "the C++ application built against the installed wheel did not run " + f"correctly:\n{result.stdout[-2000:]}\n{result.stderr[-2000:]}" + ) + return result.stdout.strip() + + +def test_runtime_alone_links_but_cannot_compute(work_dir: Path) -> None: + """Linking only the runtime builds and loads, and cannot execute. + + Measured rather than assumed: libexecutorch.so defines only primitive operators, such + as aten::sym_size.int, and none of the kernels a model computes with, so an + application linking it alone loads a program and then reports the operators it needs + as missing. That is the intended split, and stating it here documents why the kernels + are a separate component instead of leaving a reader to guess. + + The value of the check is the boundary. It fails if the runtime silently starts + carrying model kernels again, which would mean the split had regressed, and it fails + if the runtime cannot even load a program. + """ + model, reference = _export(work_dir, "plain") + consumer = _build_consumer(work_dir, "runtime-only", ["runtime"]) + + inputs = reference["inputs"] + shape_a, data_a = _write_tensor(work_dir, "ra", inputs[0]) + shape_b, data_b = _write_tensor(work_dir, "rb", inputs[1]) + expected = work_dir / "r_expected.data" + expected.write_text(" ".join(repr(v) for v in reference["expected"])) + environment = { + key: value for key, value in os.environ.items() if key != "LD_LIBRARY_PATH" + } + result = subprocess.run( + [ + str(consumer), + str(model), + str(shape_a), + str(data_a), + str(shape_b), + str(data_b), + str(expected), + ], + capture_output=True, + text=True, + check=False, + env=environment, + ) + combined = result.stdout + result.stderr + assert result.returncode != 0, ( + "an application linking only executorch::runtime executed a model, so the " + "runtime is carrying operator kernels that are supposed to live in their own " + "component" + ) + assert "Missing operator" in combined, ( + "an application linking only the runtime failed for some reason other than " + f"absent kernels, which is the documented behaviour:\n{combined[-1500:]}" + ) + print("✓ executorch::runtime alone links and loads, and has no model kernels") + + +def test_kernels_component_runs_a_model(work_dir: Path) -> None: + """Adding the CPU kernels component keeps the model running and correct.""" + model, reference = _export(work_dir, "plain") + consumer = _build_consumer( + work_dir, "with-kernels", ["runtime", "kernels_optimized"] + ) + output = _run_consumer(consumer, model, reference, work_dir) + print(f"✓ a C++ app linking executorch::kernels_optimized runs a model ({output})") + + +def test_delegated_model_needs_the_delegate_component(work_dir: Path) -> None: + """A delegated model runs when the delegate is linked, and fails when it is not. + + Both halves matter. Only running the positive case would pass even if the delegate + target did nothing, because the runtime falls back to portable kernels for + anything a backend does not claim. The negative case is what shows the delegate is + actually doing the work, and that the retention options on the target are what + make its registration reach the registry. + """ + model, reference = _export(work_dir, "delegate") + assert reference["has_xnnpack"], ( + "the exported program contains no XnnpackBackend payload, so this check would " + "prove nothing about the delegate" + ) + + # The kernels come too. A partitioner claims only what its backend supports, so a + # delegated program still has ordinary operators in it, and an application without + # the kernels fails on those rather than on anything to do with the delegate. + # Measured: this model keeps aten::mean.out outside the XNNPACK partition. + consumer = _build_consumer( + work_dir, + "with-delegate", + ["runtime", "kernels_optimized", "backend_xnnpack"], + ) + output = _run_consumer(consumer, model, reference, work_dir) + print( + f"✓ a C++ app linking executorch::backend_xnnpack runs a delegated model " + f"({output})" + ) + + # The same program, run by an application that has the kernels but not the + # delegate. Only the delegate is removed, so a failure can only be about the + # missing backend rather than about absent operators. + without = _build_consumer(work_dir, "no-delegate", ["runtime", "kernels_optimized"]) + environment = { + key: value for key, value in os.environ.items() if key != "LD_LIBRARY_PATH" + } + inputs = reference["inputs"] + shape_a, data_a = _write_tensor(work_dir, "na", inputs[0]) + shape_b, data_b = _write_tensor(work_dir, "nb", inputs[1]) + expected = work_dir / "n_expected.data" + expected.write_text(" ".join(repr(v) for v in reference["expected"])) + result = subprocess.run( + [ + str(without), + str(model), + str(shape_a), + str(data_a), + str(shape_b), + str(data_b), + str(expected), + ], + capture_output=True, + text=True, + check=False, + env=environment, + ) + assert result.returncode != 0, ( + "a delegated program ran in an application that never linked the delegate, so " + "either the delegate is reaching the registry without being asked for or the " + f"program was not delegated at all. Output was:\n{result.stdout[-1000:]}" + ) + # The exit code alone is not enough: this consumer also exits non-zero on a missing + # file and on too few arguments, so a run that failed to start would pass this check + # while proving nothing about the delegate. + combined = result.stdout + result.stderr + assert "not registered" in combined, ( + "the application failed for some reason other than the delegate being absent, " + f"which is the documented behaviour:\n{combined[-1500:]}" + ) + print( + "✓ the same delegated model fails without executorch::backend_xnnpack, " + "so the component is what registers it" + ) + + +def test_consumer_is_relocatable(work_dir: Path) -> None: + """The application still runs after being moved away from the wheel. + + Building in place leaves the wheel's absolute lib directory on the link line, which + resolves the runtime whatever $ORIGIN says. Copying the application next to a copy + of the libraries, with the original package hidden, is what actually shows the + package is relocatable rather than only working where it was built. + """ + model, reference = _export(work_dir, "plain") + consumer = _build_consumer(work_dir, "relocate", ["runtime", "kernels_optimized"]) + + assert shutil.which("readelf") is not None, "readelf is needed to read the RUNPATH" + dynamic = subprocess.run( + [_tool("readelf"), "-d", str(consumer)], + capture_output=True, + text=True, + check=True, + ).stdout + assert "libexecutorch.so" in dynamic, ( + "the application records no dependency on the shipped runtime, so it is not " + f"linking what the wheel ships:\n{dynamic}" + ) + assert "$ORIGIN" in dynamic, ( + "the application has no $ORIGIN-relative runtime search path, so it cannot " + f"work anywhere but where it was built:\n{dynamic}" + ) + # The newer tag specifically, not just any search path. DT_RPATH is searched + # ahead of LD_LIBRARY_PATH and applies to a dependency's own dependencies, so + # a consumer given DT_RPATH cannot point an instrumented or locally built + # runtime at their application. Both tags satisfy the check above, so without + # this the package could silently go back to the older one. + assert "(RUNPATH)" in dynamic, ( + "the application's runtime search path is recorded as DT_RPATH rather than " + "DT_RUNPATH. DT_RPATH outranks LD_LIBRARY_PATH and is inherited by " + "dependencies, so a consumer could not override a packaged library with " + f"their own build:\n{dynamic}" + ) + + package_dir = _installed_package_dir() + deployed = work_dir / "deployed" + deployed.mkdir(parents=True, exist_ok=True) + shutil.copy2(consumer, deployed / "consumer") + # Every directory the wheel ships a library in, not just lib/. The CUDA delegate records a + # dependency on a library under backends/cuda/, so copying lib/ alone produced a deployment + # that cannot start, and this check could not see it. + for source in ("lib", "backends/cuda"): + directory = package_dir / source + if not directory.is_dir(): + continue + for library in sorted(directory.glob("lib*.so*")): + if library.is_file() and not library.is_symlink(): + shutil.copy2(library, deployed / library.name) + + moved = deployed / "consumer" + # Strip the absolute entry the build left behind, so only $ORIGIN can resolve the + # libraries. Without this the application would find the original wheel and the + # check would pass for the wrong reason. + # Fatal, not a skip. Stripping the absolute entry is the whole point: without it the + # relocated application finds the original package and this check passes for the + # wrong reason. A skip here is indistinguishable from a pass in the log, which is + # the shape of failure this suite exists to avoid. + patchelf = _tool("patchelf") + if shutil.which("patchelf") is None and not Path(patchelf).is_file(): + print("- patchelf not present, installing it so this check can run") + subprocess.run( + [sys.executable, "-m", "pip", "install", "--quiet", "patchelf"], + capture_output=True, + text=True, + check=False, + ) + patchelf = _tool("patchelf") + assert shutil.which("patchelf") or Path(patchelf).is_file(), ( + "patchelf is required to prove the application is relocatable, and could not " + "be installed. Without it the relocated application resolves the original " + "package and the check would pass without testing anything." + ) + current = subprocess.run( + [patchelf, "--print-rpath", str(moved)], + capture_output=True, + text=True, + check=True, + ).stdout.strip() + kept = [ + entry + for entry in current.split(":") + if entry and not entry.startswith(str(package_dir)) + ] + subprocess.run( + [patchelf, "--set-rpath", ":".join(kept) or "$ORIGIN", str(moved)], check=True + ) + + output = _run_consumer(moved, model, reference, work_dir) + print(f"✓ the application still runs deployed away from the wheel ({output})") + + +def test_one_registry_in_the_cpp_process(work_dir: Path) -> None: + """An application linking several components gets one registry, not one each. + + This is the property the split exists to create, checked in a C++ process rather + than only by inspecting symbol tables. Every component resolves the registry from + the one runtime library, so the count a consumer observes grows by exactly what it + links that registers a backend, rather than resetting or doubling. + """ + model, reference = _export(work_dir, "plain") + + def backends_seen(name, components) -> int: + consumer = _build_consumer(work_dir, name, components) + output = _run_consumer(consumer, model, reference, work_dir) + for field in output.split(): + if field.startswith("backends="): + return int(field.split("=", 1)[1]) + raise AssertionError(f"the application printed no backend count: {output}") + + # Both cases have to be able to run the model, so both link the kernels. The + # variable under test is how many further component libraries are linked, not + # whether the program executes. + lean = backends_seen("registry-lean", ["runtime", "kernels_optimized"]) + full = backends_seen( + "registry-full", + ["runtime", "kernels_optimized", "threadpool", "etdump", "backend_xnnpack"], + ) + # The delegate genuinely adds one backend, so the counts differ by exactly that. + # What must not happen is the count resetting or doubling, which is what a second + # registry in the process looks like. + assert full == lean + 1, ( + f"an application linking two components sees {lean} registered backends while " + f"one linking five, of which exactly one registers a backend, sees {full}. A " + "component is carrying its own registry rather than resolving the shared one." + ) + print( + f"✓ one shared registry: {lean} backends with two components, {full} with five" + ) + + +def test_find_package_honours_a_version_request(work_dir: Path) -> None: + """`find_package(executorch )` must accept and reject correctly. + + Without a version file CMake rejects every versioned request, whatever version is + actually installed, so a consumer pinning a minimum cannot configure at all. + The wheel generates the file at packaging time because the version is only known + then: the base comes from version.txt and a nightly overrides it. + """ + package_dir = _installed_package_dir() + version_file = package_dir / "share" / "cmake" / "executorch-config-version.cmake" + assert version_file.is_file(), ( + f"the wheel ships no CMake version file at {version_file}, so find_package " + "rejects every versioned request" + ) + + installed = None + build_version = None + for line in version_file.read_text().splitlines(): + if line.startswith("set(PACKAGE_VERSION"): + installed = line.split('"')[1] + elif line.startswith("set(EXECUTORCH_BUILD_VERSION"): + build_version = line.split('"')[1] + assert installed, f"could not read PACKAGE_VERSION from {version_file}" + assert not installed.startswith("@"), ( + f"the version file still holds an unsubstituted placeholder, {installed}, so " + "packaging copied the template instead of filling it in" + ) + + # The two variables report different things and are filled separately. Only checking the numeric one + # would pass on a file where the full version was truncated to it, or where its placeholder was never + # substituted, and the full version is what a consumer compares to pin an exact build. + assert build_version, f"could not read EXECUTORCH_BUILD_VERSION from {version_file}" + assert not build_version.startswith( + "@" + ), f"the build version still holds an unsubstituted placeholder, {build_version}" + assert build_version.startswith(installed), ( + f"the build version {build_version} does not start with the numeric release {installed}, " + "so they describe different builds" + ) + from executorch.version import __version__ as installed_version + + assert build_version == installed_version, ( + f"the version file says {build_version} but the installed package says {installed_version}, " + "so a consumer pinning an exact build would compare against the wrong one" + ) + + # CMake compares dotted integers only, and find_package rejects a REQUESTED version + # that is not one, so the numeric release part is what a consumer can ask for. The + # wheel's own version can carry more: a dev segment for a nightly and a local part + # such as +cpu or a commit hash. CMake truncates the stored version at the first + # non-numeric segment, which makes those compare equal to the release, so pinning + # the release is the behaviour a consumer actually gets. + release = re.match(r"\d+(?:\.\d+)*", installed) + assert release, f"no numeric release part in the installed version {installed}" + release = release.group(0) + major = release.split(".")[0] + too_new = f"{int(major) + 1}.0" + source_dir = work_dir / "version-probe" + source_dir.mkdir(parents=True, exist_ok=True) + (source_dir / "consumer.cpp").write_text("int main() { return 0; }\n") + + for requested, must_accept in ((release, True), ("0.1", True), (too_new, False)): + # Deliberately the older floor: this probe never links an imported target, so it also checks + # that version acceptance answers correctly below the version those targets need. + (source_dir / "CMakeLists.txt").write_text( + "cmake_minimum_required(VERSION 3.24)\n" + "project(probe CXX)\n" + f"find_package(executorch {requested} REQUIRED)\n" + "add_executable(consumer consumer.cpp)\n" + ) + build_dir = work_dir / f"version-probe-build-{requested}" + result = subprocess.run( + [ + _tool("cmake"), + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_PREFIX_PATH={version_file.parent}", + ], + capture_output=True, + text=True, + check=False, + ) + accepted = result.returncode == 0 + assert accepted == must_accept, ( + f"find_package(executorch {requested}) against an installed {installed} " + f"{'was rejected' if must_accept else 'was accepted'}, which is wrong:\n" + f"{result.stdout[-800:]}{result.stderr[-800:]}" + ) + print( + f"✓ find_package honours a version request (installed {installed}, " + f"accepts {release}, rejects {too_new})" + ) + + +def test_profiler_component_is_usable(work_dir: Path) -> None: + """A C++ application must be able to construct the profiler the etdump component represents. + + Linking a component proves the library resolves. It does not prove a consumer can call anything in it, + and the profiler shipped for a while with only an internal alignment helper as its public surface, so + the component could be requested and linked but not used. + """ + package_dir = _installed_package_dir() + # Globbed, not an exact name: the library carries a version suffix outside a wheel build, and an exact + # match would silently skip this check there. The profiler is required elsewhere in this suite, so its + # absence is a fault rather than a reason to skip. + shipped = sorted((package_dir / "lib").glob("libexecutorch_etdump.so*")) + assert shipped, ( + f"the wheel ships no profiler library under {package_dir / 'lib'}, so the etdump component it " + "advertises cannot be linked" + ) + + source_dir = work_dir / "with-etdump" + source_dir.mkdir(parents=True, exist_ok=True) + (source_dir / "consumer.cpp").write_text( + "#include \n" + "#include \n" + "int main() {\n" + " auto tracer = std::make_unique();\n" + " return tracer == nullptr ? 1 : 0;\n" + "}\n" + ) + (source_dir / "CMakeLists.txt").write_text(_consumer_cmake(["runtime", "etdump"])) + + build_dir = work_dir / "with-etdump-build" + configured = subprocess.run( + [ + _tool("cmake"), + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_PREFIX_PATH={package_dir}", + ], + capture_output=True, + text=True, + check=False, + ) + assert configured.returncode == 0, ( + "configuring an application that uses the profiler failed:\n" + f"{configured.stdout[-1500:]}\n{configured.stderr[-1500:]}" + ) + built = subprocess.run( + [_tool("cmake"), "--build", str(build_dir)], + capture_output=True, + text=True, + check=False, + ) + assert built.returncode == 0, ( + "an application that constructs the profiler failed to build against the installed wheel, so the " + f"component cannot be used by a consumer:\n{built.stdout[-2000:]}\n{built.stderr[-2000:]}" + ) + print("✓ a C++ app linking executorch::etdump constructs the profiler") + + +def test_every_shipped_header_compiles(work_dir: Path) -> None: + """Each installed header must compile on its own against the installed wheel. + + A header that cannot be included is worse than one that is absent, because the failure arrives in + someone else's project at compile time. This caught a profiler header that includes a regular + expression library the wheel does not carry, and whose implementation the shipped library does not + define either. + + Compiled one at a time rather than all together, so the message names the header at fault. + """ + package_dir = _installed_package_dir() + include_root = package_dir / "include" + headers = sorted(include_root.rglob("*.h")) + assert ( + headers + ), f"no headers found under {include_root}, so this check would prove nothing" + + # The same include directories the CMake package exports, since that is what a consumer gets. + includes = [ + f"-I{include_root}", + f"-I{include_root / 'executorch' / 'runtime' / 'core' / 'portable_type' / 'c10'}", + # The same definition every imported target carries. Without it the vendored c10 headers reach for + # a header generated inside a PyTorch build, which no wheel can carry, so compiling without it + # tests a configuration no consumer of this package is ever in. + "-DC10_USING_CUSTOM_GENERATED_MACROS", + ] + # A CUDA wheel's headers reference the CUDA runtime, which the wheel does not publish headers for and + # states as a requirement instead. A consumer of that component supplies a toolkit of its own, so this + # check does the same rather than treating the header as unbuildable. Taken from the toolkit itself, so + # it follows whichever toolkit the build used instead of a list of prefixes that goes stale. + nvcc = shutil.which("nvcc") + cuda_root = os.environ.get("CUDA_HOME") or ( + str(Path(nvcc).parent.parent) if nvcc else "" + ) + if cuda_root and (Path(cuda_root) / "include" / "cuda_runtime.h").is_file(): + includes.append(f"-I{Path(cuda_root) / 'include'}") + # Headers a wheel-only consumer cannot compile and is not expected to. Each needs something outside the + # package: a platform that is not the one being built for, or a third-party library the wheel does not + # carry. They ship because a source build includes them, and holding them to this rule would report a + # defect with no available fix. + needs_more_than_the_wheel = ( + # These ship because other shipped headers include them, so they cannot be left out, and they do + # not compile on their own: each needs a third-party library the wheel links but publishes no + # headers for, or a platform other than the one being built for. + "mman_windows.h", # a Windows compatibility shim, needs the MinGW headers + "testing_util/tensor_util.h", # a test helper, needs a test framework + # These say in their own text that they must not be included directly, and name the header to + # include instead. Including one anyway is a use error rather than a packaging defect. + "c10/util/complex_math.h", + "c10/util/complex_utils.h", + ) + + source = work_dir / "header_probe.cpp" + broken = [] + skipped_but_fine = [] + compiled = 0 + skipped_names = [] + for header in headers: + relative = header.relative_to(include_root) + skipped = relative.as_posix().endswith(needs_more_than_the_wheel) + source.write_text( + f"#include <{relative.as_posix()}>\nint main() {{ return 0; }}\n" + ) + result = subprocess.run( + [_tool("c++"), "-std=c++20", *includes, "-fsyntax-only", str(source)], + capture_output=True, + text=True, + check=False, + ) + if skipped: + skipped_names.append(relative.as_posix()) + if result.returncode == 0: + skipped_but_fine.append(str(relative)) + continue + compiled += 1 + if result.returncode != 0: + missing = re.search(r"fatal error: ([^:]+): No such file", result.stderr) + broken.append( + f"{relative}: {missing.group(1) if missing else 'does not compile'}" + ) + + # A skip list quietly loses value as the code changes: an entry that starts compiling stays skipped and + # nobody notices the coverage was given up for nothing. So the skipped ones are compiled too, and an + # entry that now works is reported rather than left in place. + assert not skipped_but_fine, ( + "these headers are on the skip list but compile now, so the list is stale and is giving up " + f"coverage for no reason. Remove them from it: {skipped_but_fine}" + ) + + assert not broken, ( + "the wheel ships headers that cannot be included from the installed package, so a consumer " + "following the documentation would fail to compile:\n " + "\n ".join(broken) + ) + # The number actually compiled, and the exemptions named. Reporting the total + # selected would claim coverage of headers this never compiled. + print( + f"✓ {compiled} of {len(headers)} shipped headers compile against the installed " + f"wheel; {len(skipped_names)} need something the wheel does not carry " + f"({', '.join(sorted(skipped_names))})" + ) + + +def test_shipped_headers_have_implementations(work_dir: Path) -> None: + """A header that compiles but has no implementation in any shipped library is unusable. + + Compiling proves only that the declarations parse. Two headers once shipped whose implementation + lived in a component no shipped library links, so a consumer got an undefined reference at link + time. A syntax check cannot see that, so this links a real program against the shipped libraries. + + A sample, not a sweep: one entry point per header listed below, chosen because there is no way to + guess a callable declaration from a header alone. It catches a component whose library stops being + linked, which is the failure that shipped. It does not catch a newly added declaration that nobody + implements; that needs a symbol scan over every shipped header, which is worth doing separately. + """ + package = _installed_package_dir() + include_root = package / "include" + + # One small program per header, calling the declaration a consumer would call first. Kept as source + # rather than derived, because there is no way to guess a usable call from a header alone. + probes = { + "extension/memory_allocator/malloc_memory_allocator.h": ( + "#include \n" + "using namespace executorch::extension;\n" + "int main() { MallocMemoryAllocator allocator; return allocator.allocate(16) == nullptr; }\n" + ), + "extension/module/module.h": ( + "#include \n" + "using namespace executorch::extension;\n" + 'int main() { Module module("none.pte"); return module.method_names().ok() ? 0 : 1; }\n' + ), + "extension/tensor/tensor.h": ( + "#include \n" + "using namespace executorch::extension;\n" + "int main() { float data[4] = {}; auto tensor = make_tensor_ptr({2, 2}, data); " + "return tensor->numel() == 4 ? 0 : 1; }\n" + ), + # The profiler, which lives in its own library rather than in the runtime. Included because a + # dead declaration shipped on this class for a while and the probes above could not reach it: + # they link the runtime only, so no etdump symbol was ever resolved here. + "devtools/etdump/etdump_flatcc.h": ( + "#include \n" + "using namespace executorch::etdump;\n" + 'int main() { ETDumpGen generator; generator.create_event_block("probe"); ' + "return 0; }\n" + ), + # The thread pool, which is the other component backing declarations in a shipped header. + # Compiled with ET_USE_THREADPOOL, which is what the package puts on the runtime target when + # the thread pool ships, so this probe sees the same declaration a consumer does. Without it + # the header supplies a local inline definition instead and the probe linked identically with + # and without the library, which made it unable to detect the component being dropped. + # Measured both ways: with the definition, linking the runtime alone fails on + # executorch::extension::parallel_for. + "runtime/kernel/thread_parallel_interface.h": ( + "#define ET_USE_THREADPOOL\n" + "#include \n" + "using namespace executorch::extension;\n" + "int main() { return parallel_for(0, 1, 1, [](int64_t, int64_t) {}) ? 0 : 1; }\n" + ), + } + + includes = [ + f"-I{include_root}", + f"-I{include_root / 'executorch' / 'runtime' / 'core' / 'portable_type' / 'c10'}", + "-DC10_USING_CUSTOM_GENERATED_MACROS", + ] + library_dir = package / "lib" + unresolved = [] + for header, program in probes.items(): + assert ( + include_root / "executorch" / header + ).is_file(), f"{header} is not shipped, so this probe is checking nothing" + source = work_dir / "link_probe.cpp" + source.write_text(program) + result = subprocess.run( + [ + _tool("c++"), + "-std=c++17", + *includes, + str(source), + "-o", + str(work_dir / "link_probe"), + f"-L{library_dir}", + "-lexecutorch", + # The component libraries too, not only the runtime. Linking the runtime alone left + # every declaration outside it unreachable, so a probe for one of those headers passed + # without resolving anything. Missing ones are skipped by the loop below. + *[ + f"-l{name}" + for name in ( + "executorch_etdump", + "executorch_kernels_optimized", + "executorch_threadpool", + ) + if (library_dir / f"lib{name}.so").is_file() + ], + f"-Wl,-rpath,{library_dir}", + ], + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + missing = re.findall(r"undefined reference to `([^']+)'", result.stderr) + unresolved.append(f"{header}: {sorted(set(missing))[:3] or 'did not link'}") + + assert not unresolved, ( + "the wheel ships headers whose implementation is in no shipped library, so a consumer compiles " + "and then fails to link:\n " + "\n ".join(unresolved) + ) + print(f"✓ all {len(probes)} probed headers link against the shipped libraries") + + +def test_documented_example_compiles(work_dir: Path) -> None: + """The C++ example in the documentation must compile against the installed wheel. + + Extracted from the documentation rather than copied here, so the two cannot drift. A + reader who follows the documentation gets code that builds, and a dangling include or + a renamed entry point fails this check instead of shipping. + """ + here = Path(__file__).resolve() + root = here.parents[3] if len(here.parents) > 3 else here.parent + documentation = root / "docs" / "source" / "using-executorch-cpp.md" + if not documentation.is_file(): + print("- the documentation is not present, skipping the example check") + return + + # The first fenced cpp block after the prebuilt-package heading. Anchored on the + # heading so an unrelated example elsewhere on the page is not picked up. + text = documentation.read_text() + marker = "### Using the prebuilt libraries from the pip package" + assert marker in text, ( + f"{documentation.name} no longer documents the prebuilt package, so a reader has " + "no instructions for the libraries this wheel ships" + ) + section = text[text.index(marker) :] + blocks = re.findall(r"```cpp\n(.*?)```", section, re.S) + assert blocks, ( + f"{documentation.name} documents the prebuilt package but shows no C++ example, " + "so nothing proves the documented usage compiles" + ) + + source_dir = work_dir / "documented" + source_dir.mkdir(parents=True, exist_ok=True) + (source_dir / "main.cpp").write_text(blocks[0]) + # The components the documentation itself tells a reader to ask for. + (source_dir / "CMakeLists.txt").write_text( + _consumer_cmake(["runtime", "kernels_optimized"]).replace( + "consumer.cpp", "main.cpp" + ) + ) + + package_dir = _installed_package_dir() + config = package_dir / "share" / "cmake" / "executorch-config.cmake" + build_dir = work_dir / "documented-build" + configured = subprocess.run( + [ + _tool("cmake"), + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_PREFIX_PATH={config.parent}", + ], + capture_output=True, + text=True, + check=False, + ) + assert configured.returncode == 0, ( + "the documented example does not configure against the installed package:\n" + f"{configured.stdout[-1500:]}{configured.stderr[-1500:]}" + ) + built = subprocess.run( + [_tool("cmake"), "--build", str(build_dir)], + capture_output=True, + text=True, + check=False, + ) + assert built.returncode == 0, ( + "the documented example does not build against the installed package, so a " + f"reader following the documentation gets code that fails:\n" + f"{built.stdout[-2500:]}{built.stderr[-2500:]}" + ) + print("✓ the C++ example in the documentation compiles against the wheel") + + +def _provision_pre_328_cmake(work_dir: Path) -> str: + """Return a path to a cmake older than 3.28, or "" if one cannot be had. + + The route this check exercises is only reachable with such a binary, and a + release builder is not obliged to carry one, so fetch it rather than leaving + the check permanently skipped. Failure to fetch is not a test failure: the + caller reports the missing coverage instead. + """ + override = os.environ.get("EXECUTORCH_PRE_328_CMAKE", "") + if override and Path(override).is_file(): + return override + + venv_dir = work_dir / "pre-328-cmake" + binary = venv_dir / "bin" / "cmake" + if not binary.is_file(): + try: + subprocess.run( + [sys.executable, "-m", "venv", str(venv_dir)], + check=True, + capture_output=True, + ) + subprocess.run( + [ + str(venv_dir / "bin" / "pip"), + "install", + "--quiet", + "cmake==3.24.*", + ], + check=True, + capture_output=True, + ) + except (subprocess.CalledProcessError, OSError) as error: + print(f"- could not provision a pre-3.28 cmake: {error}") + return "" + return str(binary) if binary.is_file() else "" + + +def test_pre_3_28_route_builds_a_consumer_through_variables(work_dir: Path) -> None: + """The pre-3.28 route offers only variables, and those variables have to work. + + CMake before 3.28 writes the $ORIGIN token in a runtime search path incorrectly, so + the config file skips its imported targets on those versions and exposes plain + variables instead. The whole modern-CMake test set never enters that branch, because + cmake_minimum_required(3.28) does not lower CMAKE_VERSION and the tests above run + with whatever cmake is on PATH. So a consumer stuck on an older cmake would find at + run time that the wheel produced no usable link. + + Runs only when EXECUTORCH_PRE_328_CMAKE points at a cmake binary older than 3.28, + since a released wheel is not obliged to carry one. Skipped otherwise, with a + message that says so, so a build that has no old cmake still reports the coverage + it lacks rather than reporting green. + """ + old_cmake = os.environ.get("EXECUTORCH_PRE_328_CMAKE", "") + if not old_cmake or not Path(old_cmake).is_file(): + old_cmake = _provision_pre_328_cmake(work_dir) + if not old_cmake: + print( + "- no cmake older than 3.28 is available, skipping the pre-3.28 route " + "check" + ) + return + version = subprocess.run( + [old_cmake, "--version"], capture_output=True, text=True, check=True + ).stdout.splitlines()[0] + match = re.search(r"(\d+)\.(\d+)", version) + assert match, f"could not read a version from {old_cmake}: {version!r}" + major, minor = int(match.group(1)), int(match.group(2)) + assert (major, minor) < (3, 28), ( + f"EXECUTORCH_PRE_328_CMAKE={old_cmake} is version {major}.{minor}, but this " + "check needs a binary older than 3.28 to enter the fallback route" + ) + + package_dir = _installed_package_dir() + config = package_dir / "share" / "cmake" / "executorch-config.cmake" + assert config.is_file(), f"the wheel ships no CMake package config at {config}" + + source_dir = work_dir / "pre-328" + source_dir.mkdir(parents=True, exist_ok=True) + (source_dir / "consumer.cpp").write_text(_CONSUMER_SOURCE) + # No COMPONENTS and no named target, which is the shape the older-CMake route + # forces. The variables have to carry everything a consumer needs: the runtime and + # its components, the include directories, the compile definitions and the C++ + # standard. + (source_dir / "CMakeLists.txt").write_text( + "cmake_minimum_required(VERSION 3.19)\n" + "project(consumer CXX)\n" + "find_package(executorch REQUIRED)\n" + "add_executable(consumer consumer.cpp)\n" + "target_include_directories(consumer PRIVATE ${EXECUTORCH_INCLUDE_DIRS})\n" + "target_compile_definitions(consumer PRIVATE ${EXECUTORCH_COMPILE_DEFINITIONS})\n" + "target_link_libraries(consumer PRIVATE ${EXECUTORCH_LIBRARIES})\n" + "set_target_properties(consumer PROPERTIES CXX_STANDARD ${EXECUTORCH_CXX_STANDARD})\n" + ) + build_dir = work_dir / "pre-328-build" + for command in ( + [ + old_cmake, + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_PREFIX_PATH={config.parent}", + ], + [old_cmake, "--build", str(build_dir)], + ): + result = subprocess.run(command, capture_output=True, text=True, check=False) + assert result.returncode == 0, ( + "a consumer on CMake older than 3.28 could not build against the wheel " + f"through the documented variables:\n" + f"{result.stdout[-2000:]}\n{result.stderr[-2000:]}" + ) + + consumer = build_dir / "consumer" + assert consumer.is_file(), f"the build produced no {consumer}" + # The runtime and CPU kernels have to be on the link line, since the variables are + # the only thing that carries them on this route. Reading the dynamic section rather + # than running because running needs a model, which the modern-CMake tests above + # cover once and this one only owns the variables path. + dependencies = subprocess.run( + ["readelf", "-d", str(consumer)], capture_output=True, text=True, check=True + ).stdout + assert "libexecutorch.so" in dependencies, ( + "a consumer built through EXECUTORCH_LIBRARIES on pre-3.28 CMake does not " + f"depend on the runtime:\n{dependencies}" + ) + assert "libexecutorch_kernels_optimized" in dependencies, ( + "the pre-3.28 aggregate does not carry the CPU kernels, so a consumer built " + "through it would fail at run time with operators reported missing" + ) + print( + f"✓ a consumer on CMake {major}.{minor} builds through EXECUTORCH_LIBRARIES " + "and links the runtime plus the CPU kernels" + ) + + +def run_tests(work_dir: Path) -> None: + test_find_package_honours_a_version_request(work_dir) + test_profiler_component_is_usable(work_dir) + test_every_shipped_header_compiles(work_dir) + test_shipped_headers_have_implementations(work_dir) + test_documented_example_compiles(work_dir) + test_runtime_alone_links_but_cannot_compute(work_dir) + test_kernels_component_runs_a_model(work_dir) + test_pre_3_28_route_builds_a_consumer_through_variables(work_dir) + test_delegated_model_needs_the_delegate_component(work_dir) + test_consumer_is_relocatable(work_dir) + test_one_registry_in_the_cpp_process(work_dir) + + +if __name__ == "__main__": + with tempfile.TemporaryDirectory() as directory: + run_tests(Path(directory)) diff --git a/.ci/scripts/wheel/test_linux.py b/.ci/scripts/wheel/test_linux.py index d76ed6b2462..fdc11478adc 100644 --- a/.ci/scripts/wheel/test_linux.py +++ b/.ci/scripts/wheel/test_linux.py @@ -11,6 +11,7 @@ from pathlib import Path import test_base +import test_cpp_sdk import test_shared_libraries from examples.models import Backend, Model @@ -50,6 +51,13 @@ with tempfile.TemporaryDirectory() as work_dir: test_shared_libraries.run_tests(Path(work_dir)) + # And that a C++ application outside the wheel can actually use them. + # Nothing above covers this: the Python extension links those libraries + # itself, so it passes whether or not the package config names them or the + # shipped headers are complete. + with tempfile.TemporaryDirectory() as work_dir: + test_cpp_sdk.run_tests(Path(work_dir)) + test_base.run_tests( model_tests=[ test_base.ModelTest( diff --git a/.ci/scripts/wheel/test_linux_aarch64.py b/.ci/scripts/wheel/test_linux_aarch64.py index b268c72cea3..d8e8c25dba2 100644 --- a/.ci/scripts/wheel/test_linux_aarch64.py +++ b/.ci/scripts/wheel/test_linux_aarch64.py @@ -9,6 +9,7 @@ from pathlib import Path import test_base +import test_cpp_sdk import test_shared_libraries from examples.models import Backend, Model @@ -36,6 +37,11 @@ with tempfile.TemporaryDirectory() as work_dir: test_shared_libraries.run_tests(Path(work_dir)) + # And that a C++ application outside the wheel can actually use those + # libraries, which nothing above covers. + with tempfile.TemporaryDirectory() as work_dir: + test_cpp_sdk.run_tests(Path(work_dir)) + test_base.run_tests( model_tests=[ test_base.ModelTest( diff --git a/README-wheel.md b/README-wheel.md index f3a89f342ee..5a38ba24084 100644 --- a/README-wheel.md +++ b/README-wheel.md @@ -22,8 +22,8 @@ The prebuilt `executorch.runtime` module included in this package provides a way to run ExecuTorch `.pte` files, with some restrictions: * Only [core ATen operators](docs/source/ir-ops-set-definition.md) are linked into the prebuilt module * Only the [XNNPACK backend delegate](docs/source/backends/xnnpack/xnnpack-overview.md) is linked into the prebuilt module. -* \[macOS only] [Core ML](docs/source/backends/coreml/coreml-overview.md) and [MPS](docs/source/backends/mps/mps-overview.md) backend - are also linked into the prebuilt module. +* \[macOS only] [Core ML](docs/source/backends/coreml/coreml-overview.md) backend is + also linked into the prebuilt module. * \[Linux x86_64] [QNN](docs/source/backends-qualcomm.md) backend is linked into the prebuilt module. * \[Linux] [OpenVINO](docs/source/build-run-openvino.md) backend is also linked into the prebuilt module. OpenVINO requires the runtime to be installed separately: diff --git a/devtools/etdump/etdump_flatcc.h b/devtools/etdump/etdump_flatcc.h index 8b39b243165..3a14747d096 100644 --- a/devtools/etdump/etdump_flatcc.h +++ b/devtools/etdump/etdump_flatcc.h @@ -74,7 +74,6 @@ class ETDumpGen : public ::executorch::runtime::EventTracer { public: ETDumpGen(::executorch::runtime::Span buffer = {nullptr, (size_t)0}); ~ETDumpGen() override; - void clear_builder(); void create_event_block(const char* name) override; virtual ::executorch::runtime::EventTracerEntry start_profiling( diff --git a/docs/source/using-executorch-cpp.md b/docs/source/using-executorch-cpp.md index 5505ade9573..f825fe0785b 100644 --- a/docs/source/using-executorch-cpp.md +++ b/docs/source/using-executorch-cpp.md @@ -40,6 +40,151 @@ Running a model using the low-level runtime APIs allows for a high-degree of con ## Building with CMake +There are two ways to get the C++ runtime. Linking the prebuilt libraries from the pip +package needs no source checkout and is the quicker option. Building from source gives +you every option the project has, and is what you need for a platform the wheel does not +cover. + +### Using the prebuilt libraries from the pip package + +On Linux, `pip install executorch` includes prebuilt shared libraries, the public +headers, and a CMake package, so a C++ application can link the runtime without building +ExecuTorch itself: + +```cmake +# CMakeLists.txt +cmake_minimum_required(VERSION 3.28) +project(my_app CXX) + +find_package(executorch REQUIRED COMPONENTS kernels_optimized) + +add_executable(my_app main.cpp) +target_link_libraries(my_app PRIVATE executorch::runtime + executorch::kernels_optimized) +``` + +Point CMake at the installed package when you configure: + +``` +cmake -S . -B build \ + -DCMAKE_PREFIX_PATH="$(python -c 'import executorch, pathlib; print(pathlib.Path(executorch.__path__[0]) / "share" / "cmake")')" +cmake --build build +``` + +The application uses the same `Module` and `TensorPtr` APIs described above: + +```cpp +// main.cpp +#include +#include + +#include +#include + +using namespace executorch::extension; + +int main() { + Module module("model.pte"); + + std::vector data(2 * 8, 1.0f); + auto input = make_tensor_ptr({2, 8}, data.data()); + + const auto result = module.forward(input); + if (!result.ok()) { + std::printf("forward failed: 0x%x\n", (unsigned)result.error()); + return 1; + } + std::printf("ok, %zu outputs\n", result->size()); + return 0; +} +``` + +#### What each component provides + +Ask for the components your model needs. A component the wheel was not built with is +reported while CMake configures, rather than failing later at link time. + +| Component | What it provides | +| --- | --- | +| `executorch::runtime` | the program loader and executor. Always present. | +| `executorch::kernels_optimized` | CPU operator kernels. Needed for any operator a delegate does not claim. | +| `executorch::backend_xnnpack` | the XNNPACK delegate. | +| `executorch::threadpool` | the shared thread pool. | +| `executorch::etdump` | the profiler. | + +The runtime on its own loads a program but registers only primitive operators, not the +kernels a model computes with, so a model that is not fully delegated needs a kernel +component too. Linking a delegate is what registers it: a program delegated to XNNPACK +fails to load in an application that did not link `executorch::backend_xnnpack`. + +To require a minimum version, pass it to `find_package`: + +```cmake +find_package(executorch 1.0 REQUIRED) +``` + +#### On CMake older than 3.28 + +The example above needs CMake 3.28. Older versions write the `$ORIGIN` marker (the +"look next to me" token in a library search path) incorrectly, which would leave you with +a target that runs where it was built and fails once the application is copied +elsewhere. Rather than hand you a target that behaves that way, the package defines no +imported targets below 3.28 and exports plain variables instead. + +An imported target carries more than a library path, so on this route you have to apply +the rest yourself. Linking the libraries alone does not compile: + +```cmake +cmake_minimum_required(VERSION 3.19) +project(my_app CXX) + +find_package(executorch REQUIRED) + +add_executable(my_app main.cpp) +target_include_directories(my_app PRIVATE ${EXECUTORCH_INCLUDE_DIRS}) +target_compile_definitions(my_app PRIVATE ${EXECUTORCH_COMPILE_DEFINITIONS}) +target_link_libraries(my_app PRIVATE ${EXECUTORCH_LIBRARIES}) +set_property(TARGET my_app PROPERTY CXX_STANDARD ${EXECUTORCH_CXX_STANDARD}) +set_property(TARGET my_app PROPERTY CXX_STANDARD_REQUIRED ON) +``` + +On this route the application also has to record where the libraries live, or it runs +from its build directory and then fails to start once installed with a message like +`libexecutorch.so: cannot open shared object file`. CMake records the wheel's library +directory while building, because the libraries are named by absolute path, but it removes +that entry on install. Ask for it to be kept: + +```cmake +set_property(TARGET my_app PROPERTY INSTALL_RPATH "${EXECUTORCH_RUNTIME_LIBRARY_DIR}") +target_link_options(my_app PRIVATE "LINKER:--enable-new-dtags") +``` + +The second line matters on Linux. Without it this linker records the older `DT_RPATH` tag, which is +searched before `LD_LIBRARY_PATH` and also applies to your dependencies' own dependencies, so you +could not point the application at a different build of the runtime. With it you get `DT_RUNPATH`, +which only affects your application and stays overridable. + +The imported target route does not need this on Linux: the package sets its search paths as +explicit link options, and those survive installation. + +On macOS it does need one line. CMake removes an entry that points at a directory holding a +library the application linked, so the entry naming the wheel's own directory is deleted from +the installed binary and it stops finding the runtime: + +```cmake +set_property(TARGET my_app PROPERTY INSTALL_RPATH_USE_LINK_PATH TRUE) +``` + +An application deployed beside the libraries is unaffected either way, because the +`@loader_path` and `$ORIGIN` entries are kept. + +`EXECUTORCH_LIBRARIES` names the runtime and every component the wheel shipped, so you +cannot choose components on this route. Upgrade to CMake 3.28 and link the specific +targets you need instead. + +### Building from source + + ExecuTorch uses CMake as the primary build system. Inclusion of the module and tensor APIs are controlled by the `EXECUTORCH_BUILD_EXTENSION_MODULE` and `EXECUTORCH_BUILD_EXTENSION_TENSOR` CMake options. As these APIs may not be supported on embedded systems, they are disabled by default when building from source. The low-level API surface is always included. To link, add the `executorch` target as a CMake dependency, along with `executorch_backends`, `executorch_extensions`, and `extension_kernels`, to link all configured backends, extensions, and kernels. ``` diff --git a/extension/memory_allocator/memory_allocator_utils.h b/extension/memory_allocator/memory_allocator_utils.h index 079537e60cd..369d9c07648 100644 --- a/extension/memory_allocator/memory_allocator_utils.h +++ b/extension/memory_allocator/memory_allocator_utils.h @@ -16,12 +16,10 @@ #include #include -using executorch::runtime::Error; -using executorch::runtime::Result; namespace executorch::extension::utils { // Util to get alighment adjusted allocation size -inline Result get_aligned_size(size_t size, size_t alignment) { +inline runtime::Result get_aligned_size(size_t size, size_t alignment) { // The minimum alignment that malloc() is guaranteed to provide. static constexpr size_t kMallocAlignment = alignof(std::max_align_t); if (alignment > kMallocAlignment) { @@ -31,7 +29,7 @@ inline Result get_aligned_size(size_t size, size_t alignment) { const size_t extra = alignment - 1; if ET_UNLIKELY (extra >= SIZE_MAX - size) { ET_LOG(Error, "Malloc size overflow: size=%zu + extra=%zu", size, extra); - return Result(Error::InvalidArgument); + return runtime::Result(runtime::Error::InvalidArgument); } size += extra; } diff --git a/runtime/executor/platform_memory_allocator.h b/runtime/executor/platform_memory_allocator.h index 601a4c19c85..a103da15501 100644 --- a/runtime/executor/platform_memory_allocator.h +++ b/runtime/executor/platform_memory_allocator.h @@ -13,6 +13,7 @@ #include #include +#include #include #include #include diff --git a/setup.py b/setup.py index 4505cae1bf4..dddac8771a0 100644 --- a/setup.py +++ b/setup.py @@ -85,6 +85,42 @@ format="%(asctime)s [%(levelname)s] %(message)s", ) +# Headers swept in by a directory copy that a consumer of the wheel cannot use, because each needs +# something the wheel does not carry. Publishing one is worse than leaving it out: the failure arrives in +# someone else's project rather than here. +# +# Matched on the path ending, not the bare file name. Two different headers here share the name +# tensor_util.h, one a widely included utility and one a test helper, so a name match either kept the test +# helper or removed the utility everything needs. +# +# Only headers that nothing else the wheel installs includes belong here. A header other shipped headers +# pull in must keep shipping even when it cannot be compiled on its own. +_UNSHIPPABLE_HEADERS = frozenset( + { + # Needs a header generated when the schema is compiled, which in turn needs the FlatBuffers C++ + # headers. Those are a third-party library this wheel does not vendor. + "runtime/executor/tensor_parser.h", + # A test helper, needing a test framework the wheel does not ship. + "runtime/core/testing_util/error_matchers.h", + # Reads processor details through cpuinfo, whose headers the wheel does not publish. + "extension/threadpool/cpuinfo_utils.h", + # Holds a pthreadpool member by value, so it needs that library's header, which the wheel does not + # publish either. The component it belongs to is a link dependency the runtime carries, not + # something a consumer includes. + "extension/threadpool/threadpool.h", + # Declares CPUCachingAllocator, whose implementation is in a component no shipped library links, + # so including it compiles and then fails at link time with an undefined reference. + "extension/memory_allocator/cpu_caching_malloc_allocator.h", + # Declares BundledModule, which is built only for the Python bindings, so its implementation is in + # the Python extension. A C++ application cannot link that, and building the source instead needs + # bundled-program headers the wheel does not publish. + "extension/module/bundled_module.h", + # Declares FileDescriptorDataLoader, whose implementation is in no CMake target at all, so no + # shipped library defines it. Including it compiles and then fails at link time. + "extension/data_loader/file_descriptor_data_loader.h", + } +) + try: from tools.cmake.cmake_cache import CMakeCache except ImportError: @@ -867,10 +903,21 @@ def run(self): "tools/cmake/executorch-wheel-config.cmake", "share/cmake/executorch-config.cmake", ), + # And again where CMake looks when a consumer points CMAKE_PREFIX_PATH at the + # package root, which is the ordinary way to use an installed package. CMake + # searches /lib/cmake/, not /share/cmake directly, so + # without this copy the root is not a usable prefix and a consumer needs a + # path that names this project's layout. The first location stays because the + # existing contract uses it. + ( + "tools/cmake/executorch-wheel-config.cmake", + "lib/cmake/executorch/executorch-config.cmake", + ), ] - # Copy all the necessary headers into include/executorch/ so that they can - # be found in the pip package. This is the subset of headers that are - # essential for building custom ops extensions. + # The headers the package installs. Two audiences now: a custom-operator + # build, which needs the kernel and tensor helpers, and a C++ application + # using the shipped libraries as an SDK, which needs the documented entry + # points as well. # TODO: Use cmake to gather the headers instead of hard-coding them here. # For example: # https://discourse.cmake.org/t/installing-headers-the-modern-way-regurgitated-and-revisited/3238/3 @@ -883,9 +930,40 @@ def run(self): "extension/kernel_util/", "extension/tensor/", "extension/threadpool/", + # Module is how the documentation tells a C++ application to load and + # run a program. Without it the package ships the libraries to do that + # and no way to call them, which the C++ consumer check catches. + "extension/module/", + # Module's constructors take unique_ptr to the runtime's allocator + # and loader bases, whose headers already ship. These supply the + # concrete subclasses a caller has to construct to pass one, such as + # MallocMemoryAllocator and FileDataLoader. + "extension/memory_allocator/", + "extension/data_loader/", + # The MergedDataMap and FlatTensorDataMap entry points, whose + # implementations ship inside libexecutorch.so. The .ptd file header + # is included too, so a caller writing or reading a .ptd by hand has + # its declarations. + "extension/named_data_map/merged_data_map.h", + "extension/flat_tensor/flat_tensor_data_map.h", + "extension/flat_tensor/serialize/flat_tensor_header.h", + # ETDump, whose library the package ships as a component. A profiler + # that cannot be included is a library nobody can call. + # + # The whole directory except the filter, which includes a regular + # expression library the wheel does not carry and whose implementation + # is not in the shipped library either. Publishing a header that cannot + # be included is worse than not publishing it, because the failure + # arrives at compile time in someone else's project. + "devtools/etdump/etdump_flatcc.h", + "devtools/etdump/emitter.h", + "devtools/etdump/utils.h", + "devtools/etdump/data_sinks/", ]: - src_list = Path(include_dir).rglob("*.h") - for src in src_list: + # A directory entry publishes everything under it, and a file entry publishes + # just that file. Some directories hold headers a consumer cannot compile + # against, so those are named individually rather than swept in. + for src in _headers_to_install(Path(include_dir)): src_to_dst.append( (str(src), os.path.join("include/executorch", str(src))) ) @@ -923,6 +1001,93 @@ def run(self): self.mkpath(os.path.dirname(dst_file)) self.copy_file(src_file, dst_file, preserve_mode=False) + if not _is_minimal_build(): + self._write_cmake_version_file(dst_root) + + def _write_cmake_version_file(self, dst_root: str) -> None: + """Write the CMake package version file, so `find_package(executorch 1.2)` works. + + Generated rather than copied, because the version is only known here: + version.txt gives the base and BUILD_VERSION overrides it for a nightly. A + checked-in file would go stale the first time either changed. + """ + template = os.path.join( + os.path.dirname(os.path.abspath(__file__)), + "tools", + "cmake", + "executorch-wheel-config-version.cmake.in", + ) + with open(template) as handle: + contents = handle.read() + # Only the numeric release part. A Python version can carry a local segment + # such as "1.5.0+cpu" or a development suffix, and `find_package(executorch + # 1.5.0+cpu)` is rejected by CMake as an invalid argument, so a consumer could + # not name the version this file reports. Strip to the dotted numbers CMake + # can compare, which is what a consumer asks for in practice. + # Two variables with different jobs. CMake compares PACKAGE_VERSION, so it has to be the + # numeric release and nothing else. EXECUTORCH_BUILD_VERSION is documented as the full + # version, which is what a consumer pinning an exact build compares against, so filling it + # from the numeric part would make that comparison pass against a different wheel. + build_version = Version.string() + cmake_version = re.match(r"\d+(?:\.\d+)*", build_version) + if not cmake_version: + # A version file claiming 0 would satisfy every version request, which is worse than + # not building at all. + raise RuntimeError( + f"cannot derive a numeric CMake version from {build_version!r}; the version file " + "would claim 0 and satisfy every version request" + ) + contents = contents.replace("@EXECUTORCH_VERSION@", cmake_version.group(0)) + contents = contents.replace("@EXECUTORCH_BUILD_VERSION@", build_version) + # CMake only reads a version file that sits beside the configuration file it found, so this + # goes to both locations the configuration is installed to. Writing it to one would leave a + # version request silently unchecked when the other location was used. + for destination in ( + os.path.join(dst_root, "share", "cmake", "executorch-config-version.cmake"), + os.path.join( + dst_root, + "lib", + "cmake", + "executorch", + "executorch-config-version.cmake", + ), + ): + self.mkpath(os.path.dirname(destination)) + with open(destination, "w") as handle: + handle.write(contents) + + +def _headers_to_install(entry: Path): + """The headers a copy list entry publishes, skipping any a consumer could not or should not use. + + A directory entry publishes everything under it, and a file entry publishes just that file. A header a + consumer cannot compile is worse than an absent one, because the failure lands in their project rather + than here, and the directory entries sweep in a few of those. + + Test directories are skipped as a whole rather than by name. They hold mocks and stubs for this + project's own tests, nothing the wheel installs includes them, and a consumer linking a mock allocator + or a stub platform would get behaviour no release intends. Matched on any part starting with "test", so + a directory named testing_util counts too, which a plain equality check missed. + """ + candidates = entry.rglob("*.h") if entry.is_dir() else [entry] + return [ + src + for src in candidates + if not _is_unshippable_header(src) + and not any(part.startswith("test") for part in src.parts[:-1]) + ] + + +def _is_unshippable_header(src: Path) -> bool: + """Whether a header is on the list of ones a consumer of the wheel could not compile. + + Compared on the path ending rather than the file name. Two headers here are both called + tensor_util.h, one a utility that many shipped headers include and one a test helper, so matching the + bare name either kept the helper or removed the utility everything needs. + """ + posix = src.as_posix() + return any(posix.endswith(entry) for entry in _UNSHIPPABLE_HEADERS) + class Buck2EnvironmentFixer(contextlib.AbstractContextManager): """Removes HOME from the environment when running as root. diff --git a/tools/cmake/executorch-wheel-config-version.cmake.in b/tools/cmake/executorch-wheel-config-version.cmake.in new file mode 100644 index 00000000000..1c3f8640fa9 --- /dev/null +++ b/tools/cmake/executorch-wheel-config-version.cmake.in @@ -0,0 +1,49 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +# Version file for the wheel's CMake package, so `find_package(executorch 1.2)` +# answers correctly instead of matching any version at all. +# +# Written by packaging rather than checked in, because the version is only known +# when the wheel is built: version.txt gives the base, and BUILD_VERSION +# overrides it for a nightly. A checked-in file would go stale the first time +# either changed. +# +# The same shape as the torch wheel's TorchConfigVersion.cmake, which is the +# file a consumer of this ecosystem will already have met. +set(PACKAGE_VERSION "@EXECUTORCH_VERSION@") + +# The same version again, unabridged. find_package compares dotted integers +# only, so the version above is read as its numeric release part and a consumer +# cannot pin a nightly or a specific build through it: passing the full string +# to find_package is a hard argument error. This variable is what a consumer +# compares when an exact build pairing is required. +set(EXECUTORCH_BUILD_VERSION "@EXECUTORCH_BUILD_VERSION@") + +# Any version at least as new as the one requested is compatible. ExecuTorch has +# no stable ABI promise across majors yet, so this is deliberately permissive; +# when it does, this is where the rule changes. +if("${PACKAGE_VERSION}" VERSION_LESS "${PACKAGE_FIND_VERSION}") + set(PACKAGE_VERSION_COMPATIBLE FALSE) +else() + set(PACKAGE_VERSION_COMPATIBLE TRUE) + if("${PACKAGE_VERSION}" VERSION_EQUAL "${PACKAGE_FIND_VERSION}") + set(PACKAGE_VERSION_EXACT TRUE) + endif() +endif() + +# A range request names an upper bound, and without this branch the check above +# sees only the lower one, so find_package(executorch 1.0...<2.0) accepted 3.0. +# CMake sets these variables only for a range, so a plain request is unaffected. +if(PACKAGE_FIND_VERSION_RANGE AND PACKAGE_VERSION_COMPATIBLE) + if(PACKAGE_FIND_VERSION_RANGE_MAX STREQUAL "INCLUDE") + if("${PACKAGE_VERSION}" VERSION_GREATER "${PACKAGE_FIND_VERSION_MAX}") + set(PACKAGE_VERSION_COMPATIBLE FALSE) + endif() + elseif(NOT "${PACKAGE_VERSION}" VERSION_LESS "${PACKAGE_FIND_VERSION_MAX}") + set(PACKAGE_VERSION_COMPATIBLE FALSE) + endif() +endif() diff --git a/tools/cmake/executorch-wheel-config.cmake b/tools/cmake/executorch-wheel-config.cmake index fa789771872..c337bada296 100644 --- a/tools/cmake/executorch-wheel-config.cmake +++ b/tools/cmake/executorch-wheel-config.cmake @@ -8,7 +8,21 @@ # for this file and find ExecuTorch package if it is installed. Typical usage # is: # +# ~~~ # find_package(executorch REQUIRED) +# target_link_libraries(my_app PRIVATE executorch::runtime) +# ~~~ +# +# This is the wheel's own contract, written by hand rather than generated, +# because the wheel copies build products out of the build tree instead of +# running an install step. +# +# It is NOT identical to the in-tree package config. That one exposes the +# build's own bare target names, such as executorch and xnnpack_backend, while +# this one exposes namespaced imported targets like executorch::runtime, because +# a wheel consumer links prebuilt files rather than participating in the build. +# Consumer code written against one therefore does not configure against the +# other. # ------- # # Finds the ExecuTorch library @@ -16,13 +30,511 @@ # This will define the following variables: # # EXECUTORCH_FOUND -- True if the system has the ExecuTorch library +# # EXECUTORCH_INCLUDE_DIRS -- The include directories for ExecuTorch -# EXECUTORCH_LIBRARIES -- Libraries to link against # +# EXECUTORCH_COMPILE_DEFINITIONS -- Definitions a consumer must compile with. +# The vendored c10 headers otherwise reach for a header generated inside a +# PyTorch build, which no wheel can carry. +# +# EXECUTORCH_CXX_STANDARD -- The minimum C++ standard the shipped headers need. +# A consumer using these variables has to set it, because a compiler defaulting +# to an older standard cannot parse them. +# +# EXECUTORCH_LIBRARIES -- Libraries to link against: the prebuilt runtime and +# the components the wheel shipped, except the ones documented below as opt in. +# Not the Python extension, which carries unresolved interpreter symbols that +# only resolve inside an interpreter, so a standalone application linking it +# fails with a page of PyUnicode_InternFromString errors. A project building a +# custom operator against the extension asks for the _portable_lib target by +# name, which is the long-standing contract for that and also carries the C++20 +# requirement PyTorch's headers need. +# +# EXECUTORCH_BUILD_VERSION -- The full version this package was built from, +# including any prerelease suffix and local version label. Compare this when an +# exact build pairing is required, since the CMake package version keeps only +# the numeric part. +# +# and, when the prebuilt shared runtime is present, the imported target: +# +# executorch::runtime -- The prebuilt C++ runtime (libexecutorch.so). Loads +# and executes a program, and carries only primitive operators rather than the +# kernels a model computes with, so running a model needs a kernel component as +# well. +# +# Component targets are defined only when the wheel ships that component, so the +# set depends on which wheel is installed. Each one carries the runtime +# dependency and, for a registration-only library, the link options that keep it +# from being dropped. The names, when present, are: +# +# executorch::kernels_optimized -- The CPU operator kernels. Needed to run a +# model. executorch::backend_xnnpack -- The XNNPACK delegate. +# executorch::threadpool -- The shared thread pool. executorch::etdump -- +# The profiler. +# +# Check with if(TARGET executorch::) rather than assuming one exists. A +# namespaced name that was never defined is a configure-time error that names +# the component, so a consumer who links one unconditionally gets a clear +# failure rather than a broken build. Guarding is still worth doing, because a +# component's absence is a legitimate state: a CPU-only wheel ships no +# accelerator delegate, and a consumer that guards adapts instead of failing. +# +# The floor stays where it was, so a consumer that only wants the long-standing +# variables and the prebuilt Python extension keeps working on the CMake it +# already has. The shared-runtime targets below need more than this and check +# for it themselves. cmake_minimum_required(VERSION 3.19) -# Find prebuilt _portable_lib..so. This file should be installed -# under /executorch/share/cmake +# The imported targets below export "$ORIGIN"-relative runtime paths as link +# options, and CMake writes that token incorrectly before 3.28. Measured rather +# than inferred, on a consumer linking such a target: +# +# ~~~ +# 3.24.3, 3.27.9 Makefiles double the dollar sign, Ninja drops the name +# 3.28.4, 3.31.8 both write the token correctly +# ~~~ +# +# Either broken form leaves a consumer building and running in place, because +# the absolute package directory is also recorded, then failing once it is +# deployed somewhere else. Silently defining a target that behaves that way is +# worse than not defining it, so the targets are skipped and a consumer that +# asked for one gets a message naming the reason. +if(CMAKE_VERSION VERSION_LESS 3.28) + set(_executorch_targets_supported FALSE) +else() + set(_executorch_targets_supported TRUE) +endif() + +# Everything is resolved relative to this file so the wheel stays relocatable: +# no absolute path from the machine that built it is baked in here. The file is +# installed both under share/cmake, which the historical contract uses, and +# under lib/cmake/executorch, which a plain CMAKE_PREFIX_PATH pointed at the +# package root can discover, so the root is located by a marker rather than a +# fixed depth. +# +# Tested directly rather than through find_path. The root is a known relative +# offset from this file, so a search adds nothing, and find_path applies the +# consumer's find-root rules: under a cross-compiling toolchain that sets +# CMAKE_FIND_ROOT_PATH_MODE_INCLUDE to ONLY it reroots these absolute paths into +# the target sysroot, finds nothing, and reports a complete package as missing. +set(_executorch_package_root "") +foreach(_candidate + "${CMAKE_CURRENT_LIST_DIR}/.." "${CMAKE_CURRENT_LIST_DIR}/../.." + "${CMAKE_CURRENT_LIST_DIR}/../../.." +) + # share/cmake identifies the package root and only exists there. A generic + # marker such as include/executorch can also appear one level down, in which + # case the search from lib/cmake/executorch would stop at lib/ and resolve the + # wrong root. + if(EXISTS "${_candidate}/share/cmake/executorch-config.cmake") + set(_executorch_package_root "${_candidate}") + break() + endif() +endforeach() + +# Normalise the result before it is used to build paths. The search can return a +# directory with a trailing separator, which then appears doubled in every path +# derived from it and in the message reporting where the runtime was found. +if(_executorch_package_root) + string(REGEX REPLACE "/+$" "" _executorch_package_root + "${_executorch_package_root}" + ) +endif() + +# Both directories are needed for a usable package. The C10 compatibility +# headers are not optional: core headers such as runtime/core/array_ref.h +# include c10 unconditionally, so a package missing them cannot compile anything +# that touches the runtime API. +# +# A missing directory is reported as not-found rather than raised here, so an +# optional find_package gets a FALSE answer instead of a dead build. The +# REQUIRED handling at the bottom of this file turns it into an error when the +# caller asked for one. +set(_executorch_c10_include + "${_executorch_package_root}/include/executorch/runtime/core/portable_type/c10" +) +# The full version this package was built from. It lives in the generated +# version file, which CMake includes in a throwaway scope while deciding whether +# the package is acceptable, so nothing assigned there reaches a consumer. +# Reading that file here, from the config, is what makes the value visible. Both +# files are installed side by side, so the path is fixed relative to this one. +set(_executorch_version_file + "${CMAKE_CURRENT_LIST_DIR}/executorch-config-version.cmake" +) +if(EXISTS "${_executorch_version_file}") + file(STRINGS "${_executorch_version_file}" _executorch_version_lines + REGEX "^set\\(EXECUTORCH_BUILD_VERSION" + ) + foreach(_line IN LISTS _executorch_version_lines) + if(_line MATCHES "\"([^\"]+)\"") + set(EXECUTORCH_BUILD_VERSION "${CMAKE_MATCH_1}") + endif() + endforeach() +endif() +unset(_executorch_version_file) + +set(EXECUTORCH_INCLUDE_DIRS "${_executorch_package_root}/include" + "${_executorch_c10_include}" +) +# The same definition the imported targets carry. A consumer on CMake older than +# 3.28 gets no imported targets and links through EXECUTORCH_LIBRARIES instead, +# and without this it could not compile at all: the vendored c10 headers reach +# for a header generated inside a PyTorch build that no wheel can carry. +set(EXECUTORCH_COMPILE_DEFINITIONS C10_USING_CUSTOM_GENERATED_MACROS) +# The standard the imported targets require as a compile feature. Exported as a +# variable too, because a consumer on CMake older than 3.28 gets no imported +# targets and would otherwise compile these headers with whatever its compiler +# defaults to. +set(EXECUTORCH_CXX_STANDARD 17) +foreach(_required_include ${EXECUTORCH_INCLUDE_DIRS}) + if(NOT EXISTS "${_required_include}") + message( + STATUS "ExecuTorch package at ${_executorch_package_root} is missing " + "${_required_include}, so nothing can compile against it." + ) + set(EXECUTORCH_INCLUDE_DIRS) + set(EXECUTORCH_COMPILE_DEFINITIONS) + set(EXECUTORCH_CXX_STANDARD) + set(EXECUTORCH_LIBRARIES) + set(EXECUTORCH_FOUND OFF) + set(executorch_FOUND FALSE) + return() + endif() +endforeach() + +set(EXECUTORCH_LIBRARIES) +set(EXECUTORCH_FOUND OFF) + +# Locate one shipped library by base name. +# +# Sets to the full path, or to an empty string when the wheel does not +# carry that library. +# +# A wheel ships plain, unversioned names. That is deliberate: the library and +# the only things that link it ship in the same archive and are replaced +# together, so there is no upgrade during which two majors must coexist, and a +# versioned name would actively hurt because find_library(executorch) matches +# libexecutorch.so and not libexecutorch.so.1. The torch wheel ships unversioned +# names for the same reason. +# +# The versioned pattern is matched anyway, at no cost, so that a package +# assembled by other means than the wheel build, where the SONAME policy does +# apply and file names end in a major, still resolves. Highest major wins there +# rather than whichever sorts first. +function(_executorch_find_library _output _base_name) + set(${_output} + "" + PARENT_SCOPE + ) + file(GLOB _matches "${_executorch_package_root}/lib/${_base_name}.so" + "${_executorch_package_root}/lib/${_base_name}.so.*" + ) + list(LENGTH _matches _count) + if(_count EQUAL 0) + return() + endif() + # Highest major wins, so a package that somehow carries two does not silently + # select by string order. Natural ordering keeps .2 below .10. + list( + SORT _matches + COMPARE NATURAL + ORDER DESCENDING + ) + list(GET _matches 0 _selected) + set(${_output} + "${_selected}" + PARENT_SCOPE + ) +endfunction() + +# The prebuilt runtime. +_executorch_find_library(_executorch_runtime_library libexecutorch) +if(_executorch_runtime_library AND NOT _executorch_targets_supported) + # The imported targets are skipped, but the libraries themselves are present + # and linkable by path, so the long-standing variables are still honoured. + # Leaving them empty here contradicted the message below and made a REQUIRED + # find_package fail on a package that carries a working runtime. + # + # The kernel libraries are listed too, not only the runtime. The runtime + # carries only primitive operators, not the kernels a model computes with, so + # a consumer given the runtime alone links successfully and then fails at run + # time with "Missing operator: aten::mul.out", which reads as a model problem + # rather than a missing library. + # + # Each one is wrapped in scoped retention, because a registration-only library + # exports nothing the application references and the linker discards it under + # --as-needed. Measured: without this the library reaches the link line and is + # absent from the binary's dependencies, so the registrations never run. The + # imported targets express the same thing through link options, which this + # older-CMake route cannot use. + set(EXECUTORCH_FOUND ON) + list(APPEND EXECUTORCH_LIBRARIES "${_executorch_runtime_library}") + # Every shipped library, not only the kernels. A delegate registers itself + # from a static initializer, so leaving one out gave a clean configure and + # then a load failure saying the backend is not registered, which reads as a + # model problem. Anything the wheel did not ship is simply not found and + # skipped. + foreach(_executorch_component IN + ITEMS libexecutorch_kernels_optimized libexecutorch_backend_xnnpack + libexecutorch_threadpool libexecutorch_etdump + ) + _executorch_find_library( + _executorch_component_library "${_executorch_component}" + ) + if(_executorch_component_library) + # The retention option only on the platform whose linker has it. Elsewhere + # the plain path is still correct, it just leaves a registration-only + # library subject to being dropped, which is the same position a source + # build is in there. + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + list( + APPEND + EXECUTORCH_LIBRARIES + "-Wl,--push-state,--no-as-needed,${_executorch_component_library},--pop-state" + ) + else() + list(APPEND EXECUTORCH_LIBRARIES "${_executorch_component_library}") + endif() + # The switch a source build sets. The runtime header uses it to pick + # between an extern declaration (which resolves in the thread pool + # library) and a local inline serial fallback. On the imported-target + # route below it lives on the runtime target's + # INTERFACE_COMPILE_DEFINITIONS; here it goes on + # EXECUTORCH_COMPILE_DEFINITIONS, which is what the documented pre-3.28 + # recipe tells a consumer to apply. + if(_executorch_component STREQUAL "libexecutorch_threadpool") + list(APPEND EXECUTORCH_COMPILE_DEFINITIONS ET_USE_THREADPOOL) + endif() + endif() + endforeach() + unset(_executorch_component_library) + message( + STATUS + "executorch: the prebuilt runtime is present but its imported targets need CMake 3.28 or " + "newer, because older versions write the \$ORIGIN token in a runtime search path " + "incorrectly. EXECUTORCH_LIBRARIES carries the runtime and every shipped component by path " + "instead. Linking it is not sufficient on its own: an imported target would also carry the " + "include directories, the compile definitions and the C++ standard, so here a consumer has " + "to apply EXECUTORCH_INCLUDE_DIRS, EXECUTORCH_COMPILE_DEFINITIONS and " + "EXECUTORCH_CXX_STANDARD itself. The prebuilt Python extension is unaffected." + ) +elseif(_executorch_runtime_library) + set(EXECUTORCH_FOUND ON) + message(STATUS "ExecuTorch runtime found at ${_executorch_runtime_library}") + + # The documented contract is that a consumer can link ${EXECUTORCH_LIBRARIES}. + # Leaving it empty here would make find_package succeed while offering nothing + # linkable to anyone who has not moved to the imported target. + list(APPEND EXECUTORCH_LIBRARIES executorch::runtime) + + # This file can be processed more than once in a single configure, for example + # when several subprojects each call find_package(executorch). Creating the + # target twice is an error, so only define it once and set the properties + # either way. + if(TARGET executorch::runtime) + # This file ran already in the same configure, because another subproject + # also called find_package. Redefining the target would be an error, so keep + # the one that is already there. + message( + STATUS "executorch: executorch::runtime is already defined, reusing it" + ) + else() + add_library(executorch::runtime SHARED IMPORTED) + set_target_properties( + executorch::runtime + PROPERTIES IMPORTED_LOCATION "${_executorch_runtime_library}" + INTERFACE_INCLUDE_DIRECTORIES "${EXECUTORCH_INCLUDE_DIRS}" + INTERFACE_COMPILE_FEATURES cxx_std_17 + INTERFACE_COMPILE_DEFINITIONS + C10_USING_CUSTOM_GENERATED_MACROS + ) + # Consumers get the wheel's lib/ directory in their RUNPATH automatically, + # because CMake adds the imported library's directory. Also record + # $ORIGIN-relative entries so an application deployed next to a copy of the + # runtime keeps working without relinking or LD_LIBRARY_PATH. $ORIGIN is a + # loader token, so it belongs only in RUNPATH, never in IMPORTED_LOCATION. + # + # $ORIGIN is named before the wheel's own directory. An application deployed + # beside a copy of the runtime has to find that copy, and the loader takes + # the first match, so putting the install directory first would keep sending + # a relocated application back to the original wheel for as long as it + # remains installed. That also makes a relocation test that deletes the + # original pass for the wrong reason. + # + # The cost, measured rather than assumed: a library that merely shares this + # SONAME and sits in the application's own directory will win. That is what + # $ORIGIN means in every package that uses it, and a package cannot offer + # relocation while also refusing to honour what the user placed beside their + # binary. The consequence worth worrying about, a delegate pairing with a + # different registry, is caught directly by the single-registry checks, + # which inspect what the shipped libraries define instead of trusting the + # loader's choice. + # + # The package's own library directory has to be added explicitly, as an + # option rather than by relying on CMake. CMake does put it in the + # consumer's search path because the imported library is named by absolute + # path, but it records that as a link-path rpath, and install(TARGETS) + # strips exactly those. Measured: a consumer runs from its build tree and + # then fails with exit 127 and two libraries not found once installed, while + # the $ORIGIN entries survive because they are set here as explicit options. + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + # $ORIGIN-relative entries so a relocated application keeps working. The + # package's own directory does not need to be added here: CMake already + # puts it in the consumer's runtime search path because the imported + # library is named by absolute path, which is also what makes an + # application built against the installed package start at all. + # + # --enable-new-dtags asks for DT_RUNPATH. Without it this linker writes + # the older DT_RPATH, which the loader searches BEFORE LD_LIBRARY_PATH and + # applies to a dependency's dependencies too, so a consumer could not + # point an instrumented or locally built runtime at their application. + # Measured on GNU ld 2.35: -rpath alone produces DT_RPATH, and with this + # flag the same link produces DT_RUNPATH. + set_property( + TARGET executorch::runtime + APPEND + PROPERTY INTERFACE_LINK_OPTIONS "LINKER:--enable-new-dtags" + "LINKER:-rpath,$ORIGIN" "LINKER:-rpath,$ORIGIN/../lib" + "LINKER:-rpath,${_executorch_package_root}/lib" + ) + endif() + endif() +endif() + +# Define an imported target for one shipped component library. +# +# A component is a prebuilt shared library next to the runtime, such as the CPU +# kernels or a delegate backend. Without a target for each one, a consumer has +# to find the file itself and decide how to keep it on the link line, which +# means depending on the wheel's private layout. The retention part matters +# most: a registration-only library has no symbol the application references, so +# a normal link drops it and its registration never runs. +# +# Call as: _executorch_define_component( ) +function(_executorch_define_component _suffix _library_name) + # Same reason the runtime target is skipped on older CMake: a component target + # exports an $ORIGIN-relative search path, and a version that writes it wrong + # produces a target that works in place and fails once deployed. + if(NOT _executorch_targets_supported) + return() + endif() + _executorch_find_library(_library "lib${_library_name}") + if(NOT _library) + return() + endif() + + set(_target "executorch::${_suffix}") + if(TARGET ${_target}) + # This file ran already in the same configure, because another subproject + # also called find_package. Redefining the target would be an error, so keep + # the one that is already there. + message(STATUS "executorch: ${_target} is already defined, reusing it") + # Still advertise it. This file runs again whenever another subproject calls + # find_package, and that run starts from an empty EXECUTORCH_LIBRARIES, so + # returning here would hand the second caller a list with the runtime but + # none of the components. A consumer linking that variable would then be + # missing its kernels and fail at load with an unregistered operator. + set(EXECUTORCH_LIBRARIES + ${EXECUTORCH_LIBRARIES} ${_target} + PARENT_SCOPE + ) + return() + endif() + add_library(${_target} SHARED IMPORTED) + set_target_properties( + ${_target} + PROPERTIES IMPORTED_LOCATION "${_library}" + INTERFACE_INCLUDE_DIRECTORIES "${EXECUTORCH_INCLUDE_DIRS}" + INTERFACE_COMPILE_FEATURES cxx_std_17 + INTERFACE_COMPILE_DEFINITIONS C10_USING_CUSTOM_GENERATED_MACROS + ) + # Every component resolves the runtime from the same shared library, so record + # that rather than leaving a consumer to link both by hand. + if(TARGET executorch::runtime) + set_property( + TARGET ${_target} + APPEND + PROPERTY INTERFACE_LINK_LIBRARIES executorch::runtime + ) + endif() + # Guarded on Linux because these are GNU linker options. A wheel only ships + # these components on Linux, so a consumer configured for another system is + # either cross-compiling from the wrong package or has nothing to retain. + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set_property( + TARGET ${_target} + APPEND + PROPERTY INTERFACE_LINK_OPTIONS + # DT_RUNPATH rather than the older DT_RPATH, for the reason given + # on the runtime target: DT_RPATH outranks LD_LIBRARY_PATH and + # applies transitively, so it would stop a consumer overriding a + # packaged library. + "LINKER:--enable-new-dtags" + "LINKER:-rpath,$ORIGIN" + "LINKER:-rpath,$ORIGIN/../lib" + # One option per component rather than a shared push-state pair: + # CMake removes duplicate link options, so repeating the same + # push-state text for a second component silently drops its + # scoping and the library goes back to being --as-needed. Naming + # the library inside the same option keeps each one distinct. + # + # --no-as-needed applies only to what follows within the pushed + # state, so the pop restores whatever the consumer had. + "LINKER:--push-state,--no-as-needed,${_library},--pop-state" + ) + endif() + set(EXECUTORCH_LIBRARIES + ${EXECUTORCH_LIBRARIES} ${_target} + PARENT_SCOPE + ) +endfunction() + +_executorch_define_component(threadpool executorch_threadpool) + +# The merged CPU kernels. Documented as a component and asserted by the release +# checks, so it has to be defined here or a consumer following the documentation +# gets a bare name that CMake hands to the linker as a literal flag. +_executorch_define_component(kernels_optimized executorch_kernels_optimized) +# The profiler. A C++ application could not record timing data from an installed +# package before, because the implementation shipped only inside the Python +# extension. +_executorch_define_component(etdump executorch_etdump) + +# The switch a source build sets, on the runtime rather than on the thread pool +# target. The guarded declaration lives in a runtime header that every component +# exposes, and it selects between an extern declaration and a local inline +# definition. Putting it on the thread pool alone means a consumer with one +# translation unit linking that component and another linking only the kernels +# compiles two different definitions of the same function into one program, and +# the serial one silently wins wherever it was inlined. +# +# Only when the thread pool actually shipped. Packaging gates that library on +# the build flags it needs, so a wheel built without them has no thread pool, +# and switching the declaration on there would leave a consumer calling a +# function nothing defines. +if(TARGET executorch::runtime AND TARGET executorch::threadpool) + set_property( + TARGET executorch::runtime + APPEND + PROPERTY INTERFACE_COMPILE_DEFINITIONS ET_USE_THREADPOOL + ) + # The definition selects a declaration, and the thread pool library holds the + # only definition of what it declares, so a consumer linking just the runtime + # would fail to link. Carried on the runtime rather than left to the caller, + # since the caller cannot see which header a compile definition on an imported + # target switched. + set_property( + TARGET executorch::runtime + APPEND + PROPERTY INTERFACE_LINK_LIBRARIES executorch::threadpool + ) +endif() + +_executorch_define_component(backend_xnnpack executorch_backend_xnnpack) + +# Find prebuilt _portable_lib..so. This is the legacy contract used +# to build custom-op extensions against the Python module, and is kept working +# independently of the runtime target above. # Find python if(DEFINED ENV{CONDA_DEFAULT_ENV} AND NOT $ENV{CONDA_DEFAULT_ENV} STREQUAL @@ -45,6 +557,20 @@ execute_process( if(SYSCONFIG_RESULT EQUAL 0) message(STATUS "Sysconfig extension suffix: ${EXT_SUFFIX}") +elseif(_executorch_runtime_library) # Tested on the located library rather than + # the imported target, because the + # targets are not defined below the CMake version they need, and a C++ + # application on an older CMake is exactly the case this branch exists to keep + # working. A C++ application linking only the shared runtime does not need + # Python at all, so a missing interpreter must not fail its configure. Skip + # locating the Python extension instead; the legacy _portable_lib target is + # simply not offered in that case. + message( + STATUS + "Python not usable, skipping the Python extension: ${SYSCONFIG_ERROR}" + ) + set(EXT_SUFFIX "") + set(_portable_lib_LIBRARY "") else() message( FATAL_ERROR @@ -52,53 +578,98 @@ else() ) endif() -find_library( - _portable_lib_LIBRARY - NAMES _portable_lib${EXT_SUFFIX} - PATHS "${CMAKE_CURRENT_LIST_DIR}/../../extension/pybindings/" -) +if(EXT_SUFFIX) + # Tested directly rather than through find_library, for the same reason as the + # package root: the path and the file name are both already known, so a search + # only adds the consumer's find-root rules, which reroot an absolute wheel + # path into a cross-compile sysroot and report a present extension as missing. + set(_portable_lib_candidate + "${_executorch_package_root}/extension/pybindings/_portable_lib${EXT_SUFFIX}" + ) + if(EXISTS "${_portable_lib_candidate}") + set(_portable_lib_LIBRARY "${_portable_lib_candidate}") + else() + set(_portable_lib_LIBRARY "") + endif() +endif() + +if(NOT _portable_lib_LIBRARY) + # The interpreter that answered above is whichever python3 is on PATH, which + # is not necessarily the one this wheel was built for. A cp310 wheel inspected + # by a 3.12 interpreter yields a suffix that names no file here, and the + # package then reported itself as not found on a complete install. The shipped + # extension carries its own suffix in its name, so take it from the package. + file(GLOB _portable_lib_matches + "${_executorch_package_root}/extension/pybindings/_portable_lib.*" + ) + foreach(_candidate IN LISTS _portable_lib_matches) + if(_candidate MATCHES "\\.(so|pyd|dylib)$") + set(_portable_lib_LIBRARY "${_candidate}") + break() + endif() + endforeach() + unset(_portable_lib_matches) +endif() -set(EXECUTORCH_LIBRARIES) -set(EXECUTORCH_FOUND OFF) if(_portable_lib_LIBRARY) set(EXECUTORCH_FOUND ON) message( STATUS "ExecuTorch portable library is found at ${_portable_lib_LIBRARY}" ) - list(APPEND EXECUTORCH_LIBRARIES _portable_lib) - add_library(_portable_lib STATIC IMPORTED) - set(EXECUTORCH_INCLUDE_DIRS ${CMAKE_CURRENT_LIST_DIR}/../../include) + # Only when nothing else is linkable, which is the fused layout: a macOS wheel + # ships this extension and no separate runtime library, so the appends above + # never ran and this is the only thing there is to offer. On a split wheel the + # runtime is already there and this is skipped, because the extension carries + # unresolved interpreter symbols and a plain C++ application that links it + # fails with a page of PyUnicode_InternFromString style errors. + # + # Measured both layouts: split gives the runtime and its components, fused + # gives _portable_lib. Callers who specifically want the extension, such as a + # custom operator project, ask for the target by name rather than relying on + # this, and that target carries the C++20 requirement PyTorch's headers need + # while the runtime components require C++17. + if(NOT EXECUTORCH_LIBRARIES) + list(APPEND EXECUTORCH_LIBRARIES _portable_lib) + endif() + if(TARGET _portable_lib) + # This file ran already in the same configure, because another subproject + # called find_package too. No in-tree target uses this name, so it can only + # be the imported one defined below, and re-setting its properties to the + # same values is harmless. + message(STATUS "executorch: _portable_lib is already defined, reusing it") + else() + add_library(_portable_lib STATIC IMPORTED) + endif() + # PyTorch requires C++20, so pybindings must be compiled with C++20. set_target_properties( _portable_lib PROPERTIES IMPORTED_LOCATION "${_portable_lib_LIBRARY}" INTERFACE_INCLUDE_DIRECTORIES "${EXECUTORCH_INCLUDE_DIRS}" - # PyTorch requires C++20, so anything linking this must compile - # as - # C++20. An interface requirement rather than CXX_STANDARD, - # because - # an imported target compiles nothing itself and CXX_STANDARD - # does - # not reach consumers, so a custom-op build could still compile - # as - # C++17 and fail against headers that need C++20. + # An interface requirement rather than CXX_STANDARD: an imported + # target compiles nothing itself, and CXX_STANDARD does not reach + # consumers, so a custom-op build linking this could still + # compile + # as C++17 and fail against headers that need C++20. INTERFACE_COMPILE_FEATURES cxx_std_20 + # The same definition the runtime target carries. A custom-op + # build that links only this target still compiles against the + # same headers and needs it too. + INTERFACE_COMPILE_DEFINITIONS C10_USING_CUSTOM_GENERATED_MACROS ) - # The extension links the runtime rather than containing it, so it no longer # satisfies the runtime symbols a custom-op library references. Put the # shipped runtime on this target's interface, which is where the definitions # moved to, so an out-of-tree operator project keeps building and loading - # against the extension exactly as it did before. - find_library( - EXECUTORCH_RUNTIME_LIBRARY executorch - PATHS "${CMAKE_CURRENT_LIST_DIR}/../../lib" - NO_DEFAULT_PATH - ) - if(EXECUTORCH_RUNTIME_LIBRARY) + # against the extension exactly as it did before. Without this a custom + # operator links and then fails to load with an undefined runtime symbol. + # + # The file path rather than executorch::runtime, because that target is only + # defined on CMake 3.28 or newer while this one has no such requirement. + if(_executorch_runtime_library) set_property( TARGET _portable_lib APPEND - PROPERTY INTERFACE_LINK_LIBRARIES "${EXECUTORCH_RUNTIME_LIBRARY}" + PROPERTY INTERFACE_LINK_LIBRARIES "${_executorch_runtime_library}" ) # CMake adds a linked library's directory to the consumer's build tree # runtime search path and strips it on install, so an installed consumer @@ -108,7 +679,7 @@ if(_portable_lib_LIBRARY) # package location into a library the consumer ships onward. A consumer that # installs elsewhere adds these to its own INSTALL_RPATH. get_filename_component( - EXECUTORCH_RUNTIME_LIBRARY_DIR "${EXECUTORCH_RUNTIME_LIBRARY}" DIRECTORY + EXECUTORCH_RUNTIME_LIBRARY_DIR "${_executorch_runtime_library}" DIRECTORY ) get_filename_component( EXECUTORCH_PYTHON_EXTENSION_DIR "${_portable_lib_LIBRARY}" DIRECTORY @@ -118,13 +689,61 @@ endif() # find_package checks _FOUND, which is case-sensitive and does not # match the EXECUTORCH_FOUND spelling this file documents. Without this, a -# REQUIRED find_package succeeds even when nothing usable was located, and the -# consumer goes on to link nothing. +# REQUIRED find_package would succeed even when nothing usable was located. set(executorch_FOUND ${EXECUTORCH_FOUND}) if(NOT executorch_FOUND AND executorch_FIND_REQUIRED) message( FATAL_ERROR - "Found the ExecuTorch package but could not locate the Python extension " - "inside it, so there is nothing to link." + "Found the ExecuTorch package but neither the shared runtime nor the Python " + "extension could be located inside it." ) endif() + +# Component requests are answered from the targets that were actually defined +# above, so a consumer asking for a component this wheel does not ship gets told +# at configure time rather than at link or load time. Without this a REQUIRED +# request for a missing component, or for a name that does not exist at all, +# would configure and then fail much later. +# +# The check is written out rather than using check_required_components, which +# comes from a module a package config cannot assume is already included. +foreach(_component ${executorch_FIND_COMPONENTS}) + if(TARGET executorch::${_component}) + set(executorch_${_component}_FOUND TRUE) + else() + set(executorch_${_component}_FOUND FALSE) + if(executorch_FIND_REQUIRED_${_component}) + set(executorch_FOUND FALSE) + # Naming the CMake version when that is the cause saves a consumer from + # concluding the component is missing from the package, which is the wrong + # thing to go looking for. + if(NOT _executorch_targets_supported) + # One string rather than several arguments. Several make a list, and + # message() joins a list with semicolons, which lands separators mid + # sentence. + string( + CONCAT + executorch_NOT_FOUND_MESSAGE + "the required component '${_component}' needs CMake 3.28 or newer, because older " + "versions write the \$ORIGIN token in a runtime search path incorrectly; this " + "package is otherwise usable through EXECUTORCH_LIBRARIES" + ) + else() + # One string rather than several arguments. Several make a list, and + # message() joins a list with semicolons, which lands separators mid + # sentence. + string( + CONCAT + executorch_NOT_FOUND_MESSAGE + "this ExecuTorch package does not provide the required component " + "'${_component}'. The installed wheel does not contain the library that " + "component wraps, either because it was not built with it or because this " + "platform's wheel ships the headers without the separate libraries" + ) + endif() + endif() + endif() +endforeach() +if(NOT executorch_FOUND AND executorch_FIND_REQUIRED) + message(FATAL_ERROR "${executorch_NOT_FOUND_MESSAGE}") +endif() From 8cc496f47cc4480377892d73e315710457985cdf Mon Sep 17 00:00:00 2001 From: shoumikhin Date: Tue, 11 Aug 2026 23:08:09 -0700 Subject: [PATCH 3/6] Ship the quantized kernels as their own library A quantized model uses smaller numbers than a normal one, so the tensors take less memory. Running one needs the quantized operator kernels. The only copy the wheel shipped is the one torch loads to export a model, which a C++ application cannot use. Such an application links the runtime, loads a quantized model, and the model fails at run time with a missing operator, which looks like a model problem rather than a packaging one. Build the quantized kernels as their own shared library and name it as a CMake component, the same way the other kernel sets are named. ```cmake find_package(executorch REQUIRED COMPONENTS kernels_quantized) target_link_libraries(my_app PRIVATE executorch::runtime executorch::kernels_quantized) ``` The wheel now ships `lib/libexecutorch_kernels_quantized.so`. Note that the wheel also ships a second copy of these kernels, inside the library torch loads when you export a model. That copy is built into the plugin rather than resolved from the shared library, so a process holding both registers the same operators twice, and the runtime treats that as fatal: ``` Re-registering quantized_decomposed::add.out ``` This affects only a process that does both, for example an application that embeds a Python interpreter. A plain C++ application can link the component freely. Because of that, this is the one component `EXECUTORCH_LIBRARIES` does not include, so an application that links whatever the package offers cannot end up in that position without asking. A consumer that wants the quantized kernels names the component, or on CMake older than 3.28, where no component targets exist, links `EXECUTORCH_QUANTIZED_KERNELS_LIBRARY` as well. That variable is now populated on both CMake routes, so a consumer that adopts the older-CMake recipe and later upgrades keeps the library on their link line instead of silently losing it. Built the wheel, installed it into a clean environment, and: - exported a quantized model and ran it from Python, matching eager PyTorch to within the quantization step (measured worst difference 0.0048 against a tolerance of 0.02). - built a C++ application that links `executorch::kernels_quantized`, ran the same program, and got the same output as Python, byte for byte. - confirmed the Python extension does not depend on the run-time copy, and that a process holding the shipped library and the export plugin aborts in either load order. - checked every shipped library the same way, to establish that this is the only pair that collides: the CPU kernels, the delegate, the thread pool, the profiler and the runtime all coexist with both the extension and the export plugin. - an application linking only `EXECUTORCH_LIBRARIES` does not depend on the quantized library while still depending on the CPU kernels, on CMake 3.28 and on real CMake 3.24. A new check asserts this, and it fails on the previous behaviour. - `EXECUTORCH_QUANTIZED_KERNELS_LIBRARY` resolves to the shipped library on both the modern-CMake route (as the imported target) and the pre-3.28 route (as a file path). - a missing quantized library now fails the checks instead of skipping them. The preset that builds the wheel enables these kernels unconditionally, so their absence is a regression rather than a configuration to tolerate, and both the ownership table and the C++ check previously treated it as an acceptable state and reported coverage they had not run. Ran on Linux x86_64 and aarch64. ghstack-source-id: d5aa85006811f5ff775cacd0c7c4ae943e228701 ghstack-comment-id: 5217087046 Pull-Request: https://github.com/pytorch/executorch/pull/21642 --- .ci/scripts/wheel/test_cpp_sdk.py | 169 ++++++++++++++++++++- .ci/scripts/wheel/test_shared_libraries.py | 109 ++++++++++++- docs/source/using-executorch-cpp.md | 27 +++- kernels/quantized/CMakeLists.txt | 21 ++- setup.py | 23 +++ tools/cmake/executorch-wheel-config.cmake | 116 +++++++++++--- 6 files changed, 429 insertions(+), 36 deletions(-) diff --git a/.ci/scripts/wheel/test_cpp_sdk.py b/.ci/scripts/wheel/test_cpp_sdk.py index 7b4aaf1a917..478554067de 100644 --- a/.ci/scripts/wheel/test_cpp_sdk.py +++ b/.ci/scripts/wheel/test_cpp_sdk.py @@ -57,6 +57,25 @@ def forward(self, x, image): with torch.no_grad(): expected = model(*example) +if mode == "quantized": + # Quantize with the same flow the documentation shows, so the exported program + # references the quantized operator set rather than the plain one. + # Importing this loads the ahead-of-time library, which is what registers the out + # variants of the quantized operators with torch. Without it the export fails with + # "Missing out variants: quantized_decomposed::quantize_per_tensor", because the + # lowering step has no out variant to select. + import executorch.kernels.quantized # noqa: F401 + from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import ( + get_symmetric_quantization_config, + XNNPACKQuantizer, + ) + from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e + + quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config()) + prepared = prepare_pt2e(torch.export.export(model, example).module(), quantizer) + prepared(*example) + model = convert_pt2e(prepared) + partitioners = [] if mode == "delegate": from executorch.backends.xnnpack.partition.xnnpack_partitioner import ( @@ -85,6 +104,11 @@ def forward(self, x, image): "expected": expected.flatten().tolist(), "delegated": mode == "delegate", "has_xnnpack": b"XnnpackBackend" in bytes(buffer), + # Whether the program actually carries quantized operators. The numeric comparison alone + # cannot tell: an unquantized export of the same model produces a closer match than the + # tolerance a quantized one needs, so it would pass while proving nothing about the + # quantized kernels. + "has_quantized": b"quantized_decomposed" in bytes(buffer), } ) ) @@ -102,6 +126,7 @@ def forward(self, x, image): #include #include +#include #include #include #include @@ -196,8 +221,14 @@ def forward(self, x, image): } worst = std::fmax(worst, diff); } - if (worst > 1e-4) { - std::printf("output differs from eager PyTorch by %g\n", worst); + // Passed in rather than fixed, because the acceptable difference depends on the + // model. A float32 model should match to within rounding, while an int8 quantized one + // legitimately differs by about one quantization step, and using the looser number + // for both would stop the float path catching a real regression. + const double tolerance = argc > 7 ? std::atof(argv[7]) : 1e-4; + if (worst > tolerance) { + std::printf( + "output differs from eager PyTorch by %g, tolerance %g\n", worst, tolerance); return 1; } @@ -335,8 +366,16 @@ def _build_consumer(work_dir: Path, name: str, components) -> Path: return consumer -def _run_consumer(consumer: Path, model: Path, reference, work_dir: Path) -> str: - """Run the application and require it to match eager PyTorch.""" +def _run_consumer( + consumer: Path, model: Path, reference, work_dir: Path, tolerance: float = 1e-4 +) -> str: + """Run the application and require it to match eager PyTorch within `tolerance`. + + The tolerance is a parameter because the acceptable difference depends on the model. + A float32 model should match to within rounding, while an int8 quantized one + legitimately differs by about one quantization step, and using the looser number for + both would stop the float path catching a real regression. + """ inputs = reference["inputs"] shape_a, data_a = _write_tensor(work_dir, "a", inputs[0]) shape_b, data_b = _write_tensor(work_dir, "b", inputs[1]) @@ -358,6 +397,7 @@ def _run_consumer(consumer: Path, model: Path, reference, work_dir: Path) -> str str(shape_b), str(data_b), str(expected), + str(tolerance), ], capture_output=True, text=True, @@ -1219,6 +1259,125 @@ def test_pre_3_28_route_builds_a_consumer_through_variables(work_dir: Path) -> N ) +def test_quantized_kernels_component_runs_a_model(work_dir: Path) -> None: + """A C++ application can run a quantized model using the shipped quantized kernels. + + Before the quantized kernels became their own library they existed only inside the + ahead-of-time extension beside the Python bindings, so a C++ application loading a + quantized program had nothing to link and failed at run time with the operators + reported missing. + + A missing library is a failure rather than a skip. The preset that builds the wheel + always enables the quantized kernels, so their absence is a regression in packaging + or in the build, not a configuration this suite has to tolerate. Skipping there + reported the whole check as coverage while running none of it. + """ + package_dir = _installed_package_dir() + # Globbed for the same reason the profiler check is: the library carries a version suffix outside a + # wheel build, and an exact name would skip this silently there rather than running it. + shipped = sorted((package_dir / "lib").glob("libexecutorch_kernels_quantized.so*")) + assert shipped, ( + "the wheel ships no quantized kernels library. The preset that builds it enables " + "them unconditionally, so this is a packaging or build regression rather than an " + "unsupported configuration." + ) + + model, reference = _export(work_dir, "quantized") + # The export has to have produced a quantized program, or the rest of this proves nothing about the + # quantized kernels. The numeric comparison cannot tell the difference: an unquantized export of the + # same model lands well inside the tolerance a quantized one needs, so it would pass while linking a + # library it never exercised. + assert reference["has_quantized"], ( + "the quantized export produced a program with no quantized operators, so this check would " + "prove nothing about the quantized kernels" + ) + consumer = _build_consumer( + work_dir, + "with-quantized", + ["runtime", "kernels_optimized", "kernels_quantized"], + ) + # One int8 quantization step over this model's output range is about 5e-3, so a + # float32 tolerance cannot be met by a correct quantized run. + output = _run_consumer(consumer, model, reference, work_dir, tolerance=2e-2) + print( + f"✓ a C++ app linking executorch::kernels_quantized runs a quantized model " + f"({output})" + ) + + +def test_aggregate_variable_excludes_the_quantized_kernels(work_dir: Path) -> None: + """`${EXECUTORCH_LIBRARIES}` must not drag in the quantized kernels. + + The export-time plugin that `executorch.kernels.quantized` loads carries its own + copy of those kernels rather than depending on the shipped library, so a process + holding both registers the same operators twice and the runtime stops on the + second one. An application that links whatever the package offers by default + would inherit that, so the component is defined but held out of the aggregate and + a consumer that wants it names it. + + Checked by reading the link line rather than by running, because the failure is a + process-wide abort that needs a Python interpreter in the same process to trigger. + What this owns is the packaging decision: is the library on the link line at all. + """ + package_dir = _installed_package_dir() + # Fatal for the same reason the check above is: the preset that builds the wheel + # always enables these kernels, so their absence is a regression rather than a + # configuration to tolerate, and skipping would report this as coverage. + assert sorted( + (package_dir / "lib").glob("libexecutorch_kernels_quantized.so*") + ), "the wheel ships no quantized kernels library, so this check cannot run" + + source_dir = work_dir / "aggregate-only" + source_dir.mkdir(parents=True, exist_ok=True) + (source_dir / "consumer.cpp").write_text(_CONSUMER_SOURCE) + # No COMPONENTS and no named target, which is the shape the older-CMake route + # forces and the documentation offers as the general case. + (source_dir / "CMakeLists.txt").write_text( + "cmake_minimum_required(VERSION 3.28)\n" + "project(consumer CXX)\n" + "find_package(executorch REQUIRED)\n" + "add_executable(consumer consumer.cpp)\n" + "target_link_libraries(consumer PRIVATE ${EXECUTORCH_LIBRARIES})\n" + ) + build_dir = work_dir / "aggregate-only-build" + config = package_dir / "share" / "cmake" / "executorch-config.cmake" + for command in ( + [ + _tool("cmake"), + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_PREFIX_PATH={config.parent}", + ], + [_tool("cmake"), "--build", str(build_dir)], + ): + result = subprocess.run(command, capture_output=True, text=True, check=False) + assert result.returncode == 0, ( + "an application linking only ${EXECUTORCH_LIBRARIES} could not be built:\n" + f"{result.stdout[-2000:]}\n{result.stderr[-2000:]}" + ) + + consumer = build_dir / "consumer" + dependencies = subprocess.run( + ["readelf", "-d", str(consumer)], capture_output=True, text=True, check=True + ).stdout + assert "libexecutorch_kernels_quantized" not in dependencies, ( + "an application that linked only ${EXECUTORCH_LIBRARIES} depends on the " + "quantized kernels. That library collides with the export-time plugin, so it " + "has to be opted into by name rather than handed to every consumer." + ) + # The rest of the aggregate still has to be there, or this would pass by shipping + # nothing at all. + assert "libexecutorch_kernels_optimized" in dependencies, ( + "the aggregate no longer carries the CPU kernels, so an application linking it " + "would fail at run time with the operators reported missing" + ) + print( + "✓ ${EXECUTORCH_LIBRARIES} carries the CPU kernels and not the quantized ones" + ) + + def run_tests(work_dir: Path) -> None: test_find_package_honours_a_version_request(work_dir) test_profiler_component_is_usable(work_dir) @@ -1228,6 +1387,8 @@ def run_tests(work_dir: Path) -> None: test_runtime_alone_links_but_cannot_compute(work_dir) test_kernels_component_runs_a_model(work_dir) test_pre_3_28_route_builds_a_consumer_through_variables(work_dir) + test_quantized_kernels_component_runs_a_model(work_dir) + test_aggregate_variable_excludes_the_quantized_kernels(work_dir) test_delegated_model_needs_the_delegate_component(work_dir) test_consumer_is_relocatable(work_dir) test_one_registry_in_the_cpp_process(work_dir) diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py index ad69e432862..76506429469 100644 --- a/.ci/scripts/wheel/test_shared_libraries.py +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -52,6 +52,14 @@ # the operators are registered twice, which aborts at startup. _KERNEL_SYMBOLS = ("torch::executor::native::abs_out",) +# The quantized kernels, whose own library the wheel ships when they are built. +# A separate group because they have a separate owner, and because a wheel built +# without them ships neither the library nor these symbols. +_QUANTIZED_KERNEL_SYMBOLS = ( + "torch::executor::native::quantize_per_tensor_out", + "torch::executor::native::dequantize_per_tensor_out", +) + # The registry entry points, kept separate from the kernel implementations above. # A library that carries its own copy of these has its own registration code, which # is what this split is meant to prevent: one owner of the operator table. Checking @@ -284,7 +292,37 @@ def _defines_symbol(library: Path, symbol: str) -> bool: return False -def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None: +def _is_export_only(library: Path) -> bool: + """Whether a library exists to export a model rather than to run one. + + The ahead-of-time operator libraries register kernels into torch so a model can be + exported, and they link torch to do it. They deliberately carry their own copy of + the kernels, because the copy a C++ application links is registered into a table + those libraries never read. + + Named by the caller per component rather than excluded everywhere. Counting them for + the component they duplicate would report a duplicate that is not one, and excusing + them for every component would stop this catching a second registry hiding inside + one of them. + + Matched on both the torch dependency and the name marker, because either alone + misfires: several shipped libraries link torch without being export-side, and a + name check alone would accept a runtime library that adopted the suffix. + """ + if _tool("readelf") is None: + return library.name.endswith("_aot_lib.so") + dynamic = subprocess.run( + [_tool("readelf"), "-d", str(library)], + capture_output=True, + text=True, + check=False, + ).stdout + return "libtorch.so" in dynamic and "_aot_lib" in library.name + + +def _assert_single_definer( + symbols, what: str, owner: str | None = None, allow_export_copy: bool = False +) -> None: """At most one shipped library may define each of `symbols`. The owner is named where one is expected, because counting definers alone does @@ -295,11 +333,25 @@ def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None A component the wheel does not ship at all is a valid configuration, not a fault. Delegates and kernel sets are build options, so a wheel built without one has zero definers and is reported as such. What must never happen is two. + + `allow_export_copy` excuses the export-side libraries for one component only. The + quantized kernels genuinely exist twice, once in the runtime library and once in the + library torch loads at export time, because each side registers into a table the + other never reads. Loading both into one process does abort on the second + registration, so what this check enforces for that component is one owner among the + runtime libraries, not the absence of the export copy. Excusing every component + would disarm the check where duplication is a real fault: two of these libraries + defined the backend registry symbols in one released wheel and not in the release + before it, so the duplication this catches does happen. """ assert _tool("nm") is not None, "nm is required to inspect the wheel" package_dir = _installed_package_dir() - libraries = _shipped_shared_objects(package_dir) + libraries = [ + library + for library in _shipped_shared_objects(package_dir) + if not (allow_export_copy and _is_export_only(library)) + ] assert libraries, f"no shared libraries found under {package_dir}" # Every symbol is resolved before anything is reported, so a component that is only @@ -372,6 +424,12 @@ def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None "libexecutorch_kernels_optimized.so", False, ), + ( + "set of quantized kernels", + _QUANTIZED_KERNEL_SYMBOLS, + "libexecutorch_kernels_quantized.so", + True, + ), # The third-party code these libraries bundle, checked separately from the # wrappers above. A wrapper can have a single owner while the implementation # underneath it is bundled into two of these, which is two real thread pools or @@ -398,6 +456,15 @@ def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None ) +# The one component that legitimately exists twice. The quantized kernels are compiled into the runtime +# library and again into the library torch loads at export time, because each side registers into a +# table the other never reads, so a second definer there is expected rather than a fault. A process +# that loads both does abort on the second registration, which is why this is named per component and +# the check stays armed for the other ten, where a second definer means two registries or two thread +# pools in one process. +_COMPONENTS_WITH_AN_EXPORT_COPY = frozenset({"set of quantized kernels"}) + + def test_each_component_has_one_owner() -> None: """No component may be defined by more than one library the wheel ships. @@ -415,7 +482,12 @@ def test_each_component_has_one_owner() -> None: f"the wheel ships no {owner}, which owns the {what}. Either packaging " "dropped it or the build did not produce it." ) - _assert_single_definer(symbols, what, owner if present else None) + _assert_single_definer( + symbols, + what, + owner if present else None, + allow_export_copy=what in _COMPONENTS_WITH_AN_EXPORT_COPY, + ) def test_python_extensions_import() -> None: @@ -1216,17 +1288,43 @@ def test_extension_contains_no_component() -> None: ).stdout.splitlines() if "NEEDED" in line } + # Only the libraries whose code the extension used to contain. Those are the ones + # this split moved out of it, so the extension must now resolve them from outside or + # a retention option silently failed. + # + # Not every shipped library serves Python. The quantized kernels and the CUDA + # delegate exist for a C++ application: Python registers quantized operators through + # the torch-linked ahead-of-time library at export time, and never loads the CUDA + # delegate from this extension at all. Requiring a dependency on those would demand + # the extension link code it has no use for. shipped = {path.name for path in _shipped_runtime_libraries(package_dir)} assert shipped, ( f"the wheel installed no runtime libraries under {package_dir / 'lib'}, so this check would " "compare the extension against nothing and pass" ) - unused = sorted(shipped - needed) + expected = { + name + for name in shipped + if not any( + marker in name + for marker in ("kernels_quantized", "backend_cuda", "extension_cuda") + ) + } + unused = sorted(expected - needed) assert not unused, ( f"the wheel ships {unused} but {extension.name} does not depend on them, so " "either they are dead weight or a retention option did not hold" ) + # Two shipped libraries register the same quantized operators, one for export and one for a C++ + # application, and the runtime treats a repeat registration as fatal. Reaching both from one process + # aborts it, and the only thing preventing that is this extension not depending on the run-time one. + assert not any("kernels_quantized" in name for name in needed), ( + f"{extension.name} depends on the run-time quantized library, which registers the same operators " + "as the export-time one it already loads. The runtime aborts on a repeat registration, so " + "importing this extension would kill the process." + ) + # Positive proof that the extension resolves these from elsewhere, rather than # only the absence of a visible definition. A hidden or local copy would not # appear in the dynamic symbol table at all, so "defines nothing" on its own is @@ -1260,7 +1358,7 @@ def test_extension_contains_no_component() -> None: print( f"✓ {extension.name} ({extension.stat().st_size // 1024} KiB) contains no " f"component, imports the runtime symbols it uses, and depends on all " - f"{len(shipped)} shipped libraries" + f"{len(expected)} shipped libraries it used to contain" ) @@ -1304,6 +1402,7 @@ def test_shipped_library_names_are_expected() -> None: known = ( "libexecutorch", "libexecutorch_kernels_optimized", + "libexecutorch_kernels_quantized", "libexecutorch_backend_xnnpack", "libexecutorch_threadpool", "libexecutorch_etdump", diff --git a/docs/source/using-executorch-cpp.md b/docs/source/using-executorch-cpp.md index f825fe0785b..271a7c09e4e 100644 --- a/docs/source/using-executorch-cpp.md +++ b/docs/source/using-executorch-cpp.md @@ -108,6 +108,7 @@ reported while CMake configures, rather than failing later at link time. | --- | --- | | `executorch::runtime` | the program loader and executor. Always present. | | `executorch::kernels_optimized` | CPU operator kernels. Needed for any operator a delegate does not claim. | +| `executorch::kernels_quantized` | quantized operator kernels, for a quantized model. Link it only when you need it: see the note below. | | `executorch::backend_xnnpack` | the XNNPACK delegate. | | `executorch::threadpool` | the shared thread pool. | | `executorch::etdump` | the profiler. | @@ -117,6 +118,22 @@ kernels a model computes with, so a model that is not fully delegated needs a ke component too. Linking a delegate is what registers it: a program delegated to XNNPACK fails to load in an application that did not link `executorch::backend_xnnpack`. +#### The quantized kernels are opt in + +`executorch::kernels_quantized` is the one component that `${EXECUTORCH_LIBRARIES}` does +not include, so you have to name it. The reason is a conflict with the Python side: +`executorch.kernels.quantized` loads a plugin that carries its own copy of the same +kernels, and the runtime stops when the same operator is registered twice: + +``` +Re-registering quantized_decomposed::add.out +``` + +That only affects a process holding both, for example an application that embeds a +Python interpreter. A plain C++ application can link this component freely. It is kept +out of the default set so that linking whatever the package offers cannot put you in that +position by accident. + To require a minimum version, pass it to `find_package`: ```cmake @@ -179,8 +196,14 @@ An application deployed beside the libraries is unaffected either way, because t `@loader_path` and `$ORIGIN` entries are kept. `EXECUTORCH_LIBRARIES` names the runtime and every component the wheel shipped, so you -cannot choose components on this route. Upgrade to CMake 3.28 and link the specific -targets you need instead. +cannot choose components on this route. The quantized kernels are the exception described +above, offered as `EXECUTORCH_QUANTIZED_KERNELS_LIBRARY` for a consumer that wants them: + +```cmake +target_link_libraries(my_app PRIVATE ${EXECUTORCH_QUANTIZED_KERNELS_LIBRARY}) +``` + +Upgrade to CMake 3.28 and link the specific targets you need instead. ### Building from source diff --git a/kernels/quantized/CMakeLists.txt b/kernels/quantized/CMakeLists.txt index 938c3bf81db..3722ec7528d 100644 --- a/kernels/quantized/CMakeLists.txt +++ b/kernels/quantized/CMakeLists.txt @@ -164,14 +164,14 @@ if(NOT CMAKE_GENERATOR STREQUAL "Xcode" endif() add_library(quantized_kernels ${_quantized_kernels__srcs}) -# The thread pool carries the define that switches parallel_for from a serial -# fallback to the real threaded implementation, so without it quantize, -# dequantize and choose_qparams run on one core. Guarded because a bare metal -# target builds these kernels without a thread pool at all, where the serial -# fallback is the only correct choice. target_link_libraries( quantized_kernels PRIVATE executorch_core kernels_util_all_deps ) +# The thread pool carries the define that switches parallel_for from a serial +# fallback to the real threaded implementation. Without it choose_qparams runs +# on one core, as does the ARM path in quantize. Guarded because a bare metal +# target builds these kernels without a thread pool at all, where the serial +# fallback is the only correct choice. if(TARGET extension_threadpool) target_link_libraries(quantized_kernels PRIVATE extension_threadpool) endif() @@ -197,4 +197,15 @@ if(EXECUTORCH_BUILD_SHARED) executorch_quantized_ops quantized_ops_lib quantized_kernels executorch_shared ) + # Named after what the library provides rather than after the target that + # produces it, matching the optimized kernels next to it, so the shipped file + # reads as libexecutorch_kernels_quantized.so. The target name stays as it is + # because a source build already refers to it. + set_target_properties( + executorch_quantized_ops PROPERTIES OUTPUT_NAME + executorch_kernels_quantized + ) + # Ships beside libexecutorch.so in the wheel's lib/ directory, so it resolves + # the runtime from there rather than from wherever it was built. + executorch_target_shipped_runtime_path(executorch_quantized_ops) endif() diff --git a/setup.py b/setup.py index dddac8771a0..9870fb5cf09 100644 --- a/setup.py +++ b/setup.py @@ -1316,6 +1316,16 @@ def run(self): # noqa C901 if cmake_cache.is_enabled("EXECUTORCH_BUILD_MLX"): cmake_build_args += ["--target", "mlxdelegate"] + # Named explicitly because nothing else links it. The other shipped + # libraries are built as dependencies of the Python extension, but a C++ + # application is the only consumer of this one, so without naming it the + # target is generated and never built, and packaging then looks for a file + # that does not exist. + if cmake_cache.is_enabled("EXECUTORCH_BUILD_SHARED") and ( + cmake_cache.is_enabled("EXECUTORCH_BUILD_KERNELS_QUANTIZED") + ): + cmake_build_args += ["--target", "executorch_quantized_ops"] + if cmake_cache.is_enabled("EXECUTORCH_BUILD_KERNELS_LLM_AOT"): cmake_build_args += ["--target", "custom_ops_aot_lib"] cmake_build_args += ["--target", "quantized_ops_aot_lib"] @@ -1433,6 +1443,19 @@ def run(self): # noqa C901 "EXECUTORCH_BUILD_KERNELS_OPTIMIZED", ], ), + # The quantized kernels, as their own library rather than code + # fused into the AOT-only extension beside the Python bindings. + # A C++ application running a quantized model could not link + # them before. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/kernels/quantized/", + src_name="libexecutorch_kernels_quantized.so", + dst="executorch/lib/libexecutorch_kernels_quantized.so", + dependent_cmake_flags=[ + "EXECUTORCH_BUILD_SHARED", + "EXECUTORCH_BUILD_KERNELS_QUANTIZED", + ], + ), # Install the XNNPACK delegate beside them, so a process has one # copy of it instead of one per component that uses it. BuiltFile( diff --git a/tools/cmake/executorch-wheel-config.cmake b/tools/cmake/executorch-wheel-config.cmake index c337bada296..ffa2e2132c3 100644 --- a/tools/cmake/executorch-wheel-config.cmake +++ b/tools/cmake/executorch-wheel-config.cmake @@ -67,10 +67,21 @@ # dependency and, for a registration-only library, the link options that keep it # from being dropped. The names, when present, are: # -# executorch::kernels_optimized -- The CPU operator kernels. Needed to run a -# model. executorch::backend_xnnpack -- The XNNPACK delegate. -# executorch::threadpool -- The shared thread pool. executorch::etdump -- -# The profiler. +# ~~~ +# executorch::kernels_optimized The CPU operator kernels. Needed to run a model. +# executorch::kernels_quantized The quantized operator kernels, for a quantized +# model. Not part of EXECUTORCH_LIBRARIES, see +# below. +# executorch::backend_xnnpack The XNNPACK delegate. +# executorch::threadpool The shared thread pool. +# executorch::etdump The profiler. +# ~~~ +# +# EXECUTORCH_LIBRARIES carries every component except the quantized kernels, +# which a consumer names explicitly instead. The export-time plugin that +# executorch.kernels.quantized loads carries its own copy of those kernels, so a +# process holding both stops on a repeated operator registration, and a consumer +# linking the aggregate would inherit that without asking for it. # # Check with if(TARGET executorch::) rather than assuming one exists. A # namespaced name that was never defined is a configure-time error that names @@ -274,6 +285,13 @@ if(_executorch_runtime_library AND NOT _executorch_targets_supported) # then a load failure saying the backend is not registered, which reads as a # model problem. Anything the wheel did not ship is simply not found and # skipped. + # + # The quantized kernels are deliberately absent, for the reason given at their + # component definition below: they collide with the export-time plugin that + # executorch.kernels.quantized loads, and a process holding both dies. This + # route has no per-component target to opt into, so they are offered through + # EXECUTORCH_QUANTIZED_KERNELS_LIBRARY instead and a consumer that wants them + # links that as well. foreach(_executorch_component IN ITEMS libexecutorch_kernels_optimized libexecutorch_backend_xnnpack libexecutorch_threadpool libexecutorch_etdump @@ -308,6 +326,21 @@ if(_executorch_runtime_library AND NOT _executorch_targets_supported) endif() endforeach() unset(_executorch_component_library) + # Held out of the aggregate above, so name it separately. A consumer that + # wants quantized operators and does not load the Python plugin in the same + # process links this too. Empty when the wheel shipped no such library. + _executorch_find_library( + EXECUTORCH_QUANTIZED_KERNELS_LIBRARY libexecutorch_kernels_quantized + ) + if(EXECUTORCH_QUANTIZED_KERNELS_LIBRARY AND CMAKE_SYSTEM_NAME STREQUAL + "Linux" + ) + # The same scoped retention the aggregate entries get, for the same reason: + # a registration-only library exports nothing the application references. + set(EXECUTORCH_QUANTIZED_KERNELS_LIBRARY + "-Wl,--push-state,--no-as-needed,${EXECUTORCH_QUANTIZED_KERNELS_LIBRARY},--pop-state" + ) + endif() message( STATUS "executorch: the prebuilt runtime is present but its imported targets need CMake 3.28 or " @@ -410,8 +443,13 @@ endif() # most: a registration-only library has no symbol the application references, so # a normal link drops it and its registration never runs. # -# Call as: _executorch_define_component( ) +# Call as: _executorch_define_component( +# [OPT_IN]) +# +# OPT_IN defines the target but keeps it out of EXECUTORCH_LIBRARIES, for a +# library a consumer has to choose deliberately rather than receive by default. function(_executorch_define_component _suffix _library_name) + cmake_parse_arguments(PARSE_ARGV 2 _component "OPT_IN" "" "") # Same reason the runtime target is skipped on older CMake: a component target # exports an $ORIGIN-relative search path, and a version that writes it wrong # produces a target that works in place and fails once deployed. @@ -434,10 +472,12 @@ function(_executorch_define_component _suffix _library_name) # returning here would hand the second caller a list with the runtime but # none of the components. A consumer linking that variable would then be # missing its kernels and fail at load with an unregistered operator. - set(EXECUTORCH_LIBRARIES - ${EXECUTORCH_LIBRARIES} ${_target} - PARENT_SCOPE - ) + if(NOT _component_OPT_IN) + set(EXECUTORCH_LIBRARIES + ${EXECUTORCH_LIBRARIES} ${_target} + PARENT_SCOPE + ) + endif() return() endif() add_library(${_target} SHARED IMPORTED) @@ -483,10 +523,12 @@ function(_executorch_define_component _suffix _library_name) "LINKER:--push-state,--no-as-needed,${_library},--pop-state" ) endif() - set(EXECUTORCH_LIBRARIES - ${EXECUTORCH_LIBRARIES} ${_target} - PARENT_SCOPE - ) + if(NOT _component_OPT_IN) + set(EXECUTORCH_LIBRARIES + ${EXECUTORCH_LIBRARIES} ${_target} + PARENT_SCOPE + ) + endif() endfunction() _executorch_define_component(threadpool executorch_threadpool) @@ -495,6 +537,25 @@ _executorch_define_component(threadpool executorch_threadpool) # checks, so it has to be defined here or a consumer following the documentation # gets a bare name that CMake hands to the linker as a literal flag. _executorch_define_component(kernels_optimized executorch_kernels_optimized) +# The quantized kernels, optional in the same way: a wheel built without them +# simply has no such library and the component is not defined. +# +# Opt in rather than part of the aggregate. The export-time plugin that +# executorch.kernels.quantized loads registers the same operator names, and the +# runtime stops on a repeat registration rather than choosing one, so a process +# holding both dies. Measured: linking this library and importing that module in +# either order aborts with "Re-registering quantized_decomposed::add.out". None +# of the other shipped components collide this way, so only this one is held +# back, and a consumer that wants it names it. +_executorch_define_component( + kernels_quantized executorch_kernels_quantized OPT_IN +) +# The same library exposed through a variable, so a consumer that follows the +# pre-3.28 recipe and later upgrades past 3.28 keeps working. Left empty when +# the wheel shipped no such library, matching the pre-3.28 branch above. +if(TARGET executorch::kernels_quantized) + set(EXECUTORCH_QUANTIZED_KERNELS_LIBRARY executorch::kernels_quantized) +endif() # The profiler. A C++ application could not record timing data from an installed # package before, because the implementation shipped only inside the Python # extension. @@ -721,13 +782,28 @@ foreach(_component ${executorch_FIND_COMPONENTS}) # One string rather than several arguments. Several make a list, and # message() joins a list with semicolons, which lands separators mid # sentence. - string( - CONCAT - executorch_NOT_FOUND_MESSAGE - "the required component '${_component}' needs CMake 3.28 or newer, because older " - "versions write the \$ORIGIN token in a runtime search path incorrectly; this " - "package is otherwise usable through EXECUTORCH_LIBRARIES" - ) + # + # The quantized kernels are held out of EXECUTORCH_LIBRARIES on purpose, + # so a consumer who wants them names + # EXECUTORCH_QUANTIZED_KERNELS_LIBRARY instead. See the OPT_IN comment + # at the component definition above. + if(_component STREQUAL "kernels_quantized") + string( + CONCAT + executorch_NOT_FOUND_MESSAGE + "the required component '${_component}' needs CMake 3.28 or newer, because " + "older versions write the \$ORIGIN token in a runtime search path incorrectly; " + "this package is otherwise usable through EXECUTORCH_QUANTIZED_KERNELS_LIBRARY" + ) + else() + string( + CONCAT + executorch_NOT_FOUND_MESSAGE + "the required component '${_component}' needs CMake 3.28 or newer, because older " + "versions write the \$ORIGIN token in a runtime search path incorrectly; this " + "package is otherwise usable through EXECUTORCH_LIBRARIES" + ) + endif() else() # One string rather than several arguments. Several make a list, and # message() joins a list with semicolons, which lands separators mid From afbbf2d04a9ad9e8e39fbc478306335edf5bf767 Mon Sep 17 00:00:00 2001 From: shoumikhin Date: Tue, 11 Aug 2026 23:08:10 -0700 Subject: [PATCH 4/6] Ship the CUDA delegate in the wheel The CUDA delegate runs a model on an NVIDIA GPU. It is built into the Python extension, so only Python can use it. A C++ application has no way to link it, and nothing else can reuse it either. There is a sharing problem too. A program may use more than one GPU backend at once, and they need to agree on which CUDA stream (the queue the GPU runs work on) the caller chose. If each backend carries its own copy of that state, work queued through one is invisible to the other. Ship the CUDA delegate and a small stream helper as their own shared libraries, and name both as CMake components. The stream helper is shared so a process has exactly one copy of the caller's stream choice, which is what lets two backends agree on it. ```cmake find_package(executorch REQUIRED COMPONENTS backend_cuda) target_link_libraries(my_app PRIVATE executorch::runtime executorch::backend_cuda) ``` A CUDA wheel does not bundle the CUDA runtime. It declares it as a dependency, the way the PyTorch CUDA wheels do, so one copy is shared with torch rather than shipping a second one: ``` executorch/lib/libexecutorch_backend_cuda.so the delegate executorch/lib/libexecutorch_extension_cuda.so the stream helper executorch/backends/cuda/libaoti_cuda_shims.so the GPU device code ``` Each library records a relative path to where pip installs the CUDA runtime, so it resolves without the caller setting a library search path and without depending on a toolkit being installed. The declared set includes the runtime compiler, because a shipped library links it to build kernels at run time. Each declared package also needs its own directory recorded, since that is where the loader looks. On the CUDA 12 packaging the compiler installs into its own directory, and omitting it left that library unable to find the compiler even though the package was installed. The CUDA 13 packaging puts every component in one directory, so the same gap does not appear there. The stream helper's header no longer includes `cuda_runtime.h`, which the wheel does not publish. It only ever uses a CUDA stream as an opaque handle, so it declares that handle itself, and a consumer can compile against the wheel with no CUDA toolkit installed. Built a CUDA wheel, installed it into a clean environment, and: - ran a GPU model from Python on an NVIDIA GPU, with output identical to eager PyTorch on the same weights and inputs (largest absolute difference 0). - built a C++ application against the installed wheel alone and ran the same model, matching the same reference. - confirmed one library defines the stream state and the GPU shims, not several. Extracting them into every consumer put three copies in one wheel, and a stream selected through one was invisible to the others. - confirmed no shipped library records a CUDA toolkit path from the build machine, and every library that links the CUDA runtime has a relative path to it. - confirmed a CPU wheel ships none of the CUDA libraries and no CUDA-only header. - a row that names a CUDA train is built with the CUDA option on rather than left to autodetection, so a builder without a matching toolkit fails while configuring. Before this, such a row produced a wheel tagged for CUDA, carrying no CUDA library, that still declared the CUDA runtime packages: it installed cleanly and then reported the backend as unregistered when a model ran. - a row whose major CUDA version does not match the installed toolkit now fails the build. The declared packages and the loader paths come from the row while the binaries come from the toolkit, and nothing compared the two, so a `cu126` row built against a 13.0 toolkit attached CUDA 12 metadata to binaries needing `libcudart.so.13`. An unrecognised train fails too, instead of silently reporting whatever the builder happened to have. Detection reads the toolkit major directly, so the guard fires on any mismatch rather than only on the three exact `(major, minor)` pairs the supported list carries; on those three pairs it behaved correctly before, and on every other minor it saw an empty detection and skipped the check. - the row classifier and the packaging read the row the same way now, so both agree on what a row spelled with an unsupported minor means. The shell classifier reduces the row to digits and matches against `SUPPORTED_CUDA_VERSIONS`. Packaging did the same shape on the outer decision and then took only the first two digits when picking runtime packages, so `cu125` classified as CPU on one side and declared CUDA 12 on the other. Packaging now matches on the same digits and raises loudly on an unsupported train instead. - whether a row is a CUDA row is decided by asking if it names a supported train, rather than by listing the spellings that mean "no CUDA". Checked 16 row values including `cpu-aarch64`, `rocm6.2` and `cu118`; the previous list-based form was wrong on several, and each wrong answer made a non-CUDA wheel declare the CUDA runtime. - the CUDA components are required when the wheel's own version says it is a CUDA wheel. They were optional unconditionally, so a wheel tagged `+cu126` with no CUDA library at all passed every check. - the stream helper ships under either name it can be built with. The shim layer records it as a dependency whenever CUDA is on, while packaging named only the shared-build spelling, so a non-shared build shipped a shim whose dependency resolved to nothing. - the relative hops between shipped libraries are sized by how deep the library sits in the package. A fixed pair was correct at one depth only: measured over every shipped location, 6 of 12 hops landed on a directory that does not exist, and the hop from `lib/` climbed out of the package entirely, where an unrelated library with a matching soname could satisfy the dependency first. - the stream helper no longer links or includes the CUDA toolkit. It uses a stream only as an opaque handle and calls no CUDA function, so it needs no toolkit include and no libcudart link. Also fixed in this commit: - The pre-build classifier resolves the Python interpreter (`python3` or `python`, whichever exists) instead of assuming one name, and no longer discards stderr. Builders disagree on the name: Linux and macOS provide `python3`, while the Windows builder runs inside a conda environment that provides only `python`. Assuming either name breaks the other platform, and treating the failure as "not a CUDA row" silently rebuilt a CUDA row as a CPU row. - `CU_VERSION=cpu pip install .` is handled explicitly instead of running the CUDA-train parser over it, which previously turned `cpu` into `pu` through a character-set strip and reached the unsupported-train error. Ran end to end on H100, A100 and Jetson Thor, covering compute capabilities 9.0, 8.0 and 11.0. Known gap, not introduced here: the Python `Runtime.load_program` path allocates activation memory on the host, so a program exported to keep activations on the GPU fails there. The supported Python loader and the C++ path both work. This is upstream in the Python bindings, which this change does not touch. ghstack-source-id: 59fa23e818b4ede2e9540de97eecc21eac68b732 ghstack-comment-id: 5219161655 Pull-Request: https://github.com/pytorch/executorch/pull/21645 --- .ci/scripts/wheel/pre_build_script.sh | 74 +++ .ci/scripts/wheel/test_cpp_sdk.py | 14 +- .ci/scripts/wheel/test_shared_libraries.py | 176 +++++-- backends/cuda/CMakeLists.txt | 52 +- docs/source/using-executorch-cpp.md | 2 + extension/cuda/CMakeLists.txt | 16 +- extension/cuda/caller_stream.h | 9 +- install_utils.py | 69 ++- setup.py | 546 ++++++++++++++++++--- tools/cmake/executorch-wheel-config.cmake | 13 +- tools/cmake/preset/pybind.cmake | 10 +- 11 files changed, 839 insertions(+), 142 deletions(-) diff --git a/.ci/scripts/wheel/pre_build_script.sh b/.ci/scripts/wheel/pre_build_script.sh index 367d398bac8..c617217009b 100755 --- a/.ci/scripts/wheel/pre_build_script.sh +++ b/.ci/scripts/wheel/pre_build_script.sh @@ -44,6 +44,80 @@ if [[ "$(uname -m)" == "aarch64" ]]; then echo "the file $file has been modified for atomic to use full path" fi +# A CPU row must say so, rather than relying on the builder having no CUDA toolkit installed. The build +# turns CUDA on when it detects one, so a builder that gains a toolkit would silently start producing a +# CPU wheel carrying the CUDA delegate. That already happened on Windows, where the image ships a toolkit +# on PATH and the resulting wheel failed to load its own extension. +# +# Stated as the inverse rule: anything that does not name a CUDA train this project supports is a CPU row. +# An allowlist of spellings was tried first and left a gap for every spelling nobody thought of, which is +# the same defect twice: testing only for empty let a row spelled "cpu" through, and listing "cpu" still +# leaves "cpu-aarch64", "rocm" and anything else the matrix generator emits. +# +# The supported trains come from install_utils.py, so both classifiers see the same list. +# A row that names CUDA and is not a supported train fails here rather than being +# rebadged as CPU. The wheel matrix comes from a different repository than this list, so +# when they drift the alternative is publishing a CPU wheel under a CUDA-named index with +# no error anywhere: setup.py cannot catch it, because the CPU option this script writes +# is read first and returns early. +CUDA_ROW=0 +if [[ -n "${CU_VERSION:-${DESIRED_CUDA:-}}" ]]; then + ROW_VALUE="${CU_VERSION:-${DESIRED_CUDA:-}}" + # Windows builders run MSYS bash in a conda environment that has python.exe + # and no python3, so either name alone breaks a platform. Mirrors + # run_python_script.sh. Resolving up front instead of testing the exit status + # keeps a missing interpreter a hard failure rather than a silent CPU build. + PYTHON_BIN=$(command -v python3 || command -v python) + row_classification=$("${PYTHON_BIN}" - "${ROW_VALUE}" <<'PY' +import re, sys +sys.path.insert(0, '.') +from install_utils import SUPPORTED_CUDA_VERSIONS +raw = sys.argv[1].strip().lower() +trains = {f'{major}{minor}' for major, minor in SUPPORTED_CUDA_VERSIONS} +# Decide by what the row NAMES, not by the digits it happens to contain. Reducing the whole +# value to digits classified rocm13.2 as CUDA and rebadged a row named plainly "cuda" as CPU. +match = re.fullmatch(r'cu(?:da)?[-_]?(.*)', raw) +if match is None: + print('cpu') +else: + digits = re.sub(r'[^0-9]', '', match.group(1)) + print('cuda' if digits in trains else 'unsupported') +PY +) + if [[ "${row_classification}" == "cuda" ]]; then + CUDA_ROW=1 + elif [[ "${row_classification}" == "unsupported" ]]; then + echo "row '${ROW_VALUE}' names a CUDA train this project does not support." >&2 + echo "Add it to SUPPORTED_CUDA_VERSIONS in install_utils.py, and to" >&2 + echo "_CUDA_RUNTIME_PACKAGES and _CUDA_LIBRARY_DIRECTORIES in setup.py, or" >&2 + echo "stop building this row. Building it as CPU would publish a CPU wheel" >&2 + echo "under a CUDA-named index." >&2 + exit 1 + fi +fi + +if [[ ${CUDA_ROW} -eq 0 ]]; then + export CMAKE_ARGS="${CMAKE_ARGS:-} -DEXECUTORCH_BUILD_CUDA=OFF" + echo "CMAKE_ARGS=${CMAKE_ARGS}" >> "${GITHUB_ENV}" + echo "row '${CU_VERSION:-${DESIRED_CUDA:-}}' names no supported CUDA train, building CPU-only" +else + # A CUDA row must produce the CUDA libraries. Left at the default, the build only + # turns CUDA on if it happens to detect a toolkit, so a builder without one produced + # a wheel tagged for CUDA, carrying no CUDA library, while still declaring the CUDA + # runtime packages. That installs cleanly and then reports the backend as + # unregistered when a model runs. Asking for it explicitly stops the decision from + # depending on detection. + # + # It does not turn a missing toolkit into a configure failure. The CUDA directory + # requires the toolkit but gates its sources on a working compiler, so a builder that + # has toolkit files and no usable nvcc still configures and simply compiles none of + # them. That leniency is deliberate, for packaging jobs that cannot complete compiler + # identification, so the row itself has to verify the libraries it expected are present. + export CMAKE_ARGS="${CMAKE_ARGS:-} -DEXECUTORCH_BUILD_CUDA=ON" + echo "CMAKE_ARGS=${CMAKE_ARGS}" >> "${GITHUB_ENV}" + echo "row '${CU_VERSION:-${DESIRED_CUDA:-}}' is a CUDA row, requiring the CUDA build" +fi + # On Windows, enable symlinks and re-checkout the current revision to create # the symlinked src/ directory. This is needed to build the wheel. if [[ $UNAME_S == *"MINGW"* || $UNAME_S == *"MSYS"* ]]; then diff --git a/.ci/scripts/wheel/test_cpp_sdk.py b/.ci/scripts/wheel/test_cpp_sdk.py index 478554067de..68597de8c03 100644 --- a/.ci/scripts/wheel/test_cpp_sdk.py +++ b/.ci/scripts/wheel/test_cpp_sdk.py @@ -865,16 +865,10 @@ def test_every_shipped_header_compiles(work_dir: Path) -> None: # tests a configuration no consumer of this package is ever in. "-DC10_USING_CUSTOM_GENERATED_MACROS", ] - # A CUDA wheel's headers reference the CUDA runtime, which the wheel does not publish headers for and - # states as a requirement instead. A consumer of that component supplies a toolkit of its own, so this - # check does the same rather than treating the header as unbuildable. Taken from the toolkit itself, so - # it follows whichever toolkit the build used instead of a list of prefixes that goes stale. - nvcc = shutil.which("nvcc") - cuda_root = os.environ.get("CUDA_HOME") or ( - str(Path(nvcc).parent.parent) if nvcc else "" - ) - if cuda_root and (Path(cuda_root) / "include" / "cuda_runtime.h").is_file(): - includes.append(f"-I{Path(cuda_root) / 'include'}") + # Deliberately no CUDA toolkit include directory. A CUDA wheel's own headers have to compile + # against nothing but the wheel, the same as every other header here. Adding the builder's toolkit + # would measure the build machine rather than the consumer, and a header that only compiles that way + # fails in the consumer's project instead of here. # Headers a wheel-only consumer cannot compile and is not expected to. Each needs something outside the # package: a platform that is not the one being built for, or a third-party library the wheel does not # carry. They ship because a source build includes them, and holding them to this rule would report a diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py index 76506429469..0484736f9f4 100644 --- a/.ci/scripts/wheel/test_shared_libraries.py +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -55,6 +55,27 @@ # The quantized kernels, whose own library the wheel ships when they are built. # A separate group because they have a separate owner, and because a wheel built # without them ships neither the library nor these symbols. +# The CUDA delegate and its stream helper, for a wheel built from a CUDA index. The +# stream helper matters most: two copies means two notions of the caller's stream, so +# work queued through one is invisible to the other. +# A strongly defined symbol, chosen by reading the built library rather than guessed. The +# CudaBackend methods are emitted weak, and a weak definition can be replaced at load time by a +# strong one elsewhere, so naming one of those would count a definition that may not be the one +# the process uses. +_CUDA_BACKEND_SYMBOLS = ("executorch::backends::cuda::load_library",) +_CUDA_STREAM_SYMBOLS = ("executorch::extension::cuda::getCallerStream",) + +# The AOTI shim layer and the stream-guard state that lives with it. This is the state that was +# genuinely duplicated: extracting the shims with a PUBLIC whole-archive replayed the extraction at +# every consumer's link, so the guard's thread_local and these shims landed in three shipped binaries +# at once, and a stream selected through one copy was invisible to the other two. The row above cannot +# catch that, because its symbol only ever existed in one unconditionally shared library. +_AOTI_SHIM_SYMBOLS = ( + "aoti_torch_empty_strided", + "aoti_torch_delete_tensor_object", + "executorch::backends::cuda::CUDAStreamGuard::create", +) + _QUANTIZED_KERNEL_SYMBOLS = ( "torch::executor::native::quantize_per_tensor_out", "torch::executor::native::dequantize_per_tensor_out", @@ -300,24 +321,29 @@ def _is_export_only(library: Path) -> bool: the kernels, because the copy a C++ application links is registered into a table those libraries never read. - Named by the caller per component rather than excluded everywhere. Counting them for - the component they duplicate would report a duplicate that is not one, and excusing - them for every component would stop this catching a second registry hiding inside - one of them. + Excluded from the single-owner check for the one component that genuinely has two + copies. Counting them there would report a duplicate for something that is not one, + and the alternative, making them resolve the kernels from the shipped library, would + mean an export-time library depending on a runtime layout it never uses. - Matched on both the torch dependency and the name marker, because either alone - misfires: several shipped libraries link torch without being export-side, and a - name check alone would accept a runtime library that adopted the suffix. + Recognised by linking torch, which is the property that makes a library export-side. + Python extensions link torch too and are not export-side operator libraries, so they + are excluded by their interpreter suffix rather than by an operator-library name, + which keeps the test's verdict the same whether or not readelf is installed. """ + if ".cpython-" in library.name or library.name.endswith(".pyd"): + return False + if library.name.endswith("_aot_lib.so"): + return True if _tool("readelf") is None: - return library.name.endswith("_aot_lib.so") + return False dynamic = subprocess.run( [_tool("readelf"), "-d", str(library)], capture_output=True, text=True, check=False, ).stdout - return "libtorch.so" in dynamic and "_aot_lib" in library.name + return "libtorch.so" in dynamic def _assert_single_definer( @@ -398,6 +424,22 @@ def _assert_single_definer( print(f"✓ single {what}{where} across {len(libraries)} shipped libraries") +def _wheel_cuda_train() -> str: + """The CUDA train the installed wheel was built for, or "" for a CPU wheel. + + Read from the local version segment, which is the only place the wheel states what + it was built for. `1.5.0+cu126` gives "126". + """ + local = importlib.metadata.version("executorch").partition("+")[2] + return local[2:] if local.startswith("cu") else "" + + +# Marker for a row whose owner is required only when the wheel is a CUDA wheel. A +# sentinel rather than a boolean, because the answer is not known until the installed +# wheel is inspected, and a row cannot call that at import time. +_REQUIRED_ON_A_CUDA_WHEEL = "cuda-wheel-only" + + # Each component the wheel ships as its own library, the symbols that identify it, # and the library that must own them. `required` says whether the owner has to be # present: the optimized kernels are optional, because a wheel built without them @@ -430,6 +472,29 @@ def _assert_single_definer( "libexecutorch_kernels_quantized.so", True, ), + # The CUDA components. Required exactly when the wheel says it is a CUDA wheel, + # which is decided at check time rather than here: a fixed False meant a wheel + # tagged +cu126 carrying no CUDA library at all passed every check in this file, + # while a fixed True would fail every CPU wheel. The marker is the string these + # rows are keyed on below. + ( + "CUDA delegate", + _CUDA_BACKEND_SYMBOLS, + "libexecutorch_backend_cuda.so", + _REQUIRED_ON_A_CUDA_WHEEL, + ), + ( + "CUDA stream helper", + _CUDA_STREAM_SYMBOLS, + "libexecutorch_extension_cuda.so", + _REQUIRED_ON_A_CUDA_WHEEL, + ), + ( + "AOTI shim layer", + _AOTI_SHIM_SYMBOLS, + "libaoti_cuda_shims.so", + _REQUIRED_ON_A_CUDA_WHEEL, + ), # The third-party code these libraries bundle, checked separately from the # wrappers above. A wrapper can have a single owner while the implementation # underneath it is bundled into two of these, which is two real thread pools or @@ -455,7 +520,6 @@ def _assert_single_definer( ), ) - # The one component that legitimately exists twice. The quantized kernels are compiled into the runtime # library and again into the library torch loads at export time, because each side registers into a # table the other never reads, so a second definer there is expected rather than a fault. A process @@ -473,10 +537,17 @@ def test_each_component_has_one_owner() -> None: registers into a table nothing else reads shows up as an operator missing at run time rather than as a link error. """ - shipped = { - path.name for path in _shipped_runtime_libraries(_installed_package_dir()) - } + # Every shipped shared object, not only the ones under lib/. One owner, libaoti_cuda_shims.so, + # ships under backends/cuda/, and scanning lib/ alone reported it as absent, which each row + # treats as an acceptable state and so would have skipped the check entirely. + shipped = {path.name for path in _shipped_shared_objects(_installed_package_dir())} + on_a_cuda_wheel = bool(_wheel_cuda_train()) for what, symbols, owner, required in _OWNED_COMPONENTS: + if required == _REQUIRED_ON_A_CUDA_WHEEL: + # Resolved here rather than in the table, because it depends on the installed + # wheel. A fixed False let a wheel tagged +cu126 ship with no CUDA library at + # all and still pass, which is the whole point of these three rows. + required = on_a_cuda_wheel present = any(name.startswith(owner) for name in shipped) assert present or not required, ( f"the wheel ships no {owner}, which owns the {what}. Either packaging " @@ -1168,7 +1239,8 @@ def names_a_build_directory(entry: str) -> bool: ) offenders = {} - checked = 0 + inspected = 0 + with_a_runtime_path = 0 for library in sorted(package_dir.rglob("*.so*")): if not library.is_file() or library.is_symlink(): continue @@ -1180,6 +1252,7 @@ def names_a_build_directory(entry: str) -> bool: ) if result.returncode != 0: continue + inspected += 1 # An absent RPATH and one containing a single empty entry both print as an # empty string, so treat empty output as "no runtime path" rather than as an # empty entry. A library with nothing to search is fine; the defect is @@ -1187,7 +1260,7 @@ def names_a_build_directory(entry: str) -> bool: raw = result.stdout.strip() if not raw: continue - checked += 1 + with_a_runtime_path += 1 bad = [] for entry in raw.split(":"): if not entry: @@ -1213,13 +1286,15 @@ def names_a_build_directory(entry: str) -> bool: "the build tree that produced the wheel or, for an empty entry, the process " f"working directory: {offenders}" ) - assert checked, ( - f"no shipped library under {package_dir} carries a runtime search path, so this check examined " - "nothing and would pass on a wheel that shipped no libraries at all" + # Counted separately, because a wheel whose libraries all had their runtime paths removed + # entirely would satisfy a readable-file count while this check examined no path at all. + assert with_a_runtime_path, ( + f"none of the {inspected} shipped libraries under {package_dir} carries a runtime search path, " + "so this check examined nothing. The shipped libraries need a relative path to reach each other." ) print( - f"✓ none of the {checked} shipped libraries searches a build-tree or empty " - "runtime path" + f"✓ none of the {with_a_runtime_path} shipped libraries with a runtime path searches a " + f"build-tree or empty directory ({inspected} inspected)" ) @@ -1256,12 +1331,17 @@ def test_extension_contains_no_component() -> None: # # The bundled third-party groups are left out on purpose. That code is also linked by torch, and the # extension links torch, so seeing those symbols there says nothing about this split. - # Only components whose owning library is actually in this wheel. One of them is optional, so a build - # with it turned off ships no owner, and asserting the extension does not define its symbols would - # reject a configuration the table itself marks as supported. - shipped = { - path.name for path in _shipped_runtime_libraries(_installed_package_dir()) - } + # Only components whose owning library is actually in this wheel. A build with an optional component + # turned off ships no owner, and asserting the extension does not define its symbols would reject a + # configuration the table itself marks as supported. Required rows are kept regardless, because their + # owner missing is a packaging fault the owner check reports. + shipped = {path.name for path in _shipped_runtime_libraries(package_dir)} + # Guarded here rather than further down, so it protects the filter that uses it: a wheel that + # installed no runtime libraries would otherwise compare the extension against an empty set and pass. + assert shipped, ( + f"the wheel installed no runtime libraries under {package_dir / 'lib'}, so this check would " + "compare the extension against nothing and pass" + ) owned = tuple( symbol for _, symbols, owner, required in _OWNED_COMPONENTS @@ -1292,23 +1372,20 @@ def test_extension_contains_no_component() -> None: # this split moved out of it, so the extension must now resolve them from outside or # a retention option silently failed. # - # Not every shipped library serves Python. The quantized kernels and the CUDA - # delegate exist for a C++ application: Python registers quantized operators through - # the torch-linked ahead-of-time library at export time, and never loads the CUDA - # delegate from this extension at all. Requiring a dependency on those would demand - # the extension link code it has no use for. - shipped = {path.name for path in _shipped_runtime_libraries(package_dir)} - assert shipped, ( - f"the wheel installed no runtime libraries under {package_dir / 'lib'}, so this check would " - "compare the extension against nothing and pass" - ) + # Not every shipped library serves Python. The quantized kernels exist for a C++ + # application, since Python registers those operators through the torch-linked + # ahead-of-time library at export time, and requiring a dependency would demand the + # extension link code it has no use for. + # + # The CUDA delegate is NOT in that category. The build deliberately links it into the + # extension with a retention option, so it does carry a dependency, and excluding it + # switched off the one check that would notice if that retention stopped working. The + # stream helper stays excluded because the extension reaches it only through the + # delegate's public link, with no retention of its own to protect. expected = { name for name in shipped - if not any( - marker in name - for marker in ("kernels_quantized", "backend_cuda", "extension_cuda") - ) + if not any(marker in name for marker in ("kernels_quantized", "extension_cuda")) } unused = sorted(expected - needed) assert not unused, ( @@ -1371,11 +1448,13 @@ def test_shipped_library_names_are_expected() -> None: and still passed every symbol check, because those checks only ask how many definers a symbol has, never whether a file belongs in the wheel at all. - Two properties catch it. A library's recorded soname matches its file name, or a - consumer records a dependency the wheel does not contain. And its name is one - packaging knows how to produce, which is what a leftover from an older layout - fails. A wheel ships unversioned names on purpose, so the name itself carries no - version to check. + Two properties catch it. Its name is one packaging knows how to produce, which is + what a leftover from an older layout fails. And its recorded soname matches its + file name, or a consumer records a dependency the wheel does not contain. + + The names are unversioned, because these libraries ship one file each with no + symlink chain, and a versioned name without the usual symlinks is harder to load + rather than safer. """ package_dir = _installed_package_dir() lib_dir = package_dir / "lib" @@ -1403,6 +1482,13 @@ def test_shipped_library_names_are_expected() -> None: "libexecutorch", "libexecutorch_kernels_optimized", "libexecutorch_kernels_quantized", + "libexecutorch_backend_cuda", + "libexecutorch_extension_cuda", + # The same library under the name a non-shared build gives it. The shared + # build renames it to match the other shipped components; every other build + # leaves this spelling, and the shim layer records whichever one exists as a + # dependency, so both have to ship and both are expected here. + "libextension_cuda", "libexecutorch_backend_xnnpack", "libexecutorch_threadpool", "libexecutorch_etdump", diff --git a/backends/cuda/CMakeLists.txt b/backends/cuda/CMakeLists.txt index 06990692428..05f238401a4 100644 --- a/backends/cuda/CMakeLists.txt +++ b/backends/cuda/CMakeLists.txt @@ -167,16 +167,17 @@ if(_cuda_is_msvc_toolchain) # avoiding duplicate static/object inclusion and interface leakage. target_link_libraries(aoti_cuda_shims PRIVATE aoti_common_shims_slim_obj) else() + # The whole-archive pair is PRIVATE so it applies to this library's own link + # and does not replay at the link of anything that links this one. In PUBLIC + # scope the archive was extracted again by each consumer, so the translation + # unit holding SlimTensor's per-device current stream thread_local landed in + # three shipped binaries, and one copy per process is what makes that state + # agree. The MSVC branch above uses PRIVATE for the same reason. target_link_libraries( aoti_cuda_shims - PRIVATE cuda_platform - PUBLIC -Wl,--whole-archive - aoti_common_shims_slim - -Wl,--no-whole-archive - CUDA::cudart - CUDA::curand - extension_cuda - ${CMAKE_DL_LIBS} + PRIVATE cuda_platform -Wl,--whole-archive aoti_common_shims_slim + -Wl,--no-whole-archive + PUBLIC CUDA::cudart CUDA::curand extension_cuda ${CMAKE_DL_LIBS} ) endif() @@ -184,6 +185,13 @@ if(NOT _cuda_is_msvc_toolchain) executorch_target_link_options_shared_lib(aoti_cuda_shims) endif() +# This library links the CUDA runtime directly and the wheel ships it, so it +# needs a search path for the same reason the delegate does. Without one it can +# ship with no runtime path at all, on a builder where the runtime resolves from +# an implicit link directory, and the wheel then adds no relative hop either, +# because the step that adds one has nothing to rewrite. +executorch_target_shipped_runtime_path(aoti_cuda_shims) + install( TARGETS aoti_cuda_shims EXPORT ExecuTorchTargets @@ -200,7 +208,17 @@ if(_cuda_is_msvc_toolchain) list(APPEND _aoti_cuda_backend_sources runtime/cuda_allocator.cpp) endif() -add_library(aoti_cuda_backend STATIC ${_aoti_cuda_backend_sources}) +# Build the delegate as a shared library for the wheel so a process has one copy +# of it, and keep it static everywhere else so no other build changes. +if(EXECUTORCH_BUILD_SHARED) + set(_aoti_cuda_backend_library_type SHARED) +else() + set(_aoti_cuda_backend_library_type STATIC) +endif() +add_library( + aoti_cuda_backend ${_aoti_cuda_backend_library_type} + ${_aoti_cuda_backend_sources} +) target_include_directories( aoti_cuda_backend @@ -240,6 +258,22 @@ endif() executorch_target_link_options_shared_lib(aoti_cuda_backend) +if(EXECUTORCH_BUILD_SHARED) + # Named after what the library provides rather than after the target that + # produces it, matching the other shipped delegates, so the file reads as + # libexecutorch_backend_cuda.so. The target name stays as it is because the + # rest of the build already refers to it. + set_target_properties( + aoti_cuda_backend PROPERTIES OUTPUT_NAME executorch_backend_cuda + ) + executorch_target_soname_policy(aoti_cuda_backend) + # Resolve the runtime from the shared library rather than from the static + # core, so the delegate registers into the one registry the process has. + target_link_libraries(aoti_cuda_backend PUBLIC executorch_shared) + # Ships beside the runtime in the wheel's lib/ directory. + executorch_target_shipped_runtime_path(aoti_cuda_backend) +endif() + install( TARGETS aoti_cuda_backend EXPORT ExecuTorchTargets diff --git a/docs/source/using-executorch-cpp.md b/docs/source/using-executorch-cpp.md index 271a7c09e4e..b6187490bd8 100644 --- a/docs/source/using-executorch-cpp.md +++ b/docs/source/using-executorch-cpp.md @@ -110,6 +110,8 @@ reported while CMake configures, rather than failing later at link time. | `executorch::kernels_optimized` | CPU operator kernels. Needed for any operator a delegate does not claim. | | `executorch::kernels_quantized` | quantized operator kernels, for a quantized model. Link it only when you need it: see the note below. | | `executorch::backend_xnnpack` | the XNNPACK delegate. | +| `executorch::backend_cuda` | the CUDA delegate, in a CUDA wheel. | +| `executorch::extension_cuda` | the CUDA stream helper, in a CUDA wheel. Lets you pick the CUDA stream a model runs on. | | `executorch::threadpool` | the shared thread pool. | | `executorch::etdump` | the profiler. | diff --git a/extension/cuda/CMakeLists.txt b/extension/cuda/CMakeLists.txt index 0003691ac8b..1334a7140f4 100644 --- a/extension/cuda/CMakeLists.txt +++ b/extension/cuda/CMakeLists.txt @@ -16,14 +16,24 @@ if(NOT EXECUTORCH_ROOT) set(EXECUTORCH_ROOT ${CMAKE_CURRENT_SOURCE_DIR}/../..) endif() -find_package(CUDAToolkit REQUIRED) - # SHARED on purpose: the caller-stream thread-local must have a single # definition across every shared object in the process (see export.h). A static # copy linked into multiple shared libraries would create multiple thread-locals # and silently break the caller-stream handshake. add_library(extension_cuda SHARED caller_stream.cpp) -target_link_libraries(extension_cuda PUBLIC CUDA::cudart) +if(EXECUTORCH_BUILD_SHARED) + # Named after what it provides rather than after the target, matching the + # other libraries the wheel ships. The target name stays as it is because the + # rest of the build already refers to it. + set_target_properties( + extension_cuda PROPERTIES OUTPUT_NAME executorch_extension_cuda + ) + executorch_target_soname_policy(extension_cuda) +endif() +# No CUDA headers or libraries: caller_stream.cpp uses cudaStream_t as an opaque +# handle and calls no CUDA function, so it compiles without cuda_runtime.h and +# needs no libcudart. Anything else that links this library and does call CUDA +# links the runtime itself. target_include_directories(extension_cuda PUBLIC ${_common_include_directories}) target_compile_options( extension_cuda PUBLIC "$<$:${_common_compile_options}>" diff --git a/extension/cuda/caller_stream.h b/extension/cuda/caller_stream.h index a13b7a9b396..825afa14345 100644 --- a/extension/cuda/caller_stream.h +++ b/extension/cuda/caller_stream.h @@ -8,12 +8,19 @@ #pragma once -#include #include #include #include +// Declared here rather than including , so this header compiles +// against a distribution that ships it without a CUDA toolkit. A stream is only +// ever stored and handed back below, never dereferenced, so the handle is all +// this interface needs. Repeating the declaration CUDA itself makes is +// well-formed, so a consumer that also includes is unaffected, +// in either include order. +typedef struct CUstream_st* cudaStream_t; + namespace executorch::extension::cuda { /** diff --git a/install_utils.py b/install_utils.py index 5bbd3eeac73..eb40a851647 100644 --- a/install_utils.py +++ b/install_utils.py @@ -9,6 +9,7 @@ import os import platform import re +import shlex import subprocess import sys from typing import List, Optional @@ -105,8 +106,10 @@ def _get_cuda_version(): """ try: # Get CUDA version from nvcc (CUDA compiler) + # Same selection rule as _detected_cuda_major, so the two cannot disagree about which + # toolkit this build uses. nvcc_result = subprocess.run( - ["nvcc", "--version"], capture_output=True, text=True, check=True + _selected_nvcc(), capture_output=True, text=True, check=True ) # Parse nvcc output for CUDA version # Output contains line like "Cuda compilation tools, release 12.6, V12.6.68" @@ -141,6 +144,70 @@ def _get_cuda_version(): ) +def _selected_nvcc() -> List[str]: + """The nvcc command line to ask for a version, matching what the build will use. + + Reading the bare command described whichever toolkit was on PATH, while the build also honours + these variables. Packaging then declared the runtime for one toolkit while compiling against + another, or declared nothing at all when the selected compiler was not on PATH. + """ + # CMake reads -DCMAKE_CUDA_COMPILER from the command line and CUDACXX from the environment. It does + # NOT read an environment variable named CMAKE_CUDA_COMPILER, measured with cmake 3.31.8, so asking + # the environment for that name first described a compiler the build would never use. + explicit = _extract_cmake_define(_cmake_args_from_env(), "CMAKE_CUDA_COMPILER") + if not explicit: + explicit = os.environ.get("CUDACXX") + if explicit: + return [explicit, "--version"] + # CUDA_PATH is honoured by CMake's own compiler search, and /usr/local/cuda is the route + # FindCUDAToolkit resolves through when nothing else is set. Skipping both meant packaging could + # report no toolkit while the build compiled with one, which disables the mismatch guard. + for root in ( + os.environ.get("CUDAToolkit_ROOT"), + os.environ.get("CUDA_PATH"), + "/usr/local/cuda", + ): + if not root: + continue + candidate = os.path.join(root, "bin", "nvcc") + if os.path.exists(candidate): + return [candidate, "--version"] + return ["nvcc", "--version"] + + +def _cmake_args_from_env() -> List[str]: + """CMAKE_ARGS split into arguments, tolerating an unbalanced quote. + + shlex is the right parser for a value naming a shell argument list, but it raises on an unbalanced + quote, and a path containing an apostrophe is enough to trigger it. + """ + raw = os.environ.get("CMAKE_ARGS", "") + try: + return shlex.split(raw) + except ValueError: + return raw.split() + + +@functools.lru_cache(maxsize=1) +def _detected_cuda_major() -> Optional[int]: + """The CUDA major version of the installed toolkit, or None if none is installed. + + Kept separate from `_get_cuda_version` because the mismatch guard in the wheel build + needs the major regardless of whether the exact (major, minor) is listed in + SUPPORTED_CUDA_VERSIONS. Reading through the validator caused the guard to see an + empty detection for any unlisted minor (say 12.8), so a cu130 row built on a CUDA 12 + toolkit produced a wheel with no error. + """ + try: + result = subprocess.run( + _selected_nvcc(), capture_output=True, text=True, check=True + ) + except (FileNotFoundError, subprocess.CalledProcessError, OSError): + return None + match = re.search(r"release (\d+)\.\d+", result.stdout) + return int(match.group(1)) if match else None + + def _extract_cmake_define(args: List[str], name: str) -> Optional[str]: """The value CMake would use for -D, which is the last one given. diff --git a/setup.py b/setup.py index 9870fb5cf09..bf2fec8c148 100644 --- a/setup.py +++ b/setup.py @@ -55,6 +55,7 @@ import logging import os import re +import shlex import shutil import site import stat @@ -62,7 +63,7 @@ import sys from distutils import log # type: ignore[import-not-found] from distutils.sysconfig import get_python_lib # type: ignore[import-not-found] -from pathlib import Path +from pathlib import Path, PurePosixPath from typing import List, Optional # Clean dynamic import using importlib @@ -205,6 +206,297 @@ def _minimal_packages() -> List[str]: ) +# The published project names for the CUDA runtime components a CUDA wheel links but +# does not bundle, keyed by CUDA major version. Not derivable from a suffix rule: the +# CUDA 12 wheels carry a "-cu12" suffix while the CUDA 13 ones are published under +# unsuffixed names. A train with no entry here declares nothing rather than guessing a +# name that may not exist. +# +# Only what a shipped library actually loads. Measured on a built wheel, the CUDA libraries need +# the CUDA runtime and cuRAND and nothing else, and the generated model library embeds its kernels +# rather than compiling them at run time, so there is no runtime compiler to satisfy either. +_CUDA_RUNTIME_PACKAGES = { + "12": ( + "nvidia-cuda-runtime-cu12", + "nvidia-curand-cu12", + ), + "13": ( + "nvidia-cuda-runtime", + "nvidia-curand", + ), +} + +# Where each train installs its libraries under site-packages. CUDA 13 collects them in +# one directory while CUDA 12 gives each component its own, so the search path differs by +# train and cannot be a single literal. +# +# Every declared package needs its directory here, and nothing else belongs. The loader only +# searches what is recorded here, so a missing directory leaves a shipped library unable to find +# a package that is installed, and an extra one implies a dependency the wheel does not have. +_CUDA_LIBRARY_DIRECTORIES = { + "12": ( + "nvidia/cuda_runtime/lib", + "nvidia/curand/lib", + ), + "13": ("nvidia/cu13/lib",), +} + + +def _cmake_args() -> List[str]: + """CMAKE_ARGS split into arguments, tolerating an unbalanced quote. + + shlex is the correct parser for a value that names a shell argument list, but it raises on an + unbalanced quote, and a path containing an apostrophe is enough to trigger it. Both callers run at + module scope, so the exception surfaced as a traceback during the build rather than a diagnosable + error. Falling back to whitespace splitting keeps the build working for the case that caused it. + """ + raw = os.environ.get("CMAKE_ARGS", "") + try: + return shlex.split(raw) + except ValueError: + return raw.split() + + +def _row_is_cpu_only() -> bool: + """Whether the release row this build belongs to names itself a CPU row. + + The metadata side already reads the row to decide which NVIDIA packages to declare, so the build + has to read the same input or the two disagree and the wheel ships a delegate it cannot load. + Absent means unknown rather than CPU, which keeps a plain local build behaving as before. + """ + raw = ( + (os.environ.get("CU_VERSION") or os.environ.get("DESIRED_CUDA") or "") + .strip() + .lower() + ) + return raw == "cpu" + + +def _cuda_train() -> str: + """The CUDA major version this wheel is being built for, or "" for a CPU wheel. + + The release row's own field wins when it is set, because a row states the train it + targets and that is more authoritative than whichever toolkit happens to sit on the + builder. The wheel build exports CU_VERSION; DESIRED_CUDA is the matrix field name. + + Falling back to the installed toolkit matters for every build that is not a release + job. The build turns CUDA on by detecting a toolkit, so keying only off the release + field produced a wheel that carried the CUDA libraries with no dependency declarations + and no way to find the CUDA runtime. + + Returns "" when the build did not enable CUDA, so a CPU wheel declares nothing even on + a machine that has a toolkit installed. + + Raises when a release row names a train the installed toolkit does not provide. The + declared packages and the loader paths both come from this value, so disagreeing with + the toolkit that compiled the libraries produces a wheel that installs cleanly and then + cannot load: a cu126 row built against a 13.0 toolkit declares the CUDA 12 runtime for + binaries that need libcudart.so.13. + """ + # An explicit OFF first, ahead of the release field. A CPU row on a builder that has a + # toolkit installed sets both, so reading the row field first would declare a runtime + # the wheel never loads. + if not install_utils.is_cmake_option_on( + _cmake_args(), + "EXECUTORCH_BUILD_CUDA", + default=True, + ): + return "" + + raw = os.environ.get("CU_VERSION") or os.environ.get("DESIRED_CUDA") or "" + # A row spelled "cpu" is a CPU row regardless of CMAKE_ARGS. Recognised here so that a + # local build named that way with no CMAKE_ARGS set does not fall through and raise on + # the unsupported-train branch below. + if raw.lower() in ("cpu", "cpu-aarch64"): + return "" + # Reduce to digits and match against the same (major, minor) trains the shell classifier + # uses. Previously this took the first two digits and matched against major only, so a + # row spelled with an unsupported minor (say cu125) was classified CPU by the shell and + # CUDA 12 here, and the wheel then declared CUDA runtime packages for a CPU build. + digits = re.sub(r"[^0-9]", "", raw) + trains = { + f"{major}{minor}": str(major) + for major, minor in install_utils.SUPPORTED_CUDA_VERSIONS + } + requested = trains.get(digits, "") + + # Read the toolkit major directly, without the (major, minor) validator, so the guard + # below fires on any mismatch rather than only on the three listed pairs. + detected_major = install_utils._detected_cuda_major() + detected = ( + str(detected_major) + if detected_major is not None and str(detected_major) in _CUDA_RUNTIME_PACKAGES + else "" + ) + + if requested: + # A row that names a train has to be buildable for that train. Reported here + # rather than left to produce a mismatched wheel, because nothing downstream + # compares the two: the metadata comes from the row and the binaries come from + # the toolkit. + if detected and detected != requested: + raise RuntimeError( + f"this build targets CUDA {requested} (from " + f"{'CU_VERSION' if os.environ.get('CU_VERSION') else 'DESIRED_CUDA'}=" + f"{raw!r}) but the installed toolkit is CUDA {detected}. The declared " + "runtime packages and the loader search paths come from the requested " + "train while the libraries are compiled by the installed one, so the " + "wheel would install and then fail to load. Install a matching toolkit " + "or build the row that matches this one." + ) + return requested + + if raw and not requested: + # A row named something this packaging does not recognise. Silently reporting the + # builder's toolkit instead contradicts "the row's field wins" and produced a + # wheel tagged for one train carrying another. + supported = ", ".join( + f"cu{major}{minor}" + for major, minor in install_utils.SUPPORTED_CUDA_VERSIONS + ) + raise RuntimeError( + f"the release row requests CUDA {raw!r}, which is not a train this project " + f"supports ({supported}). Add it to SUPPORTED_CUDA_VERSIONS in install_utils " + "and to _CUDA_RUNTIME_PACKAGES and _CUDA_LIBRARY_DIRECTORIES here, or build " + "a supported row. Falling back to whatever toolkit this builder has would tag " + "the wheel for one train and fill it with another." + ) + + # Fall back to the installed toolkit, because keying this off a release variable alone produced a wheel + # that carried the CUDA libraries while declaring no CUDA runtime and recording no way to reach one. + # + # Two ways CUDA gets built, and both have to agree with what is declared here. The build gate turns it + # on when a SUPPORTED train is installed, so a toolkit whose minor is unlisted builds CPU-only and + # declaring runtime packages for it would make a CPU wheel demand four CUDA wheels. An explicit ON + # bypasses that gate and reaches CMake directly, where find_package(CUDAToolkit) accepts a toolkit + # this packaging does not list, so the libraries ship and the runtime has to be declared for them. + # Asking only whether the train is supported got the first case right and the second wrong. + explicit_on = install_utils.is_cmake_option_on( + _cmake_args(), + "EXECUTORCH_BUILD_CUDA", + default=False, + ) + if not install_utils.is_cuda_available() and not explicit_on: + return "" + return detected + + +def _cuda_libraries_built(cmake_cache_dir: Optional[str]) -> bool: + """Whether this build produced the CUDA libraries, read from the CMake cache. + + The build turns CUDA on from the cache, so the cache is the fact that decides what ships. The + release row's CUDA version is a different question: a build on a toolkit whose train this packaging + does not recognise still produces the libraries while declaring no train, and gating anything else on + the train left that wheel carrying libraries with no matching header. + + Falls back to the train when no cache is readable, which is the case for a source distribution where + nothing was built here anyway. + """ + cache_path = os.path.join(cmake_cache_dir or "", "CMakeCache.txt") + if os.path.exists(cache_path): + return CMakeCache(cache_path=cache_path).is_enabled("EXECUTORCH_BUILD_CUDA") + return bool(_cuda_train()) + + +def _cuda_dependencies() -> List[str]: + """Runtime libraries a CUDA wheel needs but does not bundle. + + Declared rather than vendored, the way the PyTorch CUDA wheels do it, so one copy is + shared with torch instead of shipping a second one. + """ + train = _cuda_train() + # Marked for Linux, because a CUDA wheel is only built there and these nvidia wheels publish no + # distribution for the other platforms, so an unmarked requirement would make a source install + # elsewhere fail on a dependency it cannot satisfy and does not need. + return [ + f"{name}; platform_system == 'Linux'" + for name in _CUDA_RUNTIME_PACKAGES.get(train, ()) + ] + + +# Directories inside the wheel that hold libraries a shipped library links, relative to the package +# root rather than to the linking library, because the wheel ships libraries at more than one depth. +# +# The CUDA libraries are split across two directories and reference each other in both directions: +# the delegate in lib/ links the shims library in backends/cuda/, and the shims library links the +# stream helper back in lib/. So both hops are needed. +# +# Applied to every shipped library rather than mapping each library to the directories it happens to +# need. An unused hop costs nothing at load time, while a missing one produces a wheel that installs +# and then fails to load, and a per-library mapping would have to be revisited every time a library +# moves. +_SIBLING_LIBRARY_DIRECTORIES = ("backends/cuda", "lib") + + +def _sibling_library_search_paths(depth: int = 1) -> List[str]: + """Loader paths that reach another directory inside this same package. + + `depth` is how many directories separate the linking library from the package root, and it has to + be honoured for the same reason the CUDA hops honour it: the wheel ships libraries at depth one + (lib/) and depth two (backends/cuda/, extension/pybindings/ and others). Measured with a fixed + pair sized for one depth, six of twelve hops landed somewhere that does not exist, and the hop + from lib/ escaped the package entirely into a sibling of it, where an unrelated library with a + matching SONAME could satisfy the dependency first. + """ + up = "/".join([".."] * depth) + return [f"$ORIGIN/{up}/{directory}" for directory in _SIBLING_LIBRARY_DIRECTORIES] + + +def _cuda_runtime_search_paths(depth: int = 1) -> List[str]: + """Loader paths that reach the CUDA wheels installed beside this one. + + Those wheels install as siblings of this package, so the hop has to climb out of the package first. + `depth` is how many directories separate the library from the package root, and the wheel ships + libraries at more than one depth: a hop sized for one of them lands inside this package from the + other, where nothing is found. + """ + train = _cuda_train() + out = "/".join([".."] * (depth + 1)) + return [ + f"$ORIGIN/{out}/{directory}" + for directory in _CUDA_LIBRARY_DIRECTORIES.get(train, ()) + ] + + +def _is_cuda_toolkit_directory(entry: str) -> bool: + """Whether a runtime search path entry names a library directory inside a CUDA toolkit. + + Matched on the two layouts a toolkit actually installs rather than on the word "cuda" appearing + somewhere above the directory. Scanning a window of components dropped a torch directory whose build + root happened to be named after a CUDA version, and torch's directory is the one absolute path a + shipped library has to keep. + + Position alone cannot separate the two, because a real targets layout puts the cuda-named component at + the same depth a build root does, so each layout is spelled out instead. + """ + parts = [part.lower() for part in PurePosixPath(entry).parts] + if not parts or parts[-1] not in ("lib", "lib64"): + return False + + def cuda_named(part: str) -> bool: + return bool(re.fullmatch(r"cuda(?:-\d+(?:\.\d+)*|[-_]?toolkit)?", part)) + + # /lib64 + if len(parts) >= 2 and cuda_named(parts[-2]): + return True + # /targets//lib + return len(parts) >= 4 and parts[-3] == "targets" and cuda_named(parts[-4]) + + +def _package_relative_depth(library: Path) -> int: + """How many directories separate a shipped library from the installed package root. + + Searched from the END of the path. At build time the path is absolute and a source checkout is + often named after the package too, so taking the first match found the checkout instead of the + package inside the build output and produced a hop that climbs out of the install directory. + """ + parts = list(Path(library).parts) + if "executorch" not in parts: + return 1 + index = len(parts) - 1 - parts[::-1].index("executorch") + return max(len(parts) - index - 2, 0) + + def _base_dependencies() -> List[str]: """Runtime dependencies for the full wheel. @@ -367,6 +659,20 @@ def get_dynamic_lib_name(name: str) -> str: return f"lib{name}.so" +def _dynamic_lib_suffix() -> str: + """The loadable-library suffix on this platform, including the dot. + + Separate from get_dynamic_lib_name because a file whose prefix is not known + ahead of time still needs the suffix named: globbing the suffix as well would + also match an import library, an exports file, or a soname's versioned links. + """ + if _is_windows(): + return ".dll" + if _is_macos(): + return ".dylib" + return ".so" + + def get_executable_name(name: str) -> str: if _is_windows(): return name + ".exe" @@ -730,10 +1036,85 @@ def build_extension(self, ext: _BaseExtension) -> None: if not os.access(dst_file, os.W_OK): os.chmod(dst_file, os.stat(dst_file).st_mode | stat.S_IWUSR) - _strip_absolute_runtime_paths(dst_file) + cmake_cache_dir = getattr( + self.get_finalized_command("build"), "cmake_cache_dir", None + ) + _strip_absolute_runtime_paths(dst_file, _cuda_libraries_built(cmake_cache_dir)) -def _strip_absolute_runtime_paths(library: Path) -> None: +def _append_relative_search_paths(entries: List[str], depth: int = 1) -> None: + """Add the relative hops a shipped library needs, skipping any already present. + + Two kinds, both relative so the wheel works wherever the environment lives: + the CUDA runtime, which arrives in its own wheel installed beside this one, and a sibling + ExecuTorch library that the wheel installs in a different directory from the library linking it. + + `depth` sizes the hop out of this package, since the wheel ships libraries at more than one depth. + """ + for search_path in ( + *_cuda_runtime_search_paths(depth), + *_sibling_library_search_paths(depth), + ): + if search_path not in entries: + entries.append(search_path) + + +def _is_usable_runtime_path( + entry: str, + safe_to_drop_toolkit_paths: bool, + has_relative_torch_route: bool, +) -> bool: + if not entry: + # The loader reads an empty entry as the process working directory. + return False + if not entry.startswith("/"): + return True + # Absolute, so decide by what it points at. + # + # A directory inside this build cannot exist for a user. + # + # A CUDA toolkit directory is dropped for a different reason: the wheel declares the CUDA runtime + # as a dependency and reaches it through a relative hop, so an absolute toolkit path is both + # unnecessary and harmful. It sits ahead of the hop, so a user who happens to have a toolkit at + # that prefix resolves the runtime from there instead of from the declared dependency. + # + # Whether that is safe is decided above, because the one case it is not is a CUDA build on an + # unrecognised train, which has no hop to fall back on. + # + # A torch lib directory is dropped when a relative route to torch is already recorded, since + # that route is what resolves torch on an installed wheel while the absolute one only names a + # directory from the machine that built it. It is kept when no relative route exists, because + # then it is the only way this library finds torch. + # + # Anything else absolute stays, because it is a dependency the environment provides and the wheel + # has no relative answer for. + # + # The build directories are matched as whole path components rather than as substrings. A bare + # "/cmake-out" also matches "/home/user/cmake-outputs/torchlibs", which is an unrelated directory + # a user could really have, and stripping it breaks a dependency the library resolves there. + # The setuptools staging directory is spelled build/lib.-, for example + # lib.linux-x86_64-cpython-312, so match the whole shape rather than any part starting "lib.". + if any( + part in ("pip-out", "cmake-out") + or re.fullmatch(r"lib\.[^/]+-(cpython-\d+|\d+(?:\.\d+)*)", part) + for part in entry.split("/") + ): + return False + # Matched on the layout a CUDA toolkit actually installs, not on the word "cuda" anywhere in the + # path. A substring test dropped a torch directory that merely sat under a directory named after a + # CUDA version, which is the one absolute path that has to survive. + if safe_to_drop_toolkit_paths and _is_cuda_toolkit_directory(entry): + return False + if entry.rstrip("/").endswith("/torch/lib") and has_relative_torch_route: + return False + return True + + +# Whether the library can still reach torch without the absolute entry. Read inside keep, +# which closes over this scope. + + +def _strip_absolute_runtime_paths(library: Path, ships_cuda: bool) -> None: """Remove unusable runtime search paths from a library the wheel ships. These libraries are copied out of the build tree rather than installed, so they @@ -782,48 +1163,33 @@ def _strip_absolute_runtime_paths(library: Path) -> None: # entry, so there is nothing to distinguish here and nothing to do either way. return - def keep(entry: str) -> bool: - if not entry: - # The loader reads an empty entry as the process working directory. - return False - if not entry.startswith("/"): - return True - # Absolute, so decide by what it points at. A directory inside this build - # cannot exist for a user. - # - # A torch lib directory is dropped when a relative route to torch is already - # recorded, since that route is what resolves torch on an installed wheel - # while the absolute one only names a directory from the machine that built - # it. It is kept when no relative route exists, because then it is the only - # way this library finds torch. - # - # Anything else absolute is a dependency the environment provides and the - # wheel has no relative answer for. - # - # Matched as whole path components rather than as substrings. A bare - # "/cmake-out" also matches "/home/user/cmake-outputs/torchlibs", which is - # an unrelated directory a user could really have, and stripping it breaks - # a dependency the library legitimately resolves there. - if entry.rstrip("/").endswith("/torch/lib") and has_relative_torch_route: - return False - parts = entry.split("/") - # The setuptools staging directory is spelled build/lib.-, - # for example lib.linux-x86_64-cpython-312. A bare startswith("lib.") also - # stripped a real user path like /opt/acme/lib.v2, so match the whole shape. - return not any( - part == "pip-out" - or part == "cmake-out" - or re.fullmatch(r"lib\.[^/]+-(cpython-\d+|\d+(?:\.\d+)*)", part) - for part in parts - ) + # Whether dropping an absolute CUDA toolkit path is safe. It is a cleanup when a relative hop replaces + # it, and also when this wheel carries no CUDA at all, because then nothing in it loads from that + # directory and the path only names the build machine. It is a regression only for a CUDA build whose + # train this packaging does not recognise, which declares no dependency and adds no hop, so dropping + # the path there would leave the delegate with no route to libcudart. + safe_to_drop_toolkit_paths = not ships_cuda or bool( + _cuda_runtime_search_paths(_package_relative_depth(library)) + ) - # Whether the library can still reach torch without the absolute entry. Read - # inside keep, which closes over this scope. has_relative_torch_route = any( not entry.startswith("/") and entry.rstrip("/").endswith("/torch/lib") for entry in original.split(":") ) - rewritten = ":".join(entry for entry in original.split(":") if keep(entry)) + entries = [ + entry + for entry in original.split(":") + if _is_usable_runtime_path( + entry, safe_to_drop_toolkit_paths, has_relative_torch_route + ) + ] + # A CUDA wheel links the CUDA runtime from a separate wheel installed beside this + # one, so the loader needs a relative hop to reach it. Without this the library + # resolves the runtime only through the absolute toolkit path the linker recorded, + # which names the build machine and will not exist for a user who installed from an + # index. Appended, so a path already present keeps its position. + _append_relative_search_paths(entries, _package_relative_depth(library)) + rewritten = ":".join(entries) if rewritten == original: return subprocess.run( @@ -889,6 +1255,9 @@ def run(self): ("schema/program.fbs", "exir/_serialize/program.fbs"), ] if not _is_minimal_build(): + cmake_cache_dir = getattr( + self.get_finalized_command("build"), "cmake_cache_dir", None + ) src_to_dst += [ ( "devtools/bundled_program/schema/bundled_program_schema.fbs", @@ -959,7 +1328,19 @@ def run(self): "devtools/etdump/emitter.h", "devtools/etdump/utils.h", "devtools/etdump/data_sinks/", - ]: + ] + ( + # The CUDA stream helper's public header, and the export macros it includes. Its library is + # shared so the process has one copy of the caller-stream state, and that is a handshake the + # caller takes part in, so a consumer needs the declarations to take part at all. + # + # Only when this wheel carries the CUDA delegate, and decided from the same CMake cache the + # libraries ship on. Keying it off the release row's CUDA version instead meant a build on + # an unrecognised toolkit shipped both CUDA libraries and both CMake components with no + # header, so a consumer got a component it could link and not include. + ["extension/cuda/caller_stream.h", "extension/cuda/export.h"] + if _cuda_libraries_built(cmake_cache_dir) + else [] + ): # A directory entry publishes everything under it, and a file entry publishes # just that file. Some directories hold headers a consumer cannot compile # against, so those are named individually rather than swept in. @@ -1166,11 +1547,30 @@ def run(self): # noqa C901 if minimal_build: cmake_configuration_args += _minimal_cmake_flags() - # Check if CUDA is available, and if so, enable building the CUDA - # backend by default. + # A row that names a CUDA train has already declared the NVIDIA runtime packages in + # install_requires, which is set before any build runs and therefore cannot consult the + # build. If the toolkit is not reachable here, the wheel would ship no CUDA library while + # still making pip fetch the CUDA runtime, so stop rather than publish that mismatch. + if ( + not minimal_build + and _cuda_train() + and not install_utils.is_cuda_available() + ): + raise RuntimeError( + "this row names a CUDA train but no usable CUDA toolkit was found, so the wheel " + "would declare the CUDA runtime and ship no CUDA library. Point CUDACXX or " + "CUDAToolkit_ROOT at a toolkit, or build the CPU row instead." + ) + + # Enable the CUDA delegate when a toolkit is present, unless the release row says this is a + # CPU wheel. Without that second condition a CPU row built on a machine that happens to have + # a toolkit produced a wheel carrying the delegate while its metadata declared no NVIDIA + # package and no way to reach one, so the delegate could not load. An explicit option still + # wins, so a caller can override the row on purpose. if ( not minimal_build and install_utils.is_cuda_available() + and not _row_is_cpu_only() and install_utils.is_cmake_option_on( cmake_configuration_args, "EXECUTORCH_BUILD_CUDA", default=True ) @@ -1303,6 +1703,10 @@ def run(self): # noqa C901 if cmake_cache.is_enabled("EXECUTORCH_BUILD_CUDA"): cmake_build_args += ["--target", "aoti_cuda_backend"] cmake_build_args += ["--target", "aoti_common_shims_slim"] + if cmake_cache.is_enabled("EXECUTORCH_BUILD_SHARED"): + # The stream helper ships as its own library so a process has one + # of it. Named because nothing else in a wheel build links it. + cmake_build_args += ["--target", "extension_cuda"] if cmake_cache.is_enabled("EXECUTORCH_BUILD_EXTENSION_MODULE"): cmake_build_args += ["--target", "extension_module"] @@ -1354,7 +1758,9 @@ def run(self): # noqa C901 setup_kwargs["packages"] = _minimal_packages() setup_kwargs["install_requires"] = _minimal_dependencies() else: - setup_kwargs["install_requires"] = _base_dependencies() + # A CUDA wheel links the CUDA runtime but does not bundle it, so the wheels that + # carry it are declared here. A CPU wheel adds nothing. + setup_kwargs["install_requires"] = _base_dependencies() + _cuda_dependencies() setup( @@ -1443,6 +1849,39 @@ def run(self): # noqa C901 "EXECUTORCH_BUILD_KERNELS_OPTIMIZED", ], ), + # The CUDA delegate and the process-wide CUDA stream helper, for a + # wheel built from a CUDA index. Only present when the build asks for + # CUDA, so packaging requires that rather than looking for files a + # CPU-only build never produced. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/backends/cuda/", + src_name="libexecutorch_backend_cuda.so", + dst="executorch/lib/libexecutorch_backend_cuda.so", + dependent_cmake_flags=[ + "EXECUTORCH_BUILD_SHARED", + "EXECUTORCH_BUILD_CUDA", + ], + ), + # The stream helper the delegate and the shim layer both record as a + # dependency. Globbed rather than named, because the file name depends on + # the build: a shared build renames it to libexecutorch_extension_cuda.so + # to match the other shipped components, and any other build leaves it as + # libextension_cuda.so. The shim ships whenever CUDA is on, so naming only + # the shared spelling left the non-shared build shipping a shim whose + # DT_NEEDED resolved to nothing. Two names means is_dynamic_lib cannot be + # used, since it builds one name and prepends a prefix the shared spelling + # does not have, so the prefix is globbed and the suffix is named. The + # build type is in the directory the way the sibling entries have it. + # Naming the suffix matters: this entry accepts exactly one file, and a + # bare wildcard also matches what a build leaves beside the library, an + # import library and an exports file on MSVC, or a soname's versioned + # links, and packaging then fails on a layout that is perfectly valid. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/extension/cuda/%BUILD_TYPE%/", + src_name="*extension_cuda" + _dynamic_lib_suffix(), + dst="executorch/lib/", + dependent_cmake_flags=["EXECUTORCH_BUILD_CUDA"], + ), # The quantized kernels, as their own library rather than code # fused into the AOT-only extension beside the Python bindings. # A C++ application running a quantized model could not link @@ -1549,23 +1988,8 @@ def run(self): # noqa C901 is_dynamic_lib=True, dependent_cmake_flags=["EXECUTORCH_BUILD_CUDA"], ), - # The stream helper the library above records as a dependency. It was - # never shipped, and resolved only because the copied library still - # carried the absolute directory it was linked in, which exists on a - # build machine and nowhere else. Stripping that path is what made the - # omission visible as a failed import. - # - # Shipped beside its consumer rather than in lib/, because that - # directory only exists in the shared build and this has to work - # without it. The glob covers both names the target can have: the - # shared build renames it to advertise it as a wheel component. - BuiltFile( - src_dir="%CMAKE_CACHE_DIR%/extension/cuda/%BUILD_TYPE%/", - src_name="*extension_cuda", - dst="executorch/backends/cuda/", - is_dynamic_lib=True, - dependent_cmake_flags=["EXECUTORCH_BUILD_CUDA"], - ), + # The stream helper this library needs ships in lib/ from here on, + # alongside the other components a C++ consumer links. BuiltFile( src_dir="%CMAKE_CACHE_DIR%/backends/qualcomm/%BUILD_TYPE%/", src_name="qnn_executorch_backend", diff --git a/tools/cmake/executorch-wheel-config.cmake b/tools/cmake/executorch-wheel-config.cmake index ffa2e2132c3..d34bf4198d6 100644 --- a/tools/cmake/executorch-wheel-config.cmake +++ b/tools/cmake/executorch-wheel-config.cmake @@ -292,9 +292,11 @@ if(_executorch_runtime_library AND NOT _executorch_targets_supported) # route has no per-component target to opt into, so they are offered through # EXECUTORCH_QUANTIZED_KERNELS_LIBRARY instead and a consumer that wants them # links that as well. - foreach(_executorch_component IN - ITEMS libexecutorch_kernels_optimized libexecutorch_backend_xnnpack - libexecutorch_threadpool libexecutorch_etdump + foreach( + _executorch_component IN + ITEMS libexecutorch_kernels_optimized libexecutorch_backend_xnnpack + libexecutorch_backend_cuda libexecutorch_extension_cuda + libexecutorch_threadpool libexecutorch_etdump ) _executorch_find_library( _executorch_component_library "${_executorch_component}" @@ -592,6 +594,11 @@ if(TARGET executorch::runtime AND TARGET executorch::threadpool) endif() _executorch_define_component(backend_xnnpack executorch_backend_xnnpack) +# The CUDA delegate and its stream helper, present only in a wheel built from a +# CUDA index. A CPU wheel defines neither, so a consumer asking for one is told +# while configuring. +_executorch_define_component(backend_cuda executorch_backend_cuda) +_executorch_define_component(extension_cuda executorch_extension_cuda) # Find prebuilt _portable_lib..so. This is the legacy contract used # to build custom-op extensions against the Python module, and is kept working diff --git a/tools/cmake/preset/pybind.cmake b/tools/cmake/preset/pybind.cmake index f72680836b7..068f80d1e2b 100644 --- a/tools/cmake/preset/pybind.cmake +++ b/tools/cmake/preset/pybind.cmake @@ -108,15 +108,7 @@ elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") # consumers link, so a process has a single backend registry. Linux only: # macOS C++ consumers are served by the Swift package distribution, and the # runtime has no export annotations for a Windows DLL. - # - # Not with the CUDA backend, whose libraries this build does not ship yet. The - # CUDA libraries currently reach the wheel carrying the absolute path of the - # directory they were linked in, which resolves only on the machine that built - # them. The shared build removes those paths, so enabling it here before the - # CUDA libraries ship would leave the extension unable to load at all. - if(NOT EXECUTORCH_BUILD_CUDA) - set_overridable_option(EXECUTORCH_BUILD_SHARED ON) - endif() + set_overridable_option(EXECUTORCH_BUILD_SHARED ON) elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows" OR CMAKE_SYSTEM_NAME STREQUAL "WIN32" ) From 8c477b53d8f22691fae58ec4d535f792aa2f40a3 Mon Sep 17 00:00:00 2001 From: shoumikhin Date: Tue, 11 Aug 2026 23:08:11 -0700 Subject: [PATCH 5/6] Build and publish CUDA wheels ## The problem The wheel can carry the CUDA delegate, but no published wheel contains one: there is no CUDA row in any workflow, so a GPU user has to build from source. ## The change Add the workflows that build and publish CUDA wheels for Linux x86_64 and aarch64, and a smoke test that checks each wheel from the artifact itself. The build machines for these rows have no GPU, so the smoke test does not execute a model; it verifies the CUDA libraries are present, that the declared runtime matches the wheel's CUDA version, that nothing resolves through the build machine's toolkit, and that the shipped device code covers every GPU architecture the row claims. ``` executorch-1.5.0-cp312-cp312-manylinux_2_28_x86_64.whl +cu130 ``` A release publishes CUDA 12.6, 13.0 and 13.2, for Python 3.10 through 3.13. A pull request builds a single row instead of all twelve, because a full matrix costs hours for little extra signal. Which GPU architectures each row compiles for is chosen per row rather than detected on the builder. Detecting it would produce a wheel carrying device code for whatever machine happened to build it, which installs fine and then fails at the first GPU call. The aarch64 CUDA 12.6 row also compiles for compute capability 8.7, which is an embedded module. Every other row lists only the architectures the published PyTorch build for that train covers, and by that rule 8.7 would be left out, because the generic aarch64 build of this train carries 8.0 and 9.0 only. It is included because this is the only row whose CUDA major version matches what that module's software release ships, and because this wheel declares no PyTorch dependency: a user there supplies the build that carries their architecture. Leaving 8.7 out does not protect them from a bad pairing, it only removes the device code they need. Without it, a model reaching one of the shipped optional operators, quantized matrix multiply, sort or random number generation, fails at the first launch on that device. Two guards keep a release honest: - if the shared matrix generator stops offering a combination this policy advertises, the step fails instead of quietly publishing fewer wheels. A missing job is otherwise a green check for a wheel that was never built. - if a row reaches the architecture list with no CUDA version, the build refuses rather than falling back to the builder's GPU. A TORCH_CUDA_ARCH_LIST that holds only named GPU families PyTorch accepts, such as "Hopper", now fails to configure instead of quietly leaving CMAKE_CUDA_ARCHITECTURES unset and taking the compiler default. The three named forms CMake itself understands, "native", "all" and "all-major", are rejected before this logic runs: torch resolves the list with its own bundled CUDA architecture module, which does not know those names and stops the configure. That is upstream behaviour, not something this change introduces or can work around, so a caller has to name architectures explicitly. Windows CUDA is deliberately absent. The separate shared libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. ## Test plan - built the full release matrix, twelve wheels, and confirmed each one's contents match the row it claims: the CUDA libraries present, the CUDA runtime declared, and device code for every GPU architecture the row advertises. - ran a GPU model end to end from a CI-built wheel on three NVIDIA GPUs covering three device architectures, with output identical to eager PyTorch on each (largest absolute difference 0), and inspected the wheel for a fourth device it cannot execute on. - ran the matrix filter over generated inputs, including incomplete and malformed ones, and confirmed it refuses rather than publishing a partial release: a missing CUDA version, a missing python, or a python present on rows this policy does not build are each reported by name. - confirmed a CPU row still produces a CPU wheel on a builder that happens to have a CUDA toolkit installed. - the newest architecture also ships in its portable form, so a GPU newer than any in the row can still run by having the driver compile it at load time. Checked with `cuobjdump --list-ptx`, since `--list-elf` prints identical output whether or not the portable form is present. - every library that carries GPU device code covers the whole row on its own. - the declared CUDA packages are compared against the expected set in BOTH directions. A one-way comparison accepted a wheel that omitted required packages, and a name-suffix comparison accepted cross-train names because for CUDA 13 the suffix is empty. - the python axis is an allowlist, matching the CUDA axis. Testing only the disabled list let any python not on it through: a 3.9 row was emitted successfully. - `install_utils.py` is in both CUDA workflows' path filters. It owns the supported CUDA train list and the toolkit detection, so a change there previously ran no CUDA wheel job. - requesting the JetPack rows fails with its own reason instead of the generic empty-matrix message, since both of its lists are deliberately empty and no workflow asks for them. - torchao keeps its CUDA channel where that channel exists. Falling back to the plain nightly index was needed only on aarch64, where the CUDA channel publishes nothing, and doing it everywhere changed which torchao an x86_64 install resolves. - the CUDA smoke test now asserts the QnnBackend and OpenvinoBackend registrations that a CPU Linux row asserts. The CUDA build enables OpenVINO on every Linux architecture and downloads the QNN SDK on x86_64, so a CUDA wheel carries both backends; a previous premise that "a CUDA row is not built with them" was false, and dropping the checks meant those two backends were unverified on every CUDA wheel. Known gap: no automated job runs a CUDA model on real hardware before publication. Running a model on real hardware is a separate release-time step that a person owns today, not an automated job wired into these workflows. ghstack-source-id: 95d67c70e217a02053ec7683c0a0ec0709d0a9d7 ghstack-comment-id: 5220374521 Pull-Request: https://github.com/pytorch/executorch/pull/21668 --- .ci/scripts/wheel/cuda_arch_list.sh | 133 +++++++ .ci/scripts/wheel/envvar_cuda_linux.sh | 42 +++ .ci/scripts/wheel/test_cuda_linux.py | 353 ++++++++++++++++++ .ci/scripts/wheel/test_shared_libraries.py | 67 +++- .github/scripts/filter_cuda_matrix.py | 238 ++++++++++++ .../build-wheels-cuda-aarch64-linux.yml | 103 +++++ .github/workflows/build-wheels-cuda-linux.yml | 99 +++++ backends/cuda/CMakeLists.txt | 68 ++++ install_requirements.py | 15 +- 9 files changed, 1114 insertions(+), 4 deletions(-) create mode 100644 .ci/scripts/wheel/cuda_arch_list.sh create mode 100644 .ci/scripts/wheel/envvar_cuda_linux.sh create mode 100644 .ci/scripts/wheel/test_cuda_linux.py create mode 100644 .github/scripts/filter_cuda_matrix.py create mode 100644 .github/workflows/build-wheels-cuda-aarch64-linux.yml create mode 100644 .github/workflows/build-wheels-cuda-linux.yml diff --git a/.ci/scripts/wheel/cuda_arch_list.sh b/.ci/scripts/wheel/cuda_arch_list.sh new file mode 100644 index 00000000000..ed14bd9a2bd --- /dev/null +++ b/.ci/scripts/wheel/cuda_arch_list.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +# GPU architectures to compile device code for, chosen per release row rather than detected from +# the build machine. +# +# Without this nothing selects the architectures, so nvcc falls back to its own default and the +# wheel carries device code for that one architecture regardless of the builder's GPU. Measured: +# with the architecture list unset the compile line has no gencode flags at all. The wheel then +# installs on every machine the row claims and fails when a model runs on a different generation, +# with an error that looks like a model problem rather than a packaging one. Detection is the right +# default for a local build and the wrong one for a published artifact. +# +# The value is published as TORCH_CUDA_ARCH_LIST rather than CMAKE_CUDA_ARCHITECTURES, because +# PyTorch's CMake rejects the latter and overrides it, so setting only that reduces the build to a +# single detected architecture. + +# The architectures each row serves. Two rules decide the list, and they pull in opposite directions. +# +# The upper end follows the published PyTorch build for that train, read from its own library rather than +# chosen by reasoning about which GPUs matter. A delegate is only useful where torch already runs, and an +# architecture torch supports but this wheel omits produces a wheel that installs and then fails at the +# first kernel launch. Two omissions found that way were the GPU on the runner that tests these wheels, +# and a common desktop card. +# +# The lower end does NOT follow torch. It stops at 8.0 even though torch reaches further down, because one +# source here compiles an integer matrix-multiply path only at 8.0 and above. Below that a user gets a +# delegate that loads, runs most models, and fails on one needing that operator, which is worse than a row +# that never claimed the device. So these lists are narrower than torch at the bottom on purpose. +_cuda_arch_x86_64_cu130="8.0 8.6 8.9 9.0 10.0 12.0" +_cuda_arch_x86_64_cu132="${_cuda_arch_x86_64_cu130}" + +# The architectures the published aarch64 PyTorch CUDA build covers, read from its own library on an ARM +# machine, for the same reason as the x86_64 rows above. Includes the ARM module whose train matches. +_cuda_arch_aarch64_cu130="8.0 9.0 10.0 11.0 12.0" +_cuda_arch_aarch64_cu132="${_cuda_arch_aarch64_cu130}" + +# The older CUDA train. +# +# The two architectures do not carry identical lists, because each covers what the published PyTorch +# build for that architecture covers, and those differ. Matching them to each other instead would mean +# advertising a GPU on one architecture that PyTorch cannot serve there. +# +# The smaller embedded modules are deliberately absent, with one exception. An embedded-only +# architecture in a generic wheel would advertise a device the row cannot otherwise serve, since +# those devices also need the CUDA, TensorRT and PyTorch pinned by their own software release +# rather than the ones a generic wheel resolves. +# +# 8.7 is that exception. This is the only row whose CUDA major matches what that module's software +# release ships, and the wheel declares no PyTorch, so the user supplies the build that carries +# their architecture. Omitting it does not protect them from a bad pairing, it only removes the +# device code they need. +# +# The floor is 8.0 rather than the oldest architecture PyTorch still carries. One of these sources compiles +# an integer matrix-multiply path only at 8.0 and newer, so an older architecture would get a delegate that +# loads, runs most models, and fails on one that needs that operator. Claiming hardware the delegate only +# partly serves is the same problem the embedded modules have, so the row leaves it out for the same reason. +_cuda_arch_x86_64_cu126="8.0 8.6 8.9 9.0" +_cuda_arch_aarch64_cu126="8.0 8.7 9.0" + +# A CUDA train with no architecture list would leave the build detecting the builder's GPU, which is +# the failure this file exists to prevent. Adding a train to the release matrix without adding its +# architectures should fail loudly rather than silently produce a single-GPU wheel. +_executorch_unknown_train() { + echo "cuda_arch_list.sh: no GPU architecture list for CUDA train '$1' on $(uname -m)." >&2 + echo "Add one before building this row, or the wheel ships device code for one GPU only." >&2 + return 64 +} + +# The architectures for the current row, space separated in the dotted form PyTorch expects. +executorch_cuda_arch_list() { + local machine + machine="$(uname -m)" + # The wheel build exports the row's CUDA train as CU_VERSION. DESIRED_CUDA is the name of the + # matrix field rather than of the variable, so reading only that leaves every row falling back to + # detecting the builder's GPU. + local train="${CU_VERSION:-${DESIRED_CUDA:-}}" + # A CPU row names no CUDA train and needs no architectures, so it is not an error. + # + # A CUDA row always names one, so an empty value there means the row lost it. Treating that as a CPU + # row let the build fall back to detecting the builder's GPU, which produces a wheel carrying device + # code for whatever machine happened to build it while every check still reports green. + case "${train}" in + "" | cpu | CPU | none | NONE) + if [ "${EXECUTORCH_BUILD_CUDA:-}" = "1" ]; then + echo "this is a CUDA build but the row's CUDA version is '${train}', which names no CUDA" >&2 + echo "train. Refusing to detect the builder GPU instead." >&2 + return 65 + fi + return 0 + ;; + esac + # The value arrives as cu130, while some callers pass 13.0 instead. + train="${train#cu}" + train="${train//./}" + + case "${machine}" in + aarch64 | arm64) + case "${train}" in + 126) printf '%s' "${_cuda_arch_aarch64_cu126}" ;; + 130) printf '%s' "${_cuda_arch_aarch64_cu130}" ;; + 132) printf '%s' "${_cuda_arch_aarch64_cu132}" ;; + *) _executorch_unknown_train "${train}" ;; + esac + ;; + x86_64) + case "${train}" in + 126) printf '%s' "${_cuda_arch_x86_64_cu126}" ;; + 130) printf '%s' "${_cuda_arch_x86_64_cu130}" ;; + 132) printf '%s' "${_cuda_arch_x86_64_cu132}" ;; + *) _executorch_unknown_train "${train}" ;; + esac + ;; + *) _executorch_unknown_train "${train}" ;; + esac +} + +# The architecture list for a row. Every entry already carries the portable form that lets a newer +# GPU compile at load time: an unsuffixed architecture asks the compiler for both the compiled and +# the portable form, measured as code=[compute_120,sm_120] for a bare "120". Nothing extra is needed +# for forward compatibility. +executorch_cuda_arch_list_with_ptx() { + local dotted + # Propagate a failed lookup rather than reporting an empty list, since a caller cannot tell an + # unknown row from a CPU row and the unknown one must not pass silently. + dotted="$(executorch_cuda_arch_list)" || return $? + [ -n "${dotted}" ] || return 0 + printf '%s' "${dotted}" +} diff --git a/.ci/scripts/wheel/envvar_cuda_linux.sh b/.ci/scripts/wheel/envvar_cuda_linux.sh new file mode 100644 index 00000000000..d66ae3f2d22 --- /dev/null +++ b/.ci/scripts/wheel/envvar_cuda_linux.sh @@ -0,0 +1,42 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +# This file is sourced into the environment before building a pip wheel. It +# should typically only contain shell variable assignments. Be sure to export +# any variables so that subprocesses will see them. + +source "${GITHUB_WORKSPACE}/${REPOSITORY}/.ci/scripts/wheel/envvar_base.sh" + +# Ask for the CUDA delegate explicitly rather than letting the build detect a toolkit. A detected +# build is fine locally, but a release row states what it is producing, and a row that silently +# produced a CPU wheel because the toolkit was missing would publish under a CUDA name. +export EXECUTORCH_BUILD_CUDA=1 +export CMAKE_ARGS="${CMAKE_ARGS} -DEXECUTORCH_BUILD_CUDA=ON" + +# Fail the build if CUDA is not actually present. Without this the packaging step would look for +# CUDA libraries that were never built and report a confusing missing-file error several minutes +# after the real problem. +if [ ! -x "${CUDA_HOME:-/usr/local/cuda}/bin/nvcc" ]; then + echo "EXECUTORCH_BUILD_CUDA is set but no nvcc was found. This row cannot build a CUDA wheel." >&2 + exit 1 +fi + +# Compile device code for the GPUs this release row claims, rather than for whichever GPU the +# builder happens to have. A wheel built by detection alone installs on every machine the row covers +# and then fails when a model runs on a different generation. +source "${GITHUB_WORKSPACE}/${REPOSITORY}/.ci/scripts/wheel/cuda_arch_list.sh" +# The status is checked rather than only the output. An unrecognised row makes the lookup fail, and +# this file is sourced rather than run under a failing-command shell, so ignoring the status would +# leave the variable unset and let the build fall back to detecting the builder's own GPU. That is +# exactly the outcome this is meant to prevent, and it would ship quietly. +if ! _executorch_cuda_arch="$(executorch_cuda_arch_list_with_ptx)"; then + echo "could not resolve GPU architectures for CU_VERSION=${CU_VERSION:-unset}" >&2 + exit 1 +fi +if [ -n "${_executorch_cuda_arch}" ]; then + export TORCH_CUDA_ARCH_LIST="${_executorch_cuda_arch}" + echo "building device code for: ${TORCH_CUDA_ARCH_LIST}" +fi diff --git a/.ci/scripts/wheel/test_cuda_linux.py b/.ci/scripts/wheel/test_cuda_linux.py new file mode 100644 index 00000000000..44728a64860 --- /dev/null +++ b/.ci/scripts/wheel/test_cuda_linux.py @@ -0,0 +1,353 @@ +#!/usr/bin/env python +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +"""Smoke test for a CUDA wheel row. + +Runs the checks a GPU wheel needs, then the packaging, backend, and C++ SDK checks a CPU wheel +gets. The extra CUDA checks exist because a GPU wheel can install cleanly, import cleanly, and +still be unusable: + + the CUDA libraries can be absent while the wheel is still named as a CUDA build + the runtime dependency can be undeclared, so a user has nothing to resolve it from + the loader path can point at the build machine's toolkit, which no user has + the device code can cover no GPU the row claims, which only appears when a model runs + +This does not execute a model. The aarch64 rows have no GPU to execute one on, because their +validation runner has no accelerator, so for those rows inspection is all that is available +here. The x86_64 rows do run on a GPU runner, so a model-execution check is possible there and +its absence is a gap rather than a limit. What runs a model on real hardware before a +publication is a separate release-time step that a person owns today, not an automated job +wired into these workflows. +""" + +import os +import pathlib +import platform +import subprocess +import tempfile +from pathlib import Path + +import test_base +import test_cpp_sdk +import test_shared_libraries +from examples.models import Backend, Model + + +def _package_dir() -> Path: + import executorch + + return Path(executorch.__path__[0]) + + +def test_cuda_libraries_are_shipped() -> None: + """The row is named for CUDA, so the CUDA libraries have to be in it.""" + lib_dir = _package_dir() / "lib" + shipped = {path.name for path in lib_dir.iterdir()} if lib_dir.is_dir() else set() + expected = { + "libexecutorch_backend_cuda.so", + "libexecutorch_extension_cuda.so", + } + missing = sorted(expected - shipped) + assert not missing, ( + f"this is a CUDA row but {missing} are not in the wheel, so it would install as a " + f"CUDA build with no CUDA delegate. Shipped: {sorted(shipped)}" + ) + print(f"✓ the CUDA libraries ship ({len(expected)} of them)") + + +def test_cuda_runtime_is_declared() -> None: + """The wheel links the CUDA runtime without bundling it, so it must declare it. + + Without this a user installs the wheel and has nothing to resolve libcudart from, which + surfaces as a loader error at the first import rather than as a resolution failure at + install time. + """ + import importlib.metadata as metadata + + requirements = metadata.requires("executorch") or [] + cuda = [ + requirement + for requirement in requirements + if "nvidia" in requirement.lower() or "cuda" in requirement.lower() + ] + assert cuda, ( + "this is a CUDA row but the wheel declares no CUDA runtime dependency, so nothing " + "would install the libraries its delegate links" + ) + print(f"✓ the CUDA runtime is declared ({len(cuda)} requirements)") + + +def test_cuda_libraries_resolve_relatively() -> None: + """Each CUDA library must reach its runtime through a relative path. + + An absolute toolkit path names the machine that built the wheel. It resolves there and + nowhere else, so the wheel would work only on a builder. + + Every shipped library that links the CUDA runtime is inspected, wherever it lives. Naming + only the two in lib/ skipped libaoti_cuda_shims.so, which sits under backends/cuda/, links + cudart and curand, and carries the device code, so an absolute toolkit path on the library + that matters most shipped green. + """ + readelf = test_shared_libraries._tool("readelf") + assert readelf is not None, "readelf is required to inspect the wheel" + + package_dir = _package_dir() + libraries = sorted(test_shared_libraries._shipped_shared_objects(package_dir)) + # Without this the loop below finds nothing on a wheel that ships no CUDA library and + # reports a pass, which is the same as having no check at all. + assert libraries, f"no shared libraries found under {package_dir}" + + linked_to_cuda = [] + for library in libraries: + output = subprocess.run( + [readelf, "-d", str(library)], capture_output=True, text=True, check=True + ).stdout + if any("NEEDED" in line and "libcud" in line for line in output.splitlines()): + linked_to_cuda.append((library, output)) + + assert linked_to_cuda, ( + "no shipped library links the CUDA runtime, so this check inspected nothing. A CUDA " + "row must ship the libraries it is named for." + ) + for library, output in linked_to_cuda: + name = library.relative_to(package_dir) + entries: list[str] = [] + for line in output.splitlines(): + if "RPATH" in line or "RUNPATH" in line: + entries += line.split("[", 1)[1].rstrip("]").strip().split(":") + relative = [ + entry + for entry in entries + if entry.startswith("$ORIGIN") and "nvidia" in entry + ] + assert relative, ( + f"{name} links the CUDA runtime but has no relative path to the CUDA wheels " + f"installed beside it, so it can only resolve where the builder had a toolkit: " + f"{entries}" + ) + print(f"✓ {name} resolves the CUDA runtime relatively ({relative[0]})") + + +def _row_architectures() -> list[str]: + """The architectures this row claims, from the same script the build uses. + + A refusal from that script is a fault, not an absence. It returns non-zero when a CUDA row reaches it + with no version, which is precisely the case that would otherwise build device code for whatever GPU the + builder happens to have, so swallowing it here would hide the one failure this check exists to catch. + + EXECUTORCH_BUILD_CUDA is passed through because that is how the build invokes the script, and the + refusal is conditional on it. Without it the script returned an empty list on a CUDA row that had lost + its version, this check reported nothing to do, and the assertion below could never fire. + """ + script = pathlib.Path(__file__).parent / "cuda_arch_list.sh" + assert script.is_file(), f"the architecture script is missing at {script}" + result = subprocess.run( + ["bash", "-c", f"source {script}; executorch_cuda_arch_list"], + capture_output=True, + text=True, + check=False, + env={**os.environ, "EXECUTORCH_BUILD_CUDA": "1"}, + ) + assert result.returncode == 0, ( + f"the architecture script refused this row with exit {result.returncode}, so the build had no list " + f"to compile against: {result.stderr.strip()[:300]}" + ) + # "8.0 9.0" describes sm_80 and sm_90. + return ["sm_" + value.replace(".", "") for value in result.stdout.split()] + + +def test_device_code_covers_the_row() -> None: + """Every GPU the row claims must have device code in the shipped libraries. + + A row that promises a GPU it did not compile for produces a wheel that installs and then dies + at the first kernel launch, which is the worst failure to publish. + """ + expected = _row_architectures() + if not expected: + print("- this row claims no GPU architectures, nothing to check") + return + + cuobjdump = test_shared_libraries._tool("cuobjdump") + if cuobjdump is None: + raise AssertionError( + "cuobjdump is required to check device code, and this is a CUDA row. Without it a " + "wheel missing code for a claimed GPU would ship unnoticed." + ) + + # Searched across every shipped library rather than a named one. The kernels are compiled + # into their own library, not into the delegate, and which library holds them is an internal + # detail. What the row promises is that the wheel covers those GPUs. + present: set[str] = set() + inspected = [] + with_device_code: dict = {} + for library in sorted(_package_dir().rglob("*.so")): + listed = subprocess.run( + [cuobjdump, "--list-elf", str(library)], + capture_output=True, + text=True, + check=False, + ).stdout + found = { + token + for token in listed.replace(".", " ").split() + if token.startswith("sm_") + } + if found: + inspected.append(f"{library.name} ({', '.join(sorted(found))})") + present |= found + with_device_code[library.name] = found + + assert inspected, ( + "no shipped library contains any GPU device code, so this wheel cannot run a model on any " + f"GPU, while the row claims {expected}" + ) + missing = sorted(set(expected) - present) + assert not missing, ( + f"the row claims {expected} but the wheel carries no device code for {missing}. " + f"Found: {inspected}. A user with one of those GPUs would install this wheel and fail at " + "the first kernel launch." + ) + # The other direction matters just as much. Device code for an architecture the row does not + # claim means the build did not use the row's list, so whatever selected the architectures + # ignored it. That went unnoticed once already: a selection bug substituted a single default + # architecture and this check stayed green because it only looked for what was absent. + unexpected = sorted(present - set(expected)) + assert not unexpected, ( + f"the row claims {sorted(set(expected))} but the wheel also carries device code for " + f"{unexpected}. Found: {inspected}. The build did not use the row's list, so the artifact " + "does not match what the row published." + ) + # Every library that carries device code has to cover the row on its own. Unioning + # across libraries let a library with kernels cover only part of the row while an + # unrelated object supplied the rest, so on a GPU the first one did not compile for + # there was no executable kernel even though the union looked complete. + short = sorted(set(expected)) + for library in sorted(with_device_code): + library_missing = sorted(set(expected) - with_device_code[library]) + assert not library_missing, ( + f"{library} carries GPU device code but none for {library_missing}, while the row " + f"claims {short}. Checking the union across libraries hid this: another shipped " + "object supplied those architectures, and on such a GPU this library would have no " + "executable kernel." + ) + print(f"✓ device code covers the row in every library that has any: {inspected}") + + +def test_portable_device_code_is_present() -> None: + """The newest architecture must also ship in its portable form. + + The build appends "+PTX" for the top architecture so a GPU newer than any in the row can + still run, by having the driver compile that portable form at load time. Without it such a + GPU gets no usable code at all. + + Checked with --list-ptx rather than --list-elf. --list-elf prints byte-identical output for + a library built with or without the portable form, so it cannot see this. --list-ptx prints + an entry only for the library that has it. The entry is named for the target architecture, + "sm_90.ptx" rather than "compute_90.ptx", which is what the real tool prints. + """ + expected = _row_architectures() + if not expected: + print("- this row claims no GPU architectures, nothing to check") + return + + cuobjdump = test_shared_libraries._tool("cuobjdump") + assert ( + cuobjdump is not None + ), "cuobjdump is required to check the portable device code, and this is a CUDA row." + + # The newest architecture in the row, which is the one the build makes portable. + newest = max(expected, key=lambda name: int(name.removeprefix("sm_"))) + + found_in = [] + for library in sorted(_package_dir().rglob("*.so")): + listed = subprocess.run( + [cuobjdump, "--list-ptx", str(library)], + capture_output=True, + text=True, + check=False, + ).stdout + if newest in listed.replace(".", " ").split(): + found_in.append(library.name) + + assert found_in, ( + f"no shipped library carries portable device code for {newest}, the newest architecture " + f"in this row ({sorted(expected)}). A GPU newer than {newest} would install this wheel and " + "find no code it can run. The build appends the portable form for exactly this case, so " + "either it was dropped or the spelling in the architecture list is wrong." + ) + print(f"✓ portable device code for {newest} ships in {', '.join(found_in)}") + + +def test_the_delegate_registers() -> None: + """The delegate has to appear in the runtime's backend list, not merely be present as a file. + + Registration happens in a static initializer, which a normal link discards because nothing in the + program references it. Keeping it alive needs a linker option, and a wheel whose delegate ships but + does not register would load a delegated program and fail with an unregistered backend. That is the + failure this whole layout is most able to introduce, so it is worth asserting rather than assuming. + + Needs no GPU: registration is a link-time property, checked by importing. + """ + from executorch.extension.pybindings.portable_lib import ( + _get_registered_backend_names, + ) + + registered = _get_registered_backend_names() + assert "CudaBackend" in registered, ( + f"the wheel ships the CUDA delegate but CudaBackend is not registered: {registered}. " + "The library is present and its static initializer did not run, which means the option " + "that keeps it on the link line stopped working." + ) + print(f"✓ the delegate registers: CudaBackend among {len(registered)} backend(s)") + + +if __name__ == "__main__": + assert platform.system() == "Linux", "the CUDA rows are Linux only" + + test_cuda_libraries_are_shipped() + test_cuda_runtime_is_declared() + test_cuda_libraries_resolve_relatively() + test_device_code_covers_the_row() + test_portable_device_code_is_present() + test_the_delegate_registers() + + # The backend registrations a CPU Linux row asserts also apply here: the CUDA build enables + # OpenVINO on every Linux architecture and downloads the QNN SDK on x86_64, so a CUDA wheel + # carries both backends and needs both to register. + from executorch.extension.pybindings.portable_lib import ( + _get_registered_backend_names, + ) + + registered = _get_registered_backend_names() + if platform.machine() in ("x86_64", "amd64"): + assert ( + "QnnBackend" in registered + ), f"QnnBackend not found in registered backends: {registered}" + print("✓ QnnBackend is registered") + assert ( + "OpenvinoBackend" in registered + ), f"OpenvinoBackend not found in registered backends: {registered}" + print("✓ OpenvinoBackend is registered") + + test_base.test_cmsis_nn_install() + + # The packaging and linking checks a CPU wheel is held to still apply: one owner per + # component, no build-tree paths, and a C++ application able to link what the wheel + # ships. + with tempfile.TemporaryDirectory() as work_dir: + test_shared_libraries.run_tests(Path(work_dir)) + with tempfile.TemporaryDirectory() as work_dir: + test_cpp_sdk.run_tests(Path(work_dir)) + + test_base.run_tests( + model_tests=[ + test_base.ModelTest( + model=Model.Mv3, + backend=Backend.XnnpackQuantizationDelegation, + ), + ] + ) diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py index 0484736f9f4..4fe89b47f28 100644 --- a/.ci/scripts/wheel/test_shared_libraries.py +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -440,6 +440,22 @@ def _wheel_cuda_train() -> str: _REQUIRED_ON_A_CUDA_WHEEL = "cuda-wheel-only" +# The exact dependency names packaging declares per CUDA train, mirroring +# _CUDA_RUNTIME_PACKAGES in setup.py. Listed here rather than imported because setup.py +# runs a build when imported, and duplicated deliberately so a rename on the packaging +# side has to be made here too rather than silently agreeing with itself. +_EXPECTED_CUDA_PACKAGES = { + "12": ( + "nvidia-cuda-runtime-cu12", + "nvidia-curand-cu12", + ), + "13": ( + "nvidia-cuda-runtime", + "nvidia-curand", + ), +} + + # Each component the wheel ships as its own library, the symbols that identify it, # and the library that must own them. `required` says whether the owner has to be # present: the optimized kernels are optional, because a wheel built without them @@ -1635,17 +1651,34 @@ def test_model_matches_eager_pytorch(work_dir: Path) -> None: def test_declared_dependencies_match_the_wheel_tag() -> None: - """A CPU wheel must not declare the CUDA runtime, and a CUDA wheel must declare it. + """A CPU wheel must not declare the CUDA runtime, and a CUDA wheel must declare its own train. The tag is what a user resolves against, so a mismatch is a promise the wheel cannot keep in either direction: a CPU wheel that pulls the CUDA packages costs a user hundreds of megabytes it never loads, and a CUDA wheel that declares nothing leaves the runtime unresolvable. + Declaring the wrong train is the quiet case, and the reason this checks the names rather than + only their presence. The CUDA 12 packages are published with a "-cu12" suffix and the CUDA 13 + ones without, so a cu130 wheel that asked for the cu12 packages would install a runtime its + libraries cannot load, while looking correctly specified. + This is metadata only, so no library check can see it. A CPU wheel that wrongly declared the CUDA runtime passed every other check in this file. """ requirements = importlib.metadata.requires("executorch") or [] - cuda = sorted(r.split()[0] for r in requirements if r.lower().startswith("nvidia")) + + # Split off any environment marker AND any version specifier. The name, the specifier + # and the marker can arrive as one token, so taking the first whitespace-separated + # word left "nvidia-cuda-runtime-cu12==12.6.77" as the name and made a correctly + # specified wheel fail the moment any CUDA dependency gained a pin. + def distribution_name(requirement: str) -> str: + return re.split(r"[\s;\[<>=!~(]", requirement.strip(), maxsplit=1)[0] + + cuda = sorted( + name + for name in (distribution_name(r) for r in requirements) + if name.lower().startswith("nvidia") + ) # The local version segment of the installed version states what the wheel was built for. version = importlib.metadata.version("executorch") @@ -1657,7 +1690,35 @@ def test_declared_dependencies_match_the_wheel_tag() -> None: f"version {version} says this is a CUDA wheel, but it declares no CUDA runtime " "packages, so nothing resolves the runtime it links" ) - print(f"✓ this CUDA wheel declares the runtime ({len(cuda)} packages)") + # Compared as sets in both directions rather than as a name suffix: for CUDA 13 the + # expected suffix is the empty string and every name ends with that, so a suffix test + # accepted a name from any train whose spelling happened not to be one of the two + # literals it also excluded. Measured: a cu130 wheel declaring nvidia-cuda-runtime-cu11 + # passed. The reverse check catches the other side of the same defect: a wheel that + # declares one package and omits the others still cannot load, and one-direction only + # would accept it. + train = local[len("cu") : len("cu") + 2] + expected = set(_EXPECTED_CUDA_PACKAGES.get(train, ())) + assert expected, ( + f"version {version} names CUDA train {train}, which this check has no expected " + f"package list for. Add it beside the packaging list it mirrors." + ) + actual = set(cuda) + wrong = sorted(actual - expected) + missing = sorted(expected - actual) + assert not wrong, ( + f"version {version} is a CUDA {train} wheel, but it declares {wrong}, which belong to " + f"another CUDA train. Expected only {sorted(expected)}. A user would install a runtime " + "this wheel's libraries cannot load." + ) + assert not missing, ( + f"version {version} is a CUDA {train} wheel, but it does not declare {missing} " + f"(expected {sorted(expected)}). A user installing this wheel would end up without part " + "of the CUDA runtime the wheel's libraries need." + ) + print( + f"✓ this CUDA {train} wheel declares its own runtime ({len(cuda)} packages)" + ) else: assert not cuda, ( f"version {version} is not a CUDA wheel, yet it declares {cuda}. A user installing it " diff --git a/.github/scripts/filter_cuda_matrix.py b/.github/scripts/filter_cuda_matrix.py new file mode 100644 index 00000000000..385ea9385cb --- /dev/null +++ b/.github/scripts/filter_cuda_matrix.py @@ -0,0 +1,238 @@ +#!/usr/bin/env python3 +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +"""Narrow the generated build matrix to the rows a GPU wheel can honestly support. + +The shared matrix generator emits every CUDA version and Python version it knows about. +Building all of them would publish wheels for combinations nothing can verify, and a GPU +wheel that installs and then cannot run is worse than one that does not exist: the failure +appears when a model runs, and it looks like a model problem rather than a packaging one. + +A row is kept only when both of these hold: + + a GPU exists that the row's device code covers + a PyTorch build is published for that CUDA version and architecture + +Running a real model before release is a separate gate, on hardware that has the matching +GPU, so a row can be published for a CUDA version no machine here can execute. + +The values below are the current answers to those questions. They are written out rather +than derived because each one is an external fact that can change independently. +""" + +import argparse +import json +import sys +from typing import Any, Dict, List + +# Python versions that are deliberately NOT published, with the reason, so a row naming one +# is rejected for a stated cause rather than for merely being absent from the supported list. +# 3.14 is excluded because the current CPU wheel rows already fail on it for an unrelated +# reason in the example requirements, so a GPU row would inherit a known-broken build. The +# free-threaded builds are excluded because the CUDA dependencies are not published for them. +# +# This is documentation, not the gate. The gate is SUPPORTED_PYTHON_VERSIONS below: anything +# not on that list is rejected whether or not it appears here. +DISABLED_PYTHON_VERSIONS: List[str] = ["3.13t", "3.14", "3.14t", "3.15", "3.15t"] + +# CUDA versions to publish. +# +# Chosen so that every consumer row can find a matching wheel rather than by what is +# convenient to verify. A delegate built against one of these has to be able to depend on an +# ExecuTorch wheel for the same CUDA version, and a missing version means that consumer has +# nothing to depend on: +# +# cu126 the floor, and what Jetson devices are limited to +# cu130 the generator's stable choice, and the default for accelerator consumers +# cu132 the newest, which consumers building against a current TensorRT need +# +# cu132 is included even though no machine here can execute it, because omitting it would +# leave a published consumer row with no ExecuTorch wheel to pair with. The packaging +# properties are checked on every row; executing a model is a release-gate step on hardware +# that has the matching GPU. +SUPPORTED_CUDA_VERSIONS: List[str] = ["cu126", "cu130", "cu132"] + +# Python versions to publish, stated rather than derived for the same reason the CUDA +# versions are. Deriving them from the rows that survived the filter made the release +# guard below unable to notice a python that disappeared from every supported train: with +# nothing left to compare, a release quietly published nine wheels instead of twelve. +# Keep in step with the python-versions list in the CUDA wheel workflows. +SUPPORTED_PYTHON_VERSIONS: List[str] = ["3.10", "3.11", "3.12", "3.13"] + +# The single row built for a pull request. A full matrix on every push would cost hours for +# little signal, and this pair is the one with a machine that can run a model on it. +PR_PYTHON_VERSION: str = "3.12" +PR_CUDA_VERSION: str = "cu130" + +# Jetson devices are their own row: a JetPack image, one Python version, and one CUDA +# version. Kept empty on purpose today, so no Jetson row is emitted. +# +# The generic aarch64 CUDA 12.6 wheel does compile sm_87 device code for one embedded +# module, so the wheel itself is not the blocker. What is: published PyTorch stopped +# shipping sm_87 device code after 2.8.0, so a Jetson row today would produce a wheel +# whose PyTorch dependency cannot execute on the device. Populate this when that +# changes. +# +# Because both lists are empty, asking for the JetPack rows can only produce an empty result. +# No workflow asks, and the request is rejected up front with that reason rather than left to +# surface as the generic "the filter produced no rows" message, which reads as a broken +# matrix rather than as a row that is deliberately not built yet. +JETPACK_PYTHON_VERSIONS: List[str] = [] +JETPACK_CUDA_VERSIONS: List[str] = [] +JETPACK_CONTAINER_IMAGE: str = "nvcr.io/nvidia/l4t-jetpack:r36.4.0" + + +def keep(item: Dict[str, Any], is_jetpack: bool) -> bool: + """Whether this row should be built, adjusting its container image where needed.""" + # An allowlist, the same shape as the CUDA test below. Testing only the disabled list + # let any python not on it through: passing a 3.9 row returned success and emitted it, + # and the only thing preventing that today is both workflows happening to pin the list + # they pass in. + if item["python_version"] not in SUPPORTED_PYTHON_VERSIONS: + return False + + if is_jetpack: + if ( + item["python_version"] in JETPACK_PYTHON_VERSIONS + and item["desired_cuda"] in JETPACK_CUDA_VERSIONS + ): + item["container_image"] = JETPACK_CONTAINER_IMAGE + return True + return False + + if item["desired_cuda"] not in SUPPORTED_CUDA_VERSIONS: + return False + + return True + + +def _version_rank(cuda: str) -> int: + """Where a CUDA version sits in the supported list, or -1 when it is not supported at all.""" + try: + return SUPPORTED_CUDA_VERSIONS.index(cuda) + except ValueError: + return -1 + + +def only_pull_request_row(items: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """One representative row, so a pull request does not build the whole matrix. + + Chosen by preference rather than exact match, so a request that does not appear in the + generated matrix degrades to the closest supported combination instead of falling off the + end. + """ + if not items: + return [] + + # Looked up once, and tolerantly: a PR_CUDA_VERSION that falls off SUPPORTED_CUDA_VERSIONS used to + # raise here and break every pull request while releases kept working, which is the wrong way round + # for a constant that only chooses which single row to build. + wanted = _version_rank(PR_CUDA_VERSION) + + def rank(item: Dict[str, Any]) -> tuple: + # Closeness peaks at the requested version, then falls off, and it outranks the python match. + # Ranking python first picked a wheel for a CUDA version nothing on hand can execute whenever the + # generator skewed the two axes, and the point of building one row is to get signal from it. + offered = _version_rank(item["desired_cuda"]) + # Negative above the requested version, so a newer one never outranks an older one a machine here + # can actually run. + closeness = offered if offered <= wanted else wanted - offered + return (closeness, item["python_version"] == PR_PYTHON_VERSION) + + return [max(items, key=rank)] + + +def main(argv: List[str]) -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--matrix", required=True, help="the generated matrix, as JSON") + parser.add_argument( + "--jetpack", default="false", help="build the Jetson row instead" + ) + parser.add_argument("--limit-pr-builds", default="false", help="build one row only") + args = parser.parse_args(argv) + + try: + matrix = json.loads(args.matrix) + except json.JSONDecodeError as error: + print(f"could not parse the matrix: {error}", file=sys.stderr) + sys.exit(1) + + is_jetpack = args.jetpack.lower() == "true" + if is_jetpack and not (JETPACK_PYTHON_VERSIONS and JETPACK_CUDA_VERSIONS): + # Rejected here rather than allowed to fall through to an empty result, so the reason + # is the actual one. Nothing passes this flag today. + print( + "the JetPack rows are not published yet: JETPACK_PYTHON_VERSIONS and " + "JETPACK_CUDA_VERSIONS are empty because published PyTorch carries no device code " + "for that GPU architecture, so any wheel built here could not run on the device. " + "Populate both lists to enable this row.", + file=sys.stderr, + ) + sys.exit(1) + items = [item for item in matrix.get("include", []) if keep(item, is_jetpack)] + + if args.limit_pr_builds.lower() == "true" and items: + items = only_pull_request_row(items) + elif items and not is_jetpack: + # A release has to publish every combination this policy advertises. Comparing the result against + # what the generator offered cannot catch anything, because both sides apply the same conditions, so + # the difference is empty by construction and the check never fires. The policy's own list is the + # thing to compare against: a CUDA version the generator stopped offering otherwise disappears from + # the release silently, and a missing job is a green check for a wheel that was never built. + # + # The generic rows only. A JetPack release advertises the single pair its own lists name rather than + # every supported CUDA version, so checking it against this list would fail a correct release. + # + # Both axes come from this policy's own lists, not from the matrix. Reading the generator's python + # axis pulled in rows this policy never builds, and deriving it from the rows that survived went + # blind to a python that disappeared from every supported train. The generator lives in another + # repository and its axes move independently of what this policy promises to publish. + built = {(item["python_version"], item["desired_cuda"]) for item in items} + # A train that produced no row at all is missing for every python, so reporting it per python + # would read as a python problem. Named on its own instead, and first, because the per-pair + # report below would otherwise bury it. + absent_trains = sorted( + set(SUPPORTED_CUDA_VERSIONS) - {cuda for _, cuda in built} + ) + if absent_trains: + print( + f"this policy publishes {SUPPORTED_CUDA_VERSIONS}, but the generator offered no row " + f"this filter could keep for {absent_trains}, so a release would publish no wheel for " + "that CUDA version at all", + file=sys.stderr, + ) + sys.exit(1) + missing = sorted( + f"{python}/{cuda}" + for python in SUPPORTED_PYTHON_VERSIONS + for cuda in SUPPORTED_CUDA_VERSIONS + if (python, cuda) not in built + ) + if missing: + print( + f"this policy publishes {SUPPORTED_CUDA_VERSIONS} for each of " + f"{SUPPORTED_PYTHON_VERSIONS}, but {len(missing)} combination(s) produced no row, so a " + f"release would publish no wheel for them: {missing}", + file=sys.stderr, + ) + sys.exit(1) + + # Fail loudly on an empty result. A silently empty matrix produces a workflow with no + # build job, which shows up as a green check for a build that never happened. + if not items: + print( + "the filter produced no rows to build, so nothing would be verified. " + f"jetpack={is_jetpack}, supported CUDA={SUPPORTED_CUDA_VERSIONS}", + file=sys.stderr, + ) + sys.exit(1) + + print(json.dumps({"include": items})) + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/.github/workflows/build-wheels-cuda-aarch64-linux.yml b/.github/workflows/build-wheels-cuda-aarch64-linux.yml new file mode 100644 index 00000000000..bb20d815b62 --- /dev/null +++ b/.github/workflows/build-wheels-cuda-aarch64-linux.yml @@ -0,0 +1,103 @@ +# From https://github.com/pytorch/test-infra/wiki/Using-Nova-Reusable-Build-Workflows +name: Build Aarch64 Linux CUDA Wheels + +on: + pull_request: + paths: + - .ci/**/* + - .github/scripts/filter_cuda_matrix.py + - .github/workflows/build-wheels-cuda-aarch64-linux.yml + - '**/CMakeLists.txt' + - backends/cuda/**/* + - examples/**/* + - extension/cuda/**/* + - install_requirements.py + - install_utils.py + - pyproject.toml + - setup.py + - tools/cmake/**/* + push: + branches: + - nightly + - release/* + tags: + # NOTE: Binary build pipelines should only get triggered on release candidate builds + # Release candidate tags look like: v1.11.0-rc1 + - v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+ + - ciflow/binaries/* + workflow_dispatch: + +jobs: + generate-matrix: + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: wheel + os: linux-aarch64 + test-infra-repository: pytorch/test-infra + test-infra-ref: main + with-cuda: enable + with-cpu: disable + with-rocm: disable + python-versions: '["3.10", "3.11", "3.12", "3.13"]' + + # The generator emits every CUDA version it knows about. Publishing all of them would ship + # wheels for combinations nothing can verify, so this keeps only the rows with a GPU to run + # them on. The script fails rather than emitting an empty matrix, because a workflow with no + # build job reads as a pass. + filter-matrix: + needs: generate-matrix + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.filter.outputs.matrix }} + steps: + - uses: actions/setup-python@v6 + with: + python-version: '3.12' + - uses: actions/checkout@v4 + - name: Filter the matrix + id: filter + run: | + set -eou pipefail + MATRIX_BLOB=${{ toJSON(needs.generate-matrix.outputs.matrix) }} + # Mirrors the shared generator, which drops its own row limit when this label is + # present. Clamping here regardless meant the label was accepted as a trigger and + # then ignored, so the full matrix could never be exercised before a release. + LIMIT_PR=${{ (github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'ciflow/binaries/all')) && 'true' || 'false' }} + MATRIX_BLOB="$(python3 .github/scripts/filter_cuda_matrix.py \ + --matrix "${MATRIX_BLOB}" --limit-pr-builds "${LIMIT_PR}")" + echo "${MATRIX_BLOB}" + echo "matrix=${MATRIX_BLOB}" >> "${GITHUB_OUTPUT}" + + build: + needs: filter-matrix + permissions: + id-token: write + contents: read + strategy: + fail-fast: false + matrix: + include: + - repository: pytorch/executorch + pre-script: .ci/scripts/wheel/pre_build_script.sh + post-script: .ci/scripts/wheel/post_build_script.sh + smoke-test-script: .ci/scripts/wheel/test_cuda_linux.py + package-name: executorch + name: ${{ matrix.repository }} + uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main + with: + repository: ${{ matrix.repository }} + ref: "" + test-infra-repository: pytorch/test-infra + test-infra-ref: main + build-matrix: ${{ needs.filter-matrix.outputs.matrix }} + submodules: recursive + env-var-script: .ci/scripts/wheel/envvar_cuda_linux.sh + pre-script: ${{ matrix.pre-script }} + post-script: ${{ matrix.post-script }} + package-name: ${{ matrix.package-name }} + smoke-test-script: ${{ matrix.smoke-test-script }} + trigger-event: ${{ github.event_name }} + # Required for aarch64. Without it the shared build workflow prepares an x86_64 job + # and skips the aarch64 conda install, so the first build step fails on a missing + # conda. + architecture: aarch64 diff --git a/.github/workflows/build-wheels-cuda-linux.yml b/.github/workflows/build-wheels-cuda-linux.yml new file mode 100644 index 00000000000..5d948370583 --- /dev/null +++ b/.github/workflows/build-wheels-cuda-linux.yml @@ -0,0 +1,99 @@ +# From https://github.com/pytorch/test-infra/wiki/Using-Nova-Reusable-Build-Workflows +name: Build Linux CUDA Wheels + +on: + pull_request: + paths: + - .ci/**/* + - .github/scripts/filter_cuda_matrix.py + - .github/workflows/build-wheels-cuda-linux.yml + - '**/CMakeLists.txt' + - backends/cuda/**/* + - examples/**/* + - extension/cuda/**/* + - install_requirements.py + - install_utils.py + - pyproject.toml + - setup.py + - tools/cmake/**/* + push: + branches: + - nightly + - release/* + tags: + # NOTE: Binary build pipelines should only get triggered on release candidate builds + # Release candidate tags look like: v1.11.0-rc1 + - v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+ + - ciflow/binaries/* + workflow_dispatch: + +jobs: + generate-matrix: + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: wheel + os: linux + test-infra-repository: pytorch/test-infra + test-infra-ref: main + with-cuda: enable + with-cpu: disable + with-rocm: disable + python-versions: '["3.10", "3.11", "3.12", "3.13"]' + + # The generator emits every CUDA version it knows about. Publishing all of them would ship + # wheels for combinations nothing can verify, so this keeps only the rows with a GPU to run + # them on. The script fails rather than emitting an empty matrix, because a workflow with no + # build job reads as a pass. + filter-matrix: + needs: generate-matrix + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.filter.outputs.matrix }} + steps: + - uses: actions/setup-python@v6 + with: + python-version: '3.12' + - uses: actions/checkout@v4 + - name: Filter the matrix + id: filter + run: | + set -eou pipefail + MATRIX_BLOB=${{ toJSON(needs.generate-matrix.outputs.matrix) }} + # Mirrors the shared generator, which drops its own row limit when this label is + # present. Clamping here regardless meant the label was accepted as a trigger and + # then ignored, so the full matrix could never be exercised before a release. + LIMIT_PR=${{ (github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'ciflow/binaries/all')) && 'true' || 'false' }} + MATRIX_BLOB="$(python3 .github/scripts/filter_cuda_matrix.py \ + --matrix "${MATRIX_BLOB}" --limit-pr-builds "${LIMIT_PR}")" + echo "${MATRIX_BLOB}" + echo "matrix=${MATRIX_BLOB}" >> "${GITHUB_OUTPUT}" + + build: + needs: filter-matrix + permissions: + id-token: write + contents: read + strategy: + fail-fast: false + matrix: + include: + - repository: pytorch/executorch + pre-script: .ci/scripts/wheel/pre_build_script.sh + post-script: .ci/scripts/wheel/post_build_script.sh + smoke-test-script: .ci/scripts/wheel/test_cuda_linux.py + package-name: executorch + name: ${{ matrix.repository }} + uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main + with: + repository: ${{ matrix.repository }} + ref: "" + test-infra-repository: pytorch/test-infra + test-infra-ref: main + build-matrix: ${{ needs.filter-matrix.outputs.matrix }} + submodules: recursive + env-var-script: .ci/scripts/wheel/envvar_cuda_linux.sh + pre-script: ${{ matrix.pre-script }} + post-script: ${{ matrix.post-script }} + package-name: ${{ matrix.package-name }} + smoke-test-script: ${{ matrix.smoke-test-script }} + trigger-event: ${{ github.event_name }} diff --git a/backends/cuda/CMakeLists.txt b/backends/cuda/CMakeLists.txt index 05f238401a4..c56e955e8a1 100644 --- a/backends/cuda/CMakeLists.txt +++ b/backends/cuda/CMakeLists.txt @@ -42,6 +42,74 @@ if(NOT CMAKE_CUDA_COMPILER) check_language(CUDA) endif() +# Take the architectures from the release row when it names them, before the +# language is enabled, since CMake fixes them at that point. Without this the +# build uses CMake's default, which on some devices is older than the intrinsics +# these sources use, and the compile fails with an undefined identifier that +# looks like a source problem. +# +# TORCH_CUDA_ARCH_LIST is the variable the surrounding build environment already +# sets, in PyTorch's dotted form. CMake wants bare integers, so "9.0" becomes +# 90. A "+PTX" suffix asks for the portable form in addition to the compiled +# one, which is what PyTorch means by it, so it adds the -virtual kind rather +# than replacing the -real one. Torch sets this to OFF at root scope when it is +# defined, warning that it ignores the value, so a defined value does not mean a +# caller chose it. OFF is treated as absent here, otherwise asking for an +# architecture through the preset silently compiles for whatever torch's own +# gencode flags select instead. CMAKE_CUDA_ARCHITECTURES is deliberately not +# read here. torch's CMake rejects and overrides it, which is why the release +# rows publish TORCH_CUDA_ARCH_LIST instead, and CMake fills the cache entry +# with a default of its own once the CUDA language is enabled. torch enables +# that language before this directory is added, so the cache always holds a +# value and cannot be read as a caller's intent: doing so replaced torch's +# autodetected architecture with CMake's default and broke the build on a device +# the default does not cover. +if(DEFINED ENV{TORCH_CUDA_ARCH_LIST}) + string(REPLACE "." "" _executorch_cuda_arch_list "$ENV{TORCH_CUDA_ARCH_LIST}") + string(REPLACE " " ";" _executorch_cuda_arch_list + "${_executorch_cuda_arch_list}" + ) + # torch spells a request for the portable form as a "+PTX" suffix, which nvcc + # rejects literally. Drop the suffix rather than expanding it into a separate + # -virtual entry: an unsuffixed CMake architecture already asks for both the + # compiled and the portable form, measured as code=[compute_120,sm_120] for a + # bare "120", so the extra entry only duplicates a gencode that is already + # there. + set(_executorch_cuda_arch_resolved "") + foreach(_arch IN LISTS _executorch_cuda_arch_list) + string(REGEX REPLACE "\\+PTX$" "" _arch "${_arch}") + list(APPEND _executorch_cuda_arch_resolved "${_arch}") + endforeach() + # A row that names both "12.0" and "12.0+PTX" collapses to the same entry + # twice, which would duplicate a gencode on the compile line. + list(REMOVE_DUPLICATES _executorch_cuda_arch_resolved) + # torch also accepts family names such as Ampere or All. They survive the + # translation above and then fail deep in a CUDA compile as an unsupported + # architecture, so reject them here where the message can name the cause. + foreach(_arch IN LISTS _executorch_cuda_arch_resolved) + if(NOT _arch MATCHES "^[0-9]+[a-z]*(-real|-virtual)?$") + message( + FATAL_ERROR + "TORCH_CUDA_ARCH_LIST entry \"${_arch}\" is not a compute capability. " + "Name capabilities numerically, for example \"8.0 9.0\" or \"12.0+PTX\". " + "Family names such as Ampere or All are accepted by torch but not by nvcc." + ) + endif() + endforeach() + set(CMAKE_CUDA_ARCHITECTURES "${_executorch_cuda_arch_resolved}") + message( + STATUS + "CUDA architectures from TORCH_CUDA_ARCH_LIST: ${CMAKE_CUDA_ARCHITECTURES}" + ) +else() + # torch sets this to OFF on purpose and drives nvcc with its own gencode + # flags, so there is nothing to choose here and overriding it would replace a + # working set of architectures with one value. + message( + STATUS "CUDA architectures left to torch: ${CMAKE_CUDA_ARCHITECTURES}" + ) +endif() + if(CMAKE_CUDA_COMPILER) enable_language(CUDA) endif() diff --git a/install_requirements.py b/install_requirements.py index 1aedcf6f0f8..648da1df243 100644 --- a/install_requirements.py +++ b/install_requirements.py @@ -7,6 +7,7 @@ import argparse import os +import platform import subprocess import sys @@ -45,7 +46,19 @@ def install_requirements(use_pytorch_nightly): # Determine the appropriate PyTorch URL based on CUDA delegate status torch_url = determine_torch_url(TORCH_URL_BASE) - torchao_url = determine_torch_url(TORCHAO_URL_BASE) + # torchao's CUDA channel publishes x86_64 only, so asking for a CUDA build makes the pin + # unsatisfiable on aarch64. Only that case is special-cased: falling back everywhere would + # change which torchao a CPU x86_64 install resolves, and the CUDA build is genuinely wanted + # where it exists. Nothing in the wheel links or bundles torchao; it is a quantization + # workflow dependency of the examples and tests. + if platform.machine().lower() in ("aarch64", "arm64"): + # The cpu channel specifically, not the index root. The root carries every variant, and a + # pin without a local segment admits all of them while ordering a local segment highest, + # so the xpu channel's pure python wheel would win on version before pip compares wheel + # tags, silently replacing the compiled aarch64 build. + torchao_url = f"{TORCHAO_URL_BASE}/cpu" + else: + torchao_url = determine_torch_url(TORCHAO_URL_BASE) # pip packages needed by exir. TORCH_PACKAGE = [ From e40f41e9ee8abff95294daa722100ea71ceb694e Mon Sep 17 00:00:00 2001 From: shoumikhin Date: Thu, 13 Aug 2026 11:48:43 -0700 Subject: [PATCH 6/6] Update [ghstack-poisoned] --- .ci/scripts/wheel/test_shared_libraries.py | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py index ae48d5ed800..5f14d8874fc 100644 --- a/.ci/scripts/wheel/test_shared_libraries.py +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -452,14 +452,8 @@ def _wheel_cuda_train() -> str: # runs a build when imported, and duplicated deliberately so a rename on the packaging # side has to be made here too rather than silently agreeing with itself. _EXPECTED_CUDA_PACKAGES = { - "12": ( - "nvidia-cuda-runtime-cu12", - "nvidia-curand-cu12", - ), - "13": ( - "nvidia-cuda-runtime", - "nvidia-curand", - ), + "12": ("nvidia-cuda-runtime-cu12",), + "13": ("nvidia-cuda-runtime",), }