diff --git a/peps/pep-0843.rst b/peps/pep-0843.rst new file mode 100644 index 00000000000..1f852212ff2 --- /dev/null +++ b/peps/pep-0843.rst @@ -0,0 +1,635 @@ +PEP: 843 +Title: Export Statement for DRY Re-exports +Author: Neil Girdhar +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 05-Aug-2026 +Python-Version: 3.16 +Post-History: `05-Aug-2026 `__ + + +Abstract +======== + +Large libraries separate their **implementation layout** (the tree of +modules convenient for maintainers) from their **public layout** (the +flat, curated tree they present to users). Building that public layout +today means choosing between two imperfect options. + +The first is writing every exported name twice: once in an import +statement, again as a string in ``__all__``. The two lists must be kept +in sync by hand every time the public layout changes. + +The second is the reflexive-alias idiom, ``from x import y as y``. This +is part of the type system: type checkers treat it as a signal that the +import is an intentional re-export. It still reads like a typo to anyone +who doesn't know the convention. Because the module has no curated +``__all__``, it loses the ``dir()`` cleanliness and +wildcard-import control a real ``__all__`` gives. + +This PEP adds a statement form that avoids both problems: + +.. code-block:: python + + # spam/__init__.py + from ._internal.core export PublicAPI + from ._internal.widgets export Widget as PublicWidget + +It imports the name, optionally under an alias exactly as ``from ... +import ... as ...`` does, and appends it to ``__all__`` in the same +statement. Nothing is left to sync by hand, and no alias needs decoding. + + +Relationship to PEP 842 +======================== + +Both PEPs start from the same discomfort with ``__all__``, and agree on +the same core mechanism for re-exports: a statement of the shape ``from + export ``, with a lazy variant (see `Lazy exports`_). +That agreement, reached independently, is confirmation that this is the +right shape for re-exports. + +PEP 842 broadens the mechanism into a keyword usable in five forms: + +* Standalone ``export NAME`` +* ``export NAME = VALUE`` assignments +* ``export def`` +* ``export class`` +* A module re-export statement, ``from MODULE export NAME`` + +All five populate ``__export__``, which also becomes ``__all__`` and +triggers an ``ExportWarning`` on access to anything left out. + +PEP 842's version has no wildcard equivalent to `Wildcard form`_. This +PEP takes only the module re-export statement, deliberately leaving out +the rest; see `Non-goals`_ for what and why. + + +Motivation +========== + +Widely-used libraries like NumPy, pandas, polars, Typer, FastAPI, and +Plotly almost universally export their public API using a common +pattern. A top-level ``__init__.py`` carries a wall of imports from +private submodules, followed by (or interleaved with) an ``__all__`` +list that repeats the same names as strings, or the reflexive-alias +idiom used throughout instead. The first looks like this: + +.. code-block:: python + + from ._internal.core import PublicAPI as PublicAPI + from ._internal.widgets import Widget as Widget + from ._internal.errors import SpamError as SpamError + # ... often hundreds of lines like this ... + + __all__ = [ + "PublicAPI", + "Widget", + "SpamError", + # ... the same names again ... + ] + +This is the file where a library flattens its implementation layout into +its public layout. As a library grows, the two diverge: code gets +reorganized into submodules for the maintainer's convenience, while the +public layout stays stable for users. + +Something has to do the flattening. Today that something is a +hand-maintained, doubly-written list, and such lists routinely drift out +of sync as the code changes. + +The alternative, ``import x as x``, is a workaround for the language's +missing export concept, and it still trips up some auto-formatters, which +see a bare ``import x`` as unused and remove it. + + +``__all__`` conflates two concerns +------------------------------------- + +Hand-maintained ``__all__`` also mixes two distinct concerns in one +file: the list of imports (an implementation detail of how the +flattening is wired up) and the declaration of the public API (a promise +to users). The two live in separate statements at different places in +the file, and nothing keeps them in sync except a reviewer checking them +against each other by eye, or a linter rule built for exactly this case. +The more common "unused import" checks don't help: if a name is +imported but never added to ``__all__``, that name reads as unused, and +autofixers routinely delete it rather than surface the omission. + + +Underscores solve a different problem +-------------------------------------- + +A natural response is: "just prefix internal names with an underscore." +But the privacy this PEP cares about lives at the package level, not the +name level: which parts of a large, multi-module package belong in the +**public layout**. Underscore-prefixing already marks a name private +within a module. + +The problem shows up in "hub" modules (usually ``__init__.py`` files) +whose only job is gathering names from internal modules and presenting +them under a stable public name. Every name that reaches a hub module is +already meant to be public: the underscore convention has done its job +by the time the name enters the hub. Hub modules need a non-repetitive +way to say "this is also part of the package's public layout." + + +Non-goals +========= + +This PEP does not aim to: + +* Restrict runtime attribute access to non-exported names, or change + ``__getattr__`` semantics. See `Why no runtime enforcement`_. +* Mark a freshly written ``def``, ``class``, or assignment as exported at + its definition site, the way the third-party ``atpublic`` package does + with ``@public``/``@private`` decorators. See `Why only re-exports`_. + + +Specification +============= + +``export`` is a soft keyword that replaces ``import`` in a ``from`` import +statement: + +.. code-block:: python + + from ._internal.core export PublicAPI + from ._internal.widgets export Widget as PublicWidget + from numpy.typing export NDArray + +A ``from export [as ]`` statement does what ``from + import [as ]`` does: it binds ````, or +```` if given, in the current namespace, and also appends that name +to the module's ``__all__``, creating ``__all__`` if it doesn't already +exist. + +Because it desugars to an ordinary import plus an append to ``__all__``, +``export`` composes with control flow exactly as ``import`` does: + +.. code-block:: python + + if sys.platform == "win32": + from ._internal.windows export WindowsThing + else: + from ._internal.posix export PosixThing + +Each branch runs its own import and its own ``__all__`` append, so the +name that ends up exported depends on which branch ran, with no separate +``__all__`` bookkeeping required. + +```` may be relative (``from .core export Thing``, ``from +..sub.core export Thing``) or absolute (``from numpy.typing export +NDArray``), exactly as in an ordinary ``from ... import ...`` statement. +A single statement may export multiple names, using the same syntax as a +regular multi-name ``from`` import, including a parenthesized, multi-line +list for long ones: + +.. code-block:: python + + from ._internal.widgets export Widget, Gadget as PublicGadget + + from ._internal.widgets export ( + Widget, + Gadget, + Doohickey, + ) + +Exporting a name is itself a use of it, so tools that flag "imported but +unused" names (linters, formatters) should treat every name bound by a +``from export ...`` statement as used, the same way they already +special-case ``from module import Thing as Thing``. This PEP doesn't +change what those tools decide. It follows from what ``export`` means: the +export is the use. + + +Wildcard form +----------------- + +This proposal also includes a wildcard form, ``from export *``. +It binds every name that ``from import *`` would bind, using the +same rule (````'s own ``__all__`` if it defines one, otherwise +every top-level name that doesn't start with an underscore), and appends +all of those names to the current module's ``__all__``: + +.. code-block:: python + + # spam/_internal/core.py + __all__ = ["PublicAPI", "Helper"] # curated by the internal module itself + ... + + # spam/__init__.py + from ._internal.core export * + # binds PublicAPI and Helper, and adds both to spam.__all__ + +This supports a common two-tier layout: an internal module curates its own +``__all__`` as it's written, and the hub re-exports that whole list in one +statement, instead of naming each item again. + +The wildcard form is equivalent to: + +.. code-block:: python + + # from ._internal.core export * + from ._internal.core import * + __all__ = list(globals().get("__all__", [])) + _names_bound_by_star_import + +where ``_names_bound_by_star_import`` is the list of names ``from +._internal.core import *`` just bound, the same list Python's import +machinery already computes to execute a wildcard import. + + +Lazy exports +----------------- + +:pep:`810` adds a ``lazy`` soft keyword that defers a ``from ... import`` +statement until the imported name is first used: ``lazy from +import `` binds a lazy proxy immediately but doesn't load +```` until that proxy is touched. + +``export`` composes with it the same way it composes with ``import``: + +.. code-block:: python + + lazy from ._internal.core export PublicAPI + +This binds ``PublicAPI`` to a lazy proxy, exactly as :pep:`810` specifies, +and appends ``"PublicAPI"`` to ``__all__`` immediately, without waiting +for the proxy to be touched. Populating ``__all__`` only needs the name as +a string, not the loaded value, so the export half of the statement stays +eager even when the import half is lazy. For a hub module with hundreds of +re-exports, this gives users a complete, accurate ``__all__`` and +``dir()`` at import time, without paying the cost of loading every +internal module up front. + +The statement is equivalent to: + +.. code-block:: python + + # lazy from ._internal.core export PublicAPI + lazy from ._internal.core import PublicAPI + __all__ = list(globals().get("__all__", [])) + ["PublicAPI"] + +``lazy from export *`` is not allowed, for two independent +reasons: :pep:`810` already disallows ``lazy from import *``, and +the wildcard export form needs ```` loaded to know what names +``__all__`` even contains, which is exactly what laziness defers. ``lazy`` +also inherits :pep:`810`'s scope restriction: it's only valid at module +level, not inside functions, classes, or ``try`` blocks. + +NumPy's ``numpy/__init__.py`` illustrates why this matters. Its +module-level ``__getattr__`` does two unrelated jobs at once: lazily +loading submodules that aren't imported at ``import numpy`` time, and +raising helpful errors for attributes that no longer exist: + +.. code-block:: python + + # numpy/__init__.py, today (abbreviated) + def __getattr__(attr): + # Warn for expired attributes + import warnings + + if attr == "linalg": + import numpy.linalg as linalg + return linalg + if attr == "fft": + import numpy.fft as fft + return fft + # ... one branch like this per lazily loaded submodule ... + if attr in __expired_attributes__: + raise AttributeError(f"`np.{attr}` was removed. ...") + raise AttributeError(f"module {__name__!r} has no attribute {attr!r}") + +Only the first job is a laziness concern, and lazy exports replace it +directly: + +.. code-block:: python + + # numpy/__init__.py, with lazy exports + lazy from . export linalg + lazy from . export fft + # ... one statement per lazily loaded submodule ... + + def __getattr__(attr): + # Only the expired-attribute branch is left + if attr in __expired_attributes__: + raise AttributeError(f"`np.{attr}` was removed. ...") + raise AttributeError(f"module {__name__!r} has no attribute {attr!r}") + +This isn't only shorter, it's more correct. Today, ``linalg`` exists only +through the ``__getattr__`` fallback, so it's invisible to ``dir(numpy)`` +and tab completion unless NumPy separately maintains a ``__dir__`` +override listing it. ``lazy from . export linalg`` binds a real (lazy) +attribute immediately and adds ``"linalg"`` to ``__all__``, so ``dir()`` +and ``__all__`` are correct automatically, and ``__getattr__`` is no +longer even called for these names, since ordinary attribute lookup now +succeeds before it would run. + +The second job, warning for attributes that no longer exist at all, isn't +something ``export`` addresses. ``export`` only concerns names that +should be bound; it has nothing to say about names that were removed. A +module ``__getattr__`` is still needed for that, just a smaller one, with +only the deprecation logic left in it once the lazy-submodule branches +move out. + +One restriction matters for this rewrite: :pep:`810` disallows ``lazy`` +inside function bodies, so the lazy-submodule branches must move out of +``__getattr__`` to module top level, not be replaced line by line inside +it. That's a restructuring, not a drop-in substitution, though it's also +exactly the shape a hub module (`How to Teach This`_) already takes. + + +Interaction with ``__all__`` +----------------------------- + +A module may freely mix ``from ... export ...`` statements with a +manually maintained ``__all__``, or with ``__all__ +=`` / +``__all__.append`` calls elsewhere in the file. Each ``export`` statement +simply appends to whatever ``__all__`` already exists in the module's +namespace, creating an empty list first if necessary. Duplicate names are +allowed: ``__all__`` was never required to be free of duplicates, and this +PEP doesn't change that. + +``from ... export ...`` affects only the contents of ``__all__``, which in +turn affects ``from module import *`` and any tool that already reads +``__all__`` (documentation generators, linters, IDEs). + +``export`` cannot hide the intermediate submodule(s) named in ```` +from a hub module's own ``dir()``. When ``spam/__init__.py`` contains +``from ._internal.core export PublicAPI``, ``_internal`` becomes an +attribute of the ``spam`` module and shows up in ``dir(spam)``, regardless +of whether the statement uses ``import``, ``from ... import``, or +``export``. ``spam/__init__.py``'s own namespace *is* ``spam.__dict__``, +and Python's import system binds an imported submodule onto its parent +package's namespace as a side effect of importing it, for both packages +and plain modules. This is a property of the import system, not something +``export`` introduces or can suppress. It is one more reason +attribute-access hiding is a `non-goal `_ of this PEP. + + +Semantic implementation +------------------------ + +Each ``from export as `` statement is equivalent +to: + +.. code-block:: python + + from import as + __all__ = list(globals().get("__all__", [])) + [""] + +For example: + +.. code-block:: python + + # from ._internal.core export PublicAPI + from ._internal.core import PublicAPI + __all__ = list(globals().get("__all__", [])) + ["PublicAPI"] + + # from ._internal.widgets export Widget as PublicWidget + from ._internal.widgets import Widget as PublicWidget + __all__ = list(globals().get("__all__", [])) + ["PublicWidget"] + +The wildcard form's equivalent is given in `Wildcard form`_, and the lazy +form's in `Lazy exports`_. + +Like other soft keywords, ``export`` remains a valid identifier everywhere +except immediately after ``from `` in an import statement. + + +Rationale +========= + +Why a keyword and not a decorator +----------------------------------- + +A ``@public``-style decorator, as in the third-party ``atpublic`` +package, works neatly for individually defined functions and classes, +but it doesn't compose with ``import`` statements: there's no object to +decorate when the "definition" is just a name entering the module +through an import. +``atpublic`` works around this with a function-call form, +``public(some_imported_name)``, but that reintroduces the double-write +this PEP removes: the name is written once in the import and again as an +argument to ``public()``. It also doesn't compose with aliases: the +function-call form only takes keyword arguments, ``public(alias=name)``, +which both adds ``alias`` to ``__all__`` and binds it, so publishing an +alias means spelling out the mapping in the call instead of using +``from x import y as z``. A statement-level ``export`` keyword avoids +both problems, because it's part of the import statement itself; it +adds nothing beyond the import that would exist anyway. + +Why only re-exports +----------------------------- + +This PEP deliberately omits a way to mark a fresh ``def``, ``class``, or +assignment as exported where it's defined. Some smaller libraries and +single-file modules would rather write ``export def public_function(): +...`` right where the function is defined, but that use case doesn't +share the DRY problem this PEP solves. When a name is defined and +exported in the same place, it's written only once; the maintainer +already chooses whether to write a leading underscore, and tools such as +``atpublic``'s ``@public`` decorator already let that choice happen at +the definition site, without a new statement. + +Smaller libraries and single-file modules that don't organize their +public layout around a re-export hub don't need this PEP at all: +``atpublic`` already serves +them. Conversely, a new project that does adopt hub-and-internals from +the start has little use for ``atpublic`` either: everything meant to be +public is already flowing through the hub's ``export`` statements. If +``atpublic`` gains wide enough adoption regardless, it, or something +like it, may eventually belong in the standard library, independent of +this proposal. + +The evidence gathered for this PEP (NumPy, pandas, polars, Typer, FastAPI, +Plotly) is uniformly about re-export hubs, not about individually defined +names wanting a decorator. A single statement form, one that extends the +familiar ``from ... import ...`` rather than teaching new prefix rules for +``def``, ``class``, and assignment statements, keeps the grammar easy to +describe and easy to review. A later, separate PEP remains free to +propose a definition-site marker if real-world evidence for that gap +emerges; this PEP doesn't need to solve it to solve the re-export problem. + +Why no runtime enforcement +----------------------------- + +The author finds runtime access restriction appealing on its own merits, +and excludes it here purely on scope grounds. :pep:`842` proposes exactly +this: an ``ExportWarning`` when code accesses a non-exported attribute. +Its discussion thread spent considerable effort on whether that access should +warn, raise, or do nothing, and on how such enforcement should interact +with legitimate internal access, drawing substantial pushback over +adversarial framing, per-access performance overhead, unreliable warning +filters, and breakage of patterns like pip's: pip has no public API at +all, yet still supports tools such as pip-tools that deliberately import +``pip._internal``. None of that debate touches the DRY problem this PEP +solves. + +The export bookkeeping problem and the "should Python police access to +internals" problem are separable. This PEP resolves only the former, +leaving module-boundary conventions (a leading underscore on a submodule, +or a private subpackage) to handle the latter. Those conventions already +work, and already ship in every library discussed in the thread. + +If runtime enforcement is wanted later, a separate proposal can layer it +on top of an accurate, non-duplicated ``__all__``, without entangling it +with the syntax that produces that ``__all__`` in the first place. + + +Backwards Compatibility +======================== + +``export`` is a soft keyword, following the same approach as ``match``, +``case``, and ``type``. Python treats it specially only in the one +position where ``import`` is otherwise required: immediately after ``from +``. Existing code that uses ``export`` as a variable, function, +parameter, or module name keeps working unchanged, including the unusual +but valid case of a module literally named ``export``. + +``from ... export ...`` only affects ``__all__``, which every Python +version already understands. Libraries that support versions before this +feature lands can write both forms, and drop the older one once their +minimum supported version catches up: + +.. code-block:: python + + # Python < 3.16 + from ._internal.core import PublicAPI + __all__ = ["PublicAPI"] + + # Python >= 3.16, once adopted + from ._internal.core export PublicAPI + + +Security Implications +======================= + +This PEP has no known security implications. + + +How to Teach This +==================== + +Documentation should teach ``from export `` as part of a +named layout, the **hub-and-internals** pattern, not as an isolated +statement. A package following this pattern has two kinds of module: + +* One **hub** module, typically ``__init__.py`` (a package can have more + than one, such as ``numpy.typing``), whose only job is gathering names + from internal modules and re-exporting them. A hub module contains + ``export`` statements and nothing else that touches ``__all__``; it + never declares ``__all__`` directly, since ``export`` builds it. +* Any number of **internal** modules, conventionally named with a leading + underscore or nested under a leading-underscore subpackage, holding the + actual implementation. Internal modules declare no ``__all__``: they + aren't meant for direct import by users, so there's nothing for + ``__all__`` to curate. + +.. code-block:: python + + # spam/__init__.py (the hub) + from ._internal.core export PublicAPI + from ._internal.widgets export Widget, Gadget + + # spam/_internal/core.py (internal -- no __all__) + class PublicAPI: + ... + + # spam/_internal/widgets.py (internal -- no __all__) + class Widget: + ... + + class Gadget: + ... + + class _Helper: + ... + +This gives one rule to teach: *if a name needs to reach users, write one +``export`` statement for it in the hub; everything else stays unexported +by default.* The whole package declares its public layout in exactly one +place, built from statements that would exist anyway, to make the names +available at all. + +Style guides that currently recommend ``from import as +`` for re-exports can point to ``export`` instead. ``export`` covers +conditional re-exports too, such as picking a platform-specific +implementation (see `Specification`_). Direct ``__all__`` manipulation +remains available, and still necessary, outside the hub-and-internals +pattern, for names discovered programmatically at runtime rather than +through a single import statement, such as a loop that registers +plugins. + + +Reference Implementation +=========================== + +No reference implementation exists yet. A prototype could be built as a +source-to-source transform (similar to early prototypes of ``match`` +statements) before committing to grammar changes in CPython. + + +Rejected Ideas +================ + +Alternative surface syntax +------------------------------- + +This PEP considered two other spellings for the re-export statement: + +* ``export from ``, mirroring ECMAScript's ``export ... + from ...``. Rejected because it puts the name before the module, + reversing the order every Python import statement uses, for no benefit + beyond matching another language's convention. +* ``export from import ``, prefixing an ordinary ``from + ... import ...`` statement with ``export``. An earlier draft of this + proposal used this form. Rejected in favor of ``from export + ``, because a leading ``export`` in front of a complete import + statement reads as two verbs for one action, and because replacing + ``import`` in place keeps the keyword's special-cased position to one + spot in the grammar, rather than requiring the parser to recognize + ``export`` as a prefix before several statement kinds. + + +Open Issues +============= + +Should ``export *`` require the source module to define ``__all__``? +--------------------------------------------------------------------- + +The two-tier layout that motivates the wildcard form (see +`Specification`_) depends on the internal module having deliberately +curated its own ``__all__``: that curated list is the reason the hub's +``export *`` is safe to write without naming each item. + +But ``from export *`` behaves exactly like ``from +import *``, which falls back, when ```` defines no ``__all__``, +to binding every top-level name that doesn't start with an underscore. +If a hub author writes ``export *`` against an internal module with no +``__all__``, that fallback silently re-exports whatever happens to lack +a leading underscore, names that may not have been curated as carefully +as an explicit ``__all__`` would require, and exactly the kind of +accidental export this PEP eliminates elsewhere. + +The open question: should ``export *`` follow ``import *``'s fallback as +is, or require ```` to define its own ``__all__`` and raise an +error if it doesn't? + + +Acknowledgements +=================== + +This PEP grew out of discussion on :pep:`842`, particularly +contributions from Peter Bierma, Alex Grönholm, Guido van Rossum, Barry +Warsaw, and Hugo van Kemenade, who supplied the real-world examples of +re-export breakage that ground this proposal in concrete libraries +rather than hypotheticals. + + +Copyright +=========== + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.