refactor: introduce `TemplateProgram` between AST and `CompiledProgram` by apoelstra · Pull Request #388 · BlockstreamResearch/SimplicityHL

apoelstra · 2026-08-13T23:31:56Z

This PR does a number of refactors ahead of introducing quoting and canonical CMRs. The goal of these changes is to rearrange compilation.

Rather than the process being

Templated AST (TemplateProgram) -> instantiated AST -> compiled program

instead it will be

Templated AST -> Templated compiled program -> compiled program

I tried to reduce churn but there are a couple renames that come with this:

TemplateProgram becomes TemplateAst (and a new TemplateProgram type is introduced)
WitnessNames, which is the name of the witness type for our compiled programs, becomes TemplateProgramWitness, and is expanded to track both witness names and parameters

In fact, WitnessNames is already (ab)used to store both witness names and parameters, which makes these refactors a little easier. But this appears to be a form a code reuse rather than a deliberate API choice; the current WitnessName struct is just a wrapper around Arc<str> that does not track what kind of object it's holding, and we distinguish based on what context it appears in. Because we instatiate before compiling, any remaining WitnessNames in a compiled program are definitely witness names and not parameter names. This PR changes that, so we need to extend the type.

Once these refactors are done, the actual rearranging of the instantiation step is suprisingly easy. But it will have to wait for the next PR.

This PR has no behavior changes, does not introduce any new functionality, and does not use any features of the upcoming rust-simplicity.

Changes the `as_inner` method to return an `&Arc` rather than an `&str`, which makes more sense given the name, and is also useful in a couple of places. Then * Add `PartialEq` between the wrapper types and `str`, eliminating a ton of .as_inner calls entirely. * Add a `from_ident` method which converts an `Identifier` into the type, which is a common operation throughout the codebase to "strongly type" an identifier. * Add a `from_str` method to cover the remaining cases where we really do need an explicit &str

The existing code has an interesting sort of type confusion: we parse Arguments, UnresolvedValues, and WitnessValues by means of a ParserVisitor which calls into chumsky to parse the key. For each of these cases, we expect an identifier, which depending on the map, may reference either a parameter or a witness. In both cases, we put them into the WitnessName type, which covers both but does not distinguish between them. To achieve this, in src/parse.rs we implement ChumskyParse on WitnessName, by parsing an identifier and then interpreting it as a WitnessName. (We never use this codepath when parsing actual programs; in actual programs identifiers are just Identifiers until a later resolution stage; this ChumsyParse impl is only used as a helper for serde-deserialization of .args and .wit files.) (In programs, witnesses and parameters must be prefixed with `witness::` or `param::`, and these have their own parsing path.) *However*, in the following commits, we will update WitnessNames so that it distinguishes between parameters and witnesses, because we are going to delay parameter resolution until after compilation. Once we do this, the ChumskyParse impl for WitnessName will no longer be tenable, because it needs to construct a WitnessName but it doesn't know whether the identifier it's converting is supposed to be a parameter or a witness name. This commit simply refactors code and does not change any data structures or behavior.

Our "template program" is really a "templated SimplicityHL AST". I'd like to introduce a new "templated Simplicity Program" for which the name TemplateProgram would be a better fit. So rename this out of the way. This commit can be reproduced with find bitcoind-tests external-jet-lib-example src fuzz tests \ -type f \ \( -name '*.rs' -o -name '*.md' \) \ -exec sed -i s/TemplateProgram/TemplateAst/g {} \;

Over the next couple commits I am going to generalize WitnessName to cover multiple possibilities (both named witnesses and parameters, in a new "template program" type). Start by moving stuff around, and replacing the macro-generated accessors with hand-written ones.

We are already using this type for both parameters and witnesses. Update the name to reflect that. This commit is just a search-and-replace. It is hopefully easy to review even though it's big, for that reason.

There are two purposes to this trait: * Moves a bunch of function definitions out of macro-generated code and into default trait methods; this is easier to read and improves LSP integration * Introduces the ident_to_key method, which will allow Arguments and WitnessValues to both be created from the UnresolvedValues map. Currently this is done using a From<HashMap> bound, but when we extend WitnessNames to hold parameters as well as witness names, this From bound won't be sufficient. Also, change UnresolvedValues to be keyed by Identifier rather than by WitnessName. Until we resolve the name, we don't know whether we have a parameter name or a witness name, so until then, Identifier is the more correct type.

stringhandler

ACK 9942a5a Ran cargo test locally

apoelstra · 2026-08-17T15:02:29Z

Oops, this triggers a fuzz failure. Will track it down and add a unit test.

After all our prep work this is fairly easy to do. The one tricky thing was that I had to adjust the Arbitrary impls for WitnessValues and Arguments so that their arbitrary values contain only the appropriate kind of TemplateProgramWitness.

apoelstra · 2026-08-17T20:41:30Z

Ah the failure was just that my Arbitrary impls were bad. No need for a unit test. The fuzztests are effectively the unit test.

apoelstra · 2026-08-17T22:12:49Z

On 917c572 successfully ran local tests

stringhandler

utACK 917c572

stringhandler · 2026-08-18T07:26:10Z

+    ///
+    /// The supplied `arguments` are consistent with the program's parameters.
+    /// Call [`Arguments::is_consistent`] before calling this method!
+    pub fn instantiate(self, _: Arguments) -> Arc<named::CommitNode> {


Strange to have the unused variable here. Perhaps we need a comment explaining its reason?

It will be used in the next PR.

KyrylR

ACK 917c572

nit: cargo test --locked --manifest-path ./bitcoind-tests/Cargo.toml fails

Not sure how to deal with this crate rn, though it would be nice if it were better integrated into the tests (if it is worth it)

KyrylR · 2026-08-19T09:53:22Z

 pub use crate::value::Value;
+pub use crate::witness::{
+    Arguments, Parameters, WitnessNameToValueMap as _, WitnessTypes, WitnessValues,
+};


nit: it looks like now we have two exports of WitnessNameToValueMap, one below that is gated by serde feature, and this one

Oo, good catch. I'm not sure why I added this new export. It kinda looks like something my IDE would've done, but I don't think it'd ever add a pub use automatically.

Oh, I think actually it was just a rebasing mistake. Anyway, fixed.

Currently this doesn't do anything. It just adds an extra step to compilation. The next commit will rearrange things such that instantiation happens -after- compilation.

apoelstra requested a review from delta1 as a code owner August 13, 2026 23:31

apoelstra added 6 commits August 13, 2026 23:37

template_program: rename WitnessName to TemplateProgramWitness

4fc6698

We are already using this type for both parameters and witnesses. Update the name to reflect that. This commit is just a search-and-replace. It is hopefully easy to review even though it's big, for that reason.

apoelstra force-pushed the 2026-08/add-template-program branch 3 times, most recently from 4881f30 to 9942a5a Compare August 14, 2026 02:54

stringhandler approved these changes Aug 14, 2026

View reviewed changes

apoelstra force-pushed the 2026-08/add-template-program branch from 9942a5a to 917c572 Compare August 17, 2026 20:40

stringhandler approved these changes Aug 18, 2026

View reviewed changes

KyrylR approved these changes Aug 19, 2026

View reviewed changes

KyrylR reviewed Aug 19, 2026

View reviewed changes

template_program: introduce TemplateProgram type

a62a3ff

Currently this doesn't do anything. It just adds an extra step to compilation. The next commit will rearrange things such that instantiation happens -after- compilation.

apoelstra force-pushed the 2026-08/add-template-program branch from 917c572 to a62a3ff Compare August 19, 2026 12:54

Navigation Menu

Uh oh!

refactor: introduce `TemplateProgram` between AST and `CompiledProgram` - #388

refactor: introduce `TemplateProgram` between AST and `CompiledProgram`#388
apoelstra wants to merge 8 commits into
BlockstreamResearch:masterfrom
apoelstra:2026-08/add-template-program

apoelstra commented Aug 13, 2026

Uh oh!

stringhandler left a comment

Uh oh!

apoelstra commented Aug 17, 2026

Uh oh!

apoelstra commented Aug 17, 2026

Uh oh!

apoelstra commented Aug 17, 2026

Uh oh!

stringhandler left a comment

Uh oh!

stringhandler Aug 18, 2026

Uh oh!

apoelstra Aug 18, 2026

Uh oh!

stringhandler Aug 18, 2026

Uh oh!

KyrylR left a comment •

edited

Loading

Uh oh!

KyrylR Aug 19, 2026

Uh oh!

apoelstra Aug 19, 2026

Uh oh!

apoelstra Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

apoelstra commented Aug 13, 2026

Uh oh!

stringhandler left a comment

Choose a reason for hiding this comment

Uh oh!

apoelstra commented Aug 17, 2026

Uh oh!

apoelstra commented Aug 17, 2026

Uh oh!

apoelstra commented Aug 17, 2026

Uh oh!

stringhandler left a comment

Choose a reason for hiding this comment

Uh oh!

stringhandler Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

apoelstra Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

stringhandler Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

KyrylR left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KyrylR Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

apoelstra Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

apoelstra Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KyrylR left a comment •

edited

Loading