Feat/cfn test v2 rqts runner - #1122
Open
ruslan-aws wants to merge 3 commits into
Open
Conversation
Add an opt-in --v2 flag to cfn test that runs contract tests through the RQTS (CTv2) executor container instead of the pytest-based suite. Without --v2 the existing path is unchanged. The new rpdk.core.rqts package resolves and pulls the pinned public executor image with a bounded retry and a cached-image fallback, aggregates Docker, artifact and credential preconditions into a single error, builds the docker run argv, and orchestrates the run: mint temporary credentials, stream container output, and map the container exit code to the CLI result. Credential values reach Docker only through the client process environment. The argv carries name-only -e flags, so secrets never appear in the command line or in debug logs. Handler invocation uses DirectJar mode, so no host networking and no handler endpoint are configured, and scenario selection is owned by the executor image. Also add --rqts-image to override the image reference when testing pre-release executor builds.
DEFAULT_HANDLER_PORT described the SAM Local handler endpoint, which the DirectJar mode targeted by cfn test --v2 never uses: the handler JAR is loaded in-process, so no port is bound and nothing referenced the value.
Bring the rqts package to full statement and branch coverage by exercising the paths the existing tests bypassed: - _run_docker reaching subprocess.run with a fixed, non-shell argv - image_present_locally when docker inspect cannot run at all - the Docker daemon ping exiting non-zero, raising, or timing out - RqtsRunner.run aggregating unmet preconditions into one error - build_docker_argv with explicit workdir, artifact and output overrides
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: N/A
Description of changes:
Adds an opt-in
--v2flag tocfn testthat runs contract tests through theRQTS (Resource Quality Test Suite) executor container instead of the existing
pytest-based suite. Without
--v2,cfn testbehavior is unchanged.Motivation
RQTS is the container-based contract test runner already used for cloud
contract-test execution. This flag lets resource developers run the same
scenarios locally against a packaged handler artifact, so local results match
what the registry runs during
RegisterType/TestType.What's added
--v2(defaultFalse): run the RQTS local test runner.--rqts-image: override the executor image reference, for testing against apre-release image. Defaults to the published
public.ecr.aws/s5r7m5i4/cfn-rqts-executor-external:latest.rpdk.core.rqtspackage:image.py— image resolution, and an anonymousdocker pullattempted onevery run (bounded to 3 attempts with a per-attempt timeout). If every
attempt fails, a locally cached copy is used with a warning; only when no
cached copy exists is the failure fatal.
preconditions.py— checks Docker availability (CLI on PATH plus adaemon ping), the built artifact package, and AWS credentials/region.
Every unmet precondition is reported in a single error rather than failing
on the first one.
argv.py— builds thedocker runargv and the container environment.runner.py— orchestrates the pipeline: artifact-type guard, preconditions,credential minting, image resolve/pull, container run with live output
streaming, and exit-code mapping.
Behavior
The container is invoked in the executor's DirectJar handler mode:
Notable points:
-eflags are name-only; values aresupplied through the Docker client process environment. The full command line
is therefore safe to log at DEBUG and never appears in
psoutput.endpoint is probed and no host networking is configured.
schema capability and resource-type namespace. The CLI passes no scenario or
exclusion arguments.
rqts-output/under the project root. A zero containerexit reports a pass; any non-zero exit raises the usual
SysExitRecommendedError, whichcli.pymaps to exit code 1.module projects short-circuit earlier as they do today.
Backward compatibility
--v2branch is additive and sits after the existing moduleshort-circuit; the legacy path is untouched.
RqtsRunneris imported lazily inside the branch, so the pytest path pays noimport cost.
--v2the pytest path runs, the runner is neverconstructed, and a non-zero pytest return still raises as before.
--v2makes the previously-undeclared--vabbreviation ambiguous with
--verbose, socfn test --vnow errors withambiguous option: --v could match --verbose, --v2.-v,-vv,--ve, and--verboseare unaffected.Testing
tests/rqts/, plus flag-registration and routing tests intests/test_test.py. Hypothesis property tests cover image resolution, thebounded/anonymous pull, exact precondition aggregation, the bind mount,
credentials never appearing in argv, DirectJar selection, region handling,
absence of SAM Local/remote-lambda flags, absence of host networking, output
directory, exit-code mapping, and absence of scenario/exclusion flags.
subprocess.run,subprocess.Popen,and the credential helpers are patched throughout.
rpdk.core.rqtsis at 100% statement and branch coverage.image pull, container execution, streamed output,
rqts-output/results, andexit-code mapping.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.