docs: add Pipeline Policy Constraints with Tekton and Kyverno solution - #830
docs: add Pipeline Policy Constraints with Tekton and Kyverno solution#830l-qing wants to merge 8 commits into
Conversation
afe55c1 to
99586ca
Compare
Add a Solution-kind article at docs/en/solutions/Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md covering admission-time policy constraints for Tekton pipelines with Kyverno on ACP: template/Task identity locking, gate-parameter effective-value validation, source/target restrictions, result-driven cancellation, scoped exemptions (PolicyException), and rollout/upgrade operations. Verified against ACP 4.3 with Alauda DevOps Pipelines v4.14.x, Kyverno v1.15.9-v4.3.2, and Alauda Artifact Hub Shim v1.0.0; every code block was validated on a live cluster. ProductsVersion is anchored to 4.3.x; the id frontmatter field is left for the add_id.sh flow to assign. Note: the document deliberately ships a fake, format-valid Slack token as the fixture for the trivy secret-scanning demonstration; it must stay format-valid for trivy to detect it.
99586ca to
2eb6d9c
Compare
yuzichen12123
left a comment
There was a problem hiding this comment.
本次结合 DEVOPS-43870 及其关联文档单 DEVOPS-43674 的验收标准完成审查,共提交 4 条行级评论:2 个 Major、2 个 Minor。重点是修正文档对 mutate-existing 故障语义的安全承诺,并补齐可直接执行、可重复验证的 probe 与测试证据。指定验收环境当前持续返回 EOF,待问题修复且环境恢复后,应补做目标版本的故障注入与完整正/负回归。
|
|
||
| | What to budget | Mechanism facts | The budget / action you must set | | ||
| |---|---|---| | ||
| | **External calls on the admission path** | Two kinds of criteria wait for an external round trip inside admission: `context.imageRegistry` ([§4.5.2](#s4-5-2)) and `context.apiCall` (one each in [§4.2.1](#s4-2-1) / [§4.2.2](#s4-2-2) / [§4.6.1](#s4-6-1) — three in the whole document). **Both fail closed**: registry unreachable → request denied (measured at roughly 5 seconds; roughly 3 seconds when reachable, see limitation 4 in [§4.5.2](#s4-5-2)); apiCall cannot fetch its target → the rule errors out and the request is denied (the error message shape is in the [§4.2.1](#s4-2-1) warning) | Decide explicitly "which request paths may carry external calls"; narrow such rules' match down to **the Tasks that genuinely need them**; load-test p95 / p99 and the timeout ratio, and confirm it stays below the webhook timeout — **that ceiling applies to the whole request, not to each rule separately** (measured `timeoutSeconds=10` in this document's verification environment, see checklist item 6 in [§3.1](#s3-1); one 5-second registry round trip fits, two stacked on the same request may not). **Registry / API server jitter turns directly into pipeline creation failures** — have the matching alerts and playbooks ready | |
There was a problem hiding this comment.
Major|mutate-existing 的 apiCall 故障语义被误写为 fail-closed
这里把 §4.2.1、§4.2.2、§4.6.1 的三个 context.apiCall 一概描述成“查询失败时 rule error,原请求被拒绝”。这只适用于同步 validate 路径,不能套用到 mutate-existing。
Kyverno 1.15 上游实现中,mutate-existing 由 handleBackgroundApplies 异步处理,并且只为 RuleStatusPass 创建 UpdateRequest;context/precondition 得到 RuleStatusError 时不会进入 admission 的 BlockRequest。因此 §4.2.2/§4.6.1 查询父 PipelineRun 失败时,原 TaskRun CREATE 或 status UPDATE 仍可能被接受,但预期的 CancelledRunFinally 补丁不会产生。此时是 fail-open,流水线可能继续执行。
修复目标:
- 分开描述同步 validate(§4.2.1)与异步 mutate-existing(§4.2.2/§4.6.1)的错误语义,不再使用统一的 “Both fail closed”。
- 明确 mutate-existing 在
apiCall、规则求值或UpdateRequest创建失败时不会完成取消,并补充 UpdateRequest、background-controller event/error 的监控与告警要求。 - 说明需要零竞态硬保证时应使用同步 validate/admission-mutate,而不能把异步取消当作硬门禁。
- 在目标 Kyverno v1.15.9-v4.3.2 环境补一条受控故障用例:制造父对象查询失败,验证原请求、取消补丁和告警三者的实际结果,并据此更新文字。
There was a problem hiding this comment.
补充目标环境实测证据(2026-08-11,Tekton Pipelines v1.12.2,Kyverno v1.15.9-v4.3.2):
- 正常对照组:mutate-existing 规则的
context.apiCall查询当前真实 PipelineRun,并校验返回 UID;PipelineRun 正常完成,目标注解成功写入,证明规则、RBAC 和 background mutation 路径本身可用。 - 故障注入组:保持 match、subject、target、RBAC 和触发方式不变,仅将
apiCall指向同命名空间内一个确定不存在的 PipelineRun。结果是原 PipelineRun 的 status UPDATE 未被拒绝,最终状态为Succeeded;预期 mutation 注解未写入;查询时没有保留该策略对应的 UpdateRequest,admission/background controller 日志中也没有观察到包含该测试策略名的错误记录。 - 因此差异可以归因于
apiCall故障:在 mutate-existing 路径中,实际行为是“原请求继续、mutation 消失”,不是本行描述的 fail-closed。且本次未自然产生可操作告警,文档还需明确补充监控手段,不能只要求查看 background-controller event/error。
这也进一步确认修复目标 1–4 均为必须项,尤其应把“异步取消失败”作为显式验收用例。
There was a problem hiding this comment.
已在 c966ff7 修复——也感谢你补充的受控故障注入实测,与机制推演完全吻合。
文档不再用一句 "both fail closed" 概括三处 apiCall:
- §3.7 拆成两行:「准入路径上的外部调用(同步 validate)」——只有 §4.2.1(以及 §4.5.2 的
imageRegistry检查)是 fail-closed;新增一行**「异步投递链路(mutate-existing 取消是 fail-open)」**覆盖 §4.2.2 / §4.6.1 / §4.6.2:该链路上任何一环失败(apiCall 查询、UpdateRequest 创建、background-controller 可用性、目标对象 update RBAC)都会放行原请求,取消补丁静默消失。 - 同一行明确**「判据方向 fail-closed ≠ 交付保证」**,并列出必须的监控项:background-controller 存活/积压、UpdateRequest 创建与失败量、策略命中数与实际打上补丁对象数的差值,以及上线前和每次升级后的受控故障注入演练(正是你复现的场景)。
- §2 观察点表与 §4.6.1 带上了同样的限定;文档现在明确写出:零竞态硬保证必须走同步路径(§4.2.1 deny / §4.2.3 admission mutate),异步取消不能当硬门禁使用。
|
|
||
| Later sections supply probes in one of two ways: some give complete manifests and commands directly ([§3.3](#s3-3), [§4.4.1](#s4-4-1), [§4.4.2](#s4-4-2), [§4.6.1](#s4-6-1)); the others give only an **expectation table** (listing which inputs should be allowed / denied / skipped) and leave the commands to this section — because these three kinds of commands are mechanical, and repeating them section by section would make the document longer, not clearer. | ||
|
|
||
| **Eight sections give only an expectation table and need you to turn it into commands yourself**: [§4.1.1](#s4-1-1), [§4.2.4](#s4-2-4), [§4.5.1](#s4-5-1), [§4.5.3](#s4-5-3), [§4.5.4](#s4-5-4), [§4.5.5](#s4-5-5), [§5.2](#s5-2), [§5.3](#s5-3). (The method behind this list is "the section has an expectation table but no `kubectl create` that submits a probe" — after the document changes, recount by that method instead of trusting the list. The `kubectl apply` commands already present in [§4.5.3](#s4-5-3), [§5.2](#s5-2), [§5.3](#s5-3) create **prerequisite objects** (ConfigMaps, namespaces), not the probes themselves.) In addition, [§4.1.4](#s4-1-4) and [§4.1.5](#s4-1-5) have no expectation table: they are Audit defense in depth on `*/status`, their criteria are stated in the body, and the expected shape is "which fail entry should appear in the report" — follow **type 2** below. |
There was a problem hiding this comment.
Major|缺少可直接执行的 probe 和可复核的 731 条断言产物
本节明确说明有 8 个章节只有 expectation table,需要读者自行选择 skeleton、编辑字段并生成 probe.yaml。受影响的章节包括 §4.1.1、§4.2.4、§4.5.1、§4.5.3、§4.5.4、§4.5.5、§5.2、§5.3。这与前文对 walkthrough operator 的 “Every step copy-pasteable” 承诺以及 Jira 的“所有示例 YAML 可直接执行并验证通过”验收标准不一致。
另外,PR 描述声称已完成 731 条 live assertions,但当前变更只包含 Markdown,没有提交或链接脱敏后的 harness、输入版本、执行日志/结果摘要以及“策略/探针 → 断言”映射。现有 GitHub build 只能证明站点可以构建,不能证明策略行为和清理流程已经回归。
修复目标:
- 为上述 8 个章节补齐完整的正/负 probe manifest,可集中成多文档 YAML,但必须能在替换环境参数后直接提交。
- 每个 probe 提供实际执行命令、机器可判定的预期结果和清理命令,避免依赖读者自行解释输出。
- 将脱敏后的测试 harness 纳入仓库,或提供稳定、可访问的结果产物;记录 ACP、Pipelines、Kyverno、Artifact Hub Shim 的精确版本。
- 建立每个完整策略及 probe 到断言结果的映射,使升级后可以重复回归。若本次只计划交付设计矩阵,则应删除 copy-paste/已全部验证的承诺,并同步调整验收范围。
There was a problem hiding this comment.
已在 c966ff7 部分采纳,并说明一下范围口径:
- "Every step copy-pasteable" 的承诺已删除。读者画像行现在明确写:策略与运行清单可直接复制;九个小节的探针按 §3.4.1 拼装——该节现在给出通用配方(三类探针的具体命令、身份参数与取证路径)加三份现成骨架 manifest,并逐一点名这九个小节。
- 关于 harness 与日志:我们有意不把过程性产物(测试 harness、执行日志、逐条断言结果)放进本仓库——这是文档仓库,不适合承载过程数据。PR 描述里的断言数量是内部验证的背景说明,不是本 PR 的交付物。
- 也说明一下与此选择相关的定位:本 KB 的目标是讲清底层机制(准入阶段、resolver 行为、mutate-existing 投递语义、判据设计)。具体的 Task/Pipeline 定义因环境和用户而异——读者需要按自己的契约改判据,这正是若干小节刻意提供期望表 + 骨架、而非钉死环境的探针的原因。
|
|
||
| **The key criterion** — each of the three channels locks the complete identity, and a false union is a denial: | ||
|
|
||
| ```yaml |
There was a problem hiding this comment.
Minor|这些 yaml fence 是非完整片段,无法独立解析
这里及另外 9 处 “Key criterion” 代码块只截取了嵌套字段,缺少外围对象。逐块运行 yq eval 时共有 10/69 个 yaml fence 解析失败,起始行分别为 2140、2434、2637、2886、3577、4413、5337、5564、6052、7675。后续完整策略 YAML 可以解析,因此这不是策略本体错误,但当前语言标记容易让读者和自动化把它们当成可复制的 YAML 示例,也与“所有示例 YAML 可直接执行”的表述冲突。
修复目标:
- 如果只用于解释逻辑,将这 10 个 fence 改为
text,并在标题或块前明确标注non-executable YAML fragment。 - 如果希望保留
yaml,则补齐到可独立解析的最小对象。 - 修复后以“所有标为
yaml的 fence 均能逐块通过 YAML 解析”作为验收条件。
There was a problem hiding this comment.
已在 c966ff7 通过标注处理:这 10 个「关键判定」块是刻意的节选——用来讲解每条策略里起决定作用的那几行,完整可安装的 YAML 一定在同节的 details 块里。10 处现在都带了行内标注:"(fragment, not a complete manifest you can kubectl apply as-is; the full policy is in this section's details block)"。保留 yaml 标记是为了语法高亮;有了标注就不存在复制歧义,且"可复制"承诺现在只覆盖策略与运行清单(见探针那条评论的回复)。
|
|
||
| Requiring merely that "a task named `scan` exists in the inline definition" is an **insufficient** security check — an attacker can drop in a hollow no-op scanner, hang a never-true `when` on it so it gets skipped, set the gate switch to `false`, or order release before it / parallel to it. | ||
|
|
||
| To open up inlining safely, you must validate the complete contract set of [§2.3](#s2-3) (scanner identity, effective values of the gate switches, must-run, DAG d |
There was a problem hiding this comment.
Minor|4.14 安装链接当前跳转到了 4.12 首页
当前 https://docs.alauda.io/alauda-devops-pipelines/4.14/install.html 返回的 HTML 使用 meta refresh 跳转到 /alauda-devops-pipelines/4.12/,并且丢失了 install.html 路径。读者按这里查找 4.14 的安装和 TektonConfig 说明时,实际看到的是错误版本、错误页面,无法核对本文声明的 4.14 前提。
修复目标:
- 替换为实际存在的 4.14 安装/
TektonConfig文档地址。 - 如果 4.14 文档尚未发布,明确说明临时引用的版本、与 4.14 的差异及后续替换条件,不能把 4.12 首页标成 4.14 安装页。
- 验收时检查最终页面内容与版本,而不只检查 HTTP 200;当前页面虽然返回 200,但正文会继续重定向。
There was a problem hiding this comment.
作为已知问题处理,修复在途:Alauda DevOps Pipelines 的 4.14 文档即将上线,届时该链接会解析到正确版本。我们保留 4.14 URL 不动,让它在文档发布后自然生效,而不是把读者永久指到 4.12。
| kubectl get cm -n "$TEKTON_NS" resolvers-feature-flags -o jsonpath='{.data}{"\n"}' | ||
| # Expect enable-cluster-resolver / enable-hub-resolver / enable-git-resolver to be | ||
| # "true" as required by the resolvers you actually use | ||
| echo "hub default-type: $(kubectl get cm -n "$TEKTON_NS" hubresolver-config -o jsonpath='{.data.default-type}')" |
There was a problem hiding this comment.
Major|Hub 检查会误报就绪,文档中的官方模板在验收环境实际无法解析
在提供的 /root/config/kb.yaml 环境实测:
enable-hub-resolver=true,本行读取的default-type=artifact,因此当前检查会判定 Hub 能力正常;- 按文档完整坐标创建
java-image-build-scan-deploy@0.3和python-image-build-scan-deploy@0.3的真实 PipelineRun,二者均终止为CouldntGetPipeline,resolver 请求返回 404,status.pipelineSpec未生成;显式type=artifact也不会改变这个 endpoint; - 同一集群内置 Artifact Hub Shim 对这两个 Pipeline 以及
sonarqube-scanner@0.7、trivy-scanner@0.6、skopeo-copy@0.1的精确坐标均返回 200,解析 manifest 后参数、结果和 DAG 合约也与 §3.2 矩阵一致。问题是当前 resolver 配置的 endpoint 没有实际提供这套内置 catalog,而现有检查完全发现不了。
这会使所有依赖真实 Hub profile 的示例在 walkthrough 开始很久以后才以解析失败结束,用户容易误判为策略问题,也不满足“示例可直接执行验证”的验收目标。
修复目标:
- 这里不能只检查 feature flag 和
default-type;增加对当前配置的 artifact-hub-api endpoint 的真实 smoke test,至少验证本文固定使用的五个catalog/name/version/kind坐标可解析。 - 提供可复制命令并机器判定:官方 Pipeline 应生成
status.pipelineSpec,Task 应生成已解析 spec;404、认证失败或 endpoint 不一致必须在进入 Cookbook 前明确失败。 - 明确“内置 Shim”与 resolver 实际配置 endpoint 的关系;若验收环境配置漂移,应先修复环境再宣称本文 live assertions 通过。
- 把该 smoke test 纳入可重复 harness,避免仅验证 Shim 自身“有包”,却漏掉 resolver 实际“取不到包”的链路断点。
There was a problem hiding this comment.
已在 c966ff7 修复——§3.1 核查第 2 项不再只信 feature flags:
- 新增 2b 冒烟:从工作负载 Pod 直接探测实际配置的
artifact-hub-apiendpoint,用精确版本详情接口(.../api/v1/packages/<type>/<catalog>/<name>/<exact-version>)核对本文钉死的五个坐标(sonarqube-scanner/0.7、trivy-scanner/0.6、skopeo-copy/0.1、java-image-build-scan-deploy/0.3、python-image-build-scan-deploy/0.3)。 - 通过标准机器可判定且严于 HTTP 200:每个坐标必须返回 200 且
data.manifestRaw非空(包列表 200 不够——钉住的版本或其 manifest 仍可能缺失);任一坐标失败整块非零退出,问题在进入 Cookbook 之前就暴露,而不是很晚才以CouldntGetPipeline的形式出现。 - 通过标准表要求五个坐标全部可用,分诊列区分你遇到的两类失败:404 → endpoint/catalog 漂移(含 flags 全绿但 endpoint 指错的情况,按环境问题先修再继续);UNREACHABLE → 工作负载 Pod 的出口/DNS 不通。
- Split failure semantics of synchronous validate vs asynchronous mutate-existing paths: the latter is fail-open (request admitted, cancellation patch silently lost); add delivery-chain monitoring and fault-injection requirements, and state that hard guarantees need the synchronous paths - Harden the Hub readiness check: probe the five pinned exact-version coordinates against the configured artifact-hub-api endpoint and require HTTP 200 plus non-empty data.manifestRaw - Label the ten "Key criterion" excerpts as non-executable fragments; scope the copy-paste promise to policies and run manifests, with the probe recipe and skeletons consolidated in §3.4.1 - Rework demo cleanup to a UID ownership ledger for cluster-scoped objects instead of delete-by-name - Assorted criterion, probe-table, and placeholder-guidance refinements across §2, §3, §4 and §5
Port the corrections found by re-verifying the document against a live Kyverno v1.15.9 / Tekton cluster: - 4.4.1: the prose listed the 6th and 7th expected verdicts in the wrong order, and claimed the counting form and the [0] form are "identical" without saying on what. Scope it to the seven normal shapes and name the forged-condition construction as the only divergence. - 2.3 / 4.2.1: "no retry" means the run will not retry forever, not that the controller attempts creation exactly once; a single run's TaskRunsCreationFailed event can carry count > 1. Add the per-run counting command so the reader can check it. - 2.1 row 8: an apiCall's failure direction depends on the rule it sits on - fail-closed on a synchronous validate rule, fail-open on mutate-existing. - 6.2.3 row 4: fail-closed is the direction of the criterion, not a delivery guarantee. - 4.5.2: build and org.opencontainers.image.source may exist only one at a time; document the earlier failure point where a private or link-local www-authenticate realm is refused before authentication is attempted. - 4.6.1: a cancellation landing inside the child TaskRun's initialization window also ends in Failed, unrelated to whether the result was written. - Quote four bare <placeholder> occurrences in bash fences: unquoted < and > are shell redirections that silently truncate the command (also clears four SC2261 findings). - Rewrite six author-process passages into reader-facing statements.
Follow-up to the previous commit, from an independent review of it: - 6.2.3 said four paths "produce Cancelled" while the same paragraph and 4.6.1 both document cancellations that end in Failed. Say the paths request a cancellation and point at the terminal state that may differ. - 2.1 row 8 narrowed a failed apiCall to "a lookup that comes back empty", which drops the unreachable and forbidden cases 4.2.1 explicitly covers. - The factory resourceFilters set and the built-in hub's catalog name were stated as product-wide facts; both are defaults to verify against the cluster, and each sentence already carries the command that reads them.
| | 3 mutate-existing RBAC | Returns `yes` if you use the [§4.6](#s4-6) cancellation capability | On `no`, grant the aggregated ClusterRole given in the [§4.6](#s4-6) preamble (the `rbac.kyverno.io/aggregate-to-background-controller: "true"` label in its labels aggregates it into the background controller's permissions). **If you want to use a namespaced Role instead, you must also change `mutate.targets[].namespace` from `{{ request.namespace }}` to a namespace literal** — otherwise Kyverno's creation-time authorization check cannot resolve that variable, recognizes only cluster-level permissions, and the policy still fails to install (see the [§4.6](#s4-6) preamble). **If you do not install the [§4.6](#s4-6) cancellation policies, this permission is not needed** | | ||
| | 4 reports-controller reads status | All six `yes` (optional, not required) | A `no` **usually needs no action** (rationale in the third interpretation below). Only when some other feature genuinely needs the reports-controller to read status directly, add one more least-privilege ClusterRole the same aggregated way as item 3, with the aggregation label swapped to `rbac.kyverno.io/aggregate-to-reports-controller: "true"` | | ||
| | 5 PolicyException switches | Both `--enablePolicyException=true` and `--exceptionNamespace=<trusted-namespace>` present | Seeing only the former is ACP's default state — per [§3.1.1](#s3-1-1), write the `enabled` / `namespace` of `features.policyExceptions` into the kyverno `ModuleInfo`'s `spec.valuesOverride["ait/chart-kyverno"]` (**`ModuleInfo` exists only on the global management cluster**, see the warning in [§3.1.1](#s3-1-1)); **do not patch the Deployment args**. [§3.1.1](#s3-1-1) provides copy-pasteable atomic patch and rollback commands. **If you do not plan to use PolicyException exemptions ([§5.3](#s5-3)), you need not configure this** | | ||
| | 6 Webhook failure policy and timeout | **No fixed expected value** — it is planning input, not a pass/fail criterion. ⚠️ **The reading is timing-sensitive**: `kyverno-resource-validating-webhook-cfg` (the one that actually governs `PipelineRun` / `TaskRun` / `Pod`) is **generated dynamically by Kyverno from the installed policies** — with none of this document's policies installed its `webhooks` is empty and this line prints `<none>`. The `Fail/10` lines you can read at that point all belong to the webhooks of Kyverno's **own CRs** (policy / exception / cleanup / ttl). To get the actual values for pipeline resources, **come back and read this line after installing any [§4](#s4) policy** | Record the actual values and build your playbook around them: `Fail` requires guaranteed controller replicas and HA ([§6.1.8](#s6-1-8)); `Ignore` means accepting a policy vacuum while Kyverno is unavailable. `timeoutSeconds` is the **total budget for a single request** (the default under the applicable versions is `failurePolicy=Fail` / `timeoutSeconds=10`); the external calls in [§3.7](#s3-7) must fit inside that number. If you genuinely need to adjust it, go through the [§3.1.1](#s3-1-1) entry point as well — **do not edit the `ValidatingWebhookConfiguration` directly**: it is an object Kyverno itself maintains (it carries `webhook.kyverno.io/managed-by=kyverno`) | |
There was a problem hiding this comment.
Major|failurePolicy 的配置入口写错,后文要求的按策略分层实际无法落地
这里把生成后的 ValidatingWebhookConfiguration 当成主要观测点,并把调整入口导向 §3.1.1 的 ModuleInfo;但在本文适用的 Kyverno 1.15 中,经典 Policy 的策略级配置入口是 spec.webhookConfiguration.failurePolicy(旧 spec.failurePolicy 已废弃)。Kyverno CRD 明确说明:同一 Policy 内的规则共享该故障行为,允许 Ignore|Fail,默认 Fail。派生 webhook 只是 Kyverno 根据各 Policy 生成的生效结果,ModuleInfo/全局 forceFailurePolicyIgnore 也不能表达 §3.7 所要求的“硬门禁偏向 Fail、记账型 Audit 可选 Ignore”的逐策略分层。
我在更新后的验收环境(Kyverno 1.15.9-v4.3.2)用独立 namespaced Policy 设置 spec.webhookConfiguration.failurePolicy: Ignore 验证,生成的 resource webhook 确实进入 Ignore 分组;而本文 25 份 Policy YAML 均未声明 webhookConfiguration,因此当前示例全部落到默认 Fail,和后文的分层建议脱节。同步 apiCall / imageRegistry 的“错误时拒绝”结论也应明确以该 Policy 的 failurePolicy 为 Fail 为前提。
修复目标:
- 在每份完整 Policy 示例中显式声明
spec.webhookConfiguration.failurePolicy(以及需要定制时的timeoutSeconds),让安全意图进入 GitOps 管理的策略本体;硬门禁及同步外部调用通常选 Fail,选择 Ignore 的 Audit 策略需明确接受的真空边界。 - 检查项先读取 Policy spec 的配置意图,再核对 Kyverno 生成的 webhook 是否按 Fail/Ignore/timeout 分组生效;不要只读派生 webhook。
- 说明 ModuleInfo/全局 override 只能用于平台级强制覆盖,不能替代 per-policy 配置;仍然不要直接编辑生成的 VWC。
| | What to budget | Mechanism facts | The budget / action you must set | | ||
| |---|---|---| | ||
| | **External calls on the admission path (synchronous validate)** | Two kinds of criteria wait for an external round trip inside admission: `context.imageRegistry` ([§4.5.2](#s4-5-2)) and `context.apiCall` (one each in [§4.2.1](#s4-2-1) / [§4.2.2](#s4-2-2) / [§4.6.1](#s4-6-1)). **Only the two on the synchronous validate path fail closed**: registry unreachable → request denied (roughly 5 seconds; roughly 3 seconds when reachable, see limitation 4 in [§4.5.2](#s4-5-2)); the [§4.2.1](#s4-2-1) apiCall cannot fetch its target → the rule errors out and the request is denied (the error message shape is in that section's warning). **The apiCalls of [§4.2.2](#s4-2-2) / [§4.6.1](#s4-6-1) do not belong in this row** — they hang off mutate-existing and fail in the opposite direction; see the next row | Decide explicitly "which request paths may carry external calls"; narrow such rules' match down to **the Tasks that genuinely need them**; load-test p95 / p99 and the timeout ratio, and confirm it stays below the webhook timeout — **that ceiling applies to the whole request, not to each rule separately** (default `timeoutSeconds=10`, see checklist item 6 in [§3.1](#s3-1); one 5-second registry round trip fits, two stacked on the same request may not). **Registry / API server jitter turns directly into pipeline creation failures** — have the matching alerts and playbooks ready | | ||
| | **The asynchronous delivery chain (mutate-existing cancellation is fail-open)** | Of the document's four cancellation paths, three are mutate-existing ([§4.2.2](#s4-2-2) / [§4.6.1](#s4-6-1) / [§4.6.2](#s4-6-2)), and they sit **outside the admission verdict**: on a hit, the background-controller patches the target object asynchronously via an UpdateRequest. So when any link in that chain fails — the rule's `context.apiCall` cannot fetch its target, the UpdateRequest never gets created, the background-controller is down or backlogged, update RBAC on the target resource has been revoked — **the original request is allowed as usual and the cancellation patch silently vanishes**: the pipeline runs to the end, with no denial message anywhere in the cluster and no PolicyReport violation record (mutate types produce no violation records — the [§4.2.3](#s4-2-3) warning), only a few error lines in the background-controller log (the 404 note in [§4.6.1](#s4-6-1)). **criterion direction fail-closed ≠ delivery guarantee**: the criterion says "missing/illegal results must cancel just the same", but whether the cancel lands depends on the health of this background chain | **For a hard guarantee of zero races and zero silent failures, only the synchronous paths qualify**: the deny of [§4.2.1](#s4-2-1) or the admission mutate of [§4.2.3](#s4-2-3) (both give a synchronous verdict inside admission). If you stay on mutate-existing, you must monitor this chain as **a delivery system with an SLA**: ① background-controller liveness, restart count, and queue backlog; ② UpdateRequest creation volume and failure/stuck volume (`kubectl get updaterequests -n kyverno`); ③ the gap between policy hit counts and "target objects that really got `spec.status` / `cancel-reason` applied" — **a gap persistently non-zero means cancellations are being lost**; ④ before rollout and after every upgrade, run one controlled fault injection (point the apiCall at a nonexistent object, or temporarily stop the background-controller), confirm the alert fires, and write the result into the change request | |
There was a problem hiding this comment.
Major|异步取消的监控与故障注入方案在目标版本上不可直接执行
本行正确修复了 mutate-existing 的异步 fail-open 边界,但给出的运维方案还有两个可操作性问题:
- 更新后的验收环境(Kyverno 1.15.9-v4.3.2)实测:制造唯一测试策略的
apiCall404 后,原触发请求成功、目标补丁未落地,background-controller 有带策略名的错误;UpdateRequest 则快速经历Pending → Failed、重试后立即删除。故障结束后普通kubectl get updaterequests -n kyverno返回 0 条,只有事先持续 watch 才能捕获。因此“creation volume / failure/stuck volume”不能仅靠这里给出的 get 命令建立,且“策略命中数”也没有说明可用的数据源。 - “temporarily stop the background-controller”是集群级故障,会同时中断该集群其他 mutate-existing / generate 交付;本文又允许在 shared cluster 操作,不能把它作为“每次升级后”的无条件常规选项。
修复目标:
- 把可观测性写成可实施方案:给出持久化指标/日志告警的数据源,或明确要求在故障注入前启动 UpdateRequest watch/事件采集并把短生命周期状态外送;说明一次性
kubectl get不能证明“无失败”。 - 定义“policy hit → target patch landed”差值的两端如何采集、关联(policy/rule、trigger UID、target UID/annotation)及告警窗口,避免留下无法计算的 SLA。
- shared/生产集群只使用唯一测试策略、测试命名空间和无害的不存在对象查询;若保留停 controller 的方式,限定为专用验收集群或维护窗口,并补充原副本状态记录、恢复、Ready 和积压恢复验证。
| |---|---|---| | ||
| | 1 | The 7-item checklist of [§3.1](#s3-1) + the four configs `config-defaults` / `feature-flags` / `hubresolver-config` / the kyverno ConfigMap | Values match the pre-upgrade state; for any mismatch, first locate the affected policies per [§3.6](#s3-6) | | ||
| | 2 | The template allowlist of [§4.1.1](#s4-1-1) | The approved template is allowed; all three of an old version number, an unknown resolver, and a request-level `url` are denied | | ||
| | 3 | The gate parameter contract of [§4.2](#s4-2) (in whichever response shape you actually chose) | Compliant parameters are allowed; all three of switching the gate off / an explicit empty value / **an override field the criterion has never seen** are blocked (the third is the key probe for the denylist shape) | |
There was a problem hiding this comment.
Major|升级回归要求旧 denylist 阻断“从未见过的字段”,验收目标与前文机制相反
前文 §3.6 已准确说明:denylist 只识别已知坏字段;新字段、新 override 入口会自动落到允许侧,升级时必须重新枚举字段面。这里却要求“criterion has never seen”的 override 字段已经被 blocked。两者不能同时成立:如果规则从未见过该入口,它按本文定义就不会阻断;只有先更新 criterion 把它纳入模型,负向探针才可能通过。
这会让升级验收产生错误判定:要么永远无法满足该行,要么测试人员把一个已知旧字段误称为“unknown”后得到虚假的回归通过。
修复目标: 将升级流程拆成两个明确阶段:
- 先以 API schema /
kubectl explain/ 真实对象 diff 发现新增字段,并用探针确认旧策略对它的实际方向;若允许,记录为待修缺口,不能判升级验收通过。 - 判断该字段是否能改变受保护行为;需要治理时先更新 allowlist/criterion,再验证更新后的策略拒绝该新入口,同时保留正常输入允许的正向用例。
- 将本行的通过标准改成“所有新增、且会影响受保护行为的 override 入口在策略更新后均被阻断”,不要要求“未见字段”天然被 denylist 阻断。
| | Triggering action | Affected criteria | Consequence | What the change must include | | ||
| |---|---|---|---| | ||
| | Adding a workload namespace, or migrating pipelines to a new namespace | The `namespaces:` enumeration of **every** policy in [§4](#s4) (demo value `policy-poc`) | **Silent allow**: the new namespace matches no rule | First add the new namespace to every scope (or switch to namespaced `Policy` per [§5](#s5); for "covered by default" switch to a platform-level ClusterPolicy with a negative `exclude` for system namespaces, instead of enumerating one by one), run the positive/negative probes against the new namespace, and only **then** let the business move in | | ||
| | **Adding a workload cluster**, or migrating pipelines to another cluster | **Every policy in the whole document** ([§4](#s4) and [§5](#s5) alike) — `ClusterPolicy` / `Policy` are in-cluster objects and **do not sync across clusters** | **Silent allow**: the new cluster has not a single policy, while the old cluster's reports look perfectly fine — completely invisible from the old cluster | Write "install the minimal set ([§4.0.1](#s4-0-1)) + run the positive/negative probes" into the cluster onboarding process; distribute the policy inventory via GitOps / a platform module rather than installing by hand; periodically diff the `kubectl get clusterpolicy` inventories across clusters (listed as a lossy item in [§7.3](#s7-3)) | |
There was a problem hiding this comment.
Major|多集群漂移检查只比较 ClusterPolicy,漏掉 namespaced Policy 和规则作用域
本行前半段已经明确“每个 ClusterPolicy / Policy 都是集群内对象”,但操作建议却只比较 kubectl get clusterpolicy。这既看不到项目自治使用的 namespaced Policy,也无法判断同名策略在不同集群上的 validate.failureAction、match/exclude、namespaceSelector 是否一致。后文 §7.3 又明确说比较不能停在策略名,至少要比较“name + 每条规则 failureAction + scope”,两处目前自相矛盾。
结果是两个集群即使 kubectl get clusterpolicy 名单相同,也可能一个仍为 Audit、另一个已 Enforce,或某个项目命名空间完全没有对应 Policy;此检查仍会误报基线一致。
修复目标:
- 将多集群基线定义为规范化后的 ClusterPolicy + 所有受管项目 namespaced Policy,而不是一条
kubectl get clusterpolicy名单。 - 比较至少覆盖 policy namespace/name、每条 rule 的
validate.failureAction、match/exclude及 namespace selector;最好以 GitOps 期望态为基准做结构化 diff。 - 新集群 onboarding 与 §7.3 统一引用 §4.0.7 的完整适配和验收流程,避免这里只写“positive/negative probes”而遗漏身份、作用域和 Audit/cancellation 类验收。
yuzichen12123
left a comment
There was a problem hiding this comment.
本轮按“机制说明文档”的定位复核最新提交 398e34f,并结合更新后的验收环境验证 Kyverno 1.15.9-v4.3.2 的 failurePolicy 与 mutate-existing/UpdateRequest 行为。已提交 4 条 Major 行级评论:逐策略 failurePolicy 配置入口、异步取消链的可观测性与安全故障注入、denylist 新字段的升级验收矛盾、多集群策略/作用域漂移检查。此前同步 validate 与异步 mutate-existing 故障语义、Hub 精确坐标 smoke test、片段标注等问题已明显改进;按约定未重复评论暂未上线的 4.14 链接,也未把 Task/Pipeline 合约随版本变化本身视为问题。
… consistency Address the four Major review comments: - failurePolicy/timeoutSeconds: the per-policy entry point is spec.webhookConfiguration (spec.failurePolicy is deprecated in 1.15); every policy asset now declares failurePolicy explicitly, the checklist reads declared intent plus generated webhook grouping, and ModuleInfo is scoped to platform-wide overrides only - async cancellation monitoring: replaced the after-the-fact updaterequests query (deleted after retries, so always empty) with three verified data sources: kyverno_policy_results_total metrics, the background-controller ERR log signature, and a pre-started watch during fault drills; stopping the background-controller is restricted to dedicated clusters or maintenance windows - upgrade regression step 3: two-phase flow (enumerate new override entry points, update the criterion, then verify blocking) instead of expecting a denylist to block fields it has never seen - multi-cluster drift: baseline includes namespaced Policies and compares name + failureAction + match/exclude, aligned with section 7.3
- metrics count attempts, not incidents: drop the exact hits-vs-landings reconciliation claim, keep the error-increase alert, and point per-run attribution back at the target objects - controller-down failures leave neither log lines nor new counters: liveness monitoring is the precondition signal, and the section 6.2 troubleshooting order now starts there - cross-cluster baseline and section 7.3 comparison now include spec.webhookConfiguration and the per-cluster forceFailurePolicyIgnore state; checklist item 6 and onboarding read namespaced Policies and the generated grouping, not the declaration alone - soften 'after-the-fact query is invariably empty' to the retry-window semantics the evidence supports; align the checklist intro with the rewritten item 6
- failurePolicy tiering: state the 17 Fail / 8 Ignore split precisely and explain why the $4.2.2 cancellation trigger stays Fail (same admission gate as $4.2.1); annotate the asset - $6.2.3 evidence script: print the decisive attribution fields first (parent spec.status / cancel-reason, per-TaskRun spec.status / statusMessage), wrap the block in a function so failed reads really stop collection, and fail explicitly on unparseable PolicyReport JSON - mutate-existing RBAC scope: unified to $4.2.2 / $4.6.1 / $4.6.2 across $3.1, $4.0.1, $4.2.2 and the $4.6 preamble ($4.2.3 needs none) - cancellation attribution: downgrade 'not yours = policy' to first suspect; note the API-server audit log as the way to confirm the writer - $4.0.7 conversion: cover the demo-namespace pins outside match (mutate targets, companion Role/RoleBinding, $4.5.3 ConfigMap shape); fix the gate-param-mutate-to-cancel anchor to $4.2.3 - $3.8 step 9: add the same-profile compliant run as a healthy-side control; $3.4.1 bare-entry recipe: require the controller normal path - en-zh parity: drop an en-only count sentence so the editions mirror
- Contracts 5/6 (DAG dominance, finally safety): guarantor corrected to template responsibility only; the shipped 4.1.4 audit checks gate identity, not runAfter/finally -- its resolved-definition snapshot is documented as the hook for building such an audit - Cancellation markers: 'decisive' wording softened to attribution clues throughout; writer confirmation stays with the audit log - PolicyException walkthrough asset: warn that governance metadata (ticket, validity window, owner) must be added for production issuance - Audit-type acceptance row: add the healthy-side criterion (compliant input must yield pass/skip, no fail)
What
Adds a Solution-kind article:
docs/en/solutions/Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.mdIt covers admission-time policy constraints for Tekton pipelines with Kyverno on ACP: template/Task identity locking, gate-parameter effective-value validation, source/target restrictions, result-driven cancellation, scoped exemptions (PolicyException), and rollout/upgrade operations.
Version anchoring
Verified against ACP 4.3 with Alauda DevOps Pipelines v4.14.x, Kyverno v1.15.9-v4.3.2, and Alauda Artifact Hub Shim v1.0.0; the applicability matrix is declared in the document front section,
ProductsVersion: 4.3.xin the frontmatter.How this was verified
Rather than reviewed on paper, the material was executed against a live ACP 4.3 cluster:
CreateRunFailednaming the policy, the approved exemption makes the identical run succeed, the same config in a normal namespace still fails, and deleting the exemption restores the refusal.Known environment dependency worth calling out for reviewers: the Hub coordinates used in §4.2.4/§4.5.1 resolve through the cluster's in-cluster Artifact Hub Shim. On a cluster whose
hubresolver-configstill points at the publicartifacthub.io, those references cannot resolve; the endpoint is set throughTektonConfig.spec.pipeline.hub-resolver-config(the ConfigMap itself is operator-owned).Notes for reviewers
Software_Supply_Chain_Security_of_Alauda_Container_Platform_with_Tekton_and_Kyverno.mdis referenced as a same-directory relative link; it exists in the en tree.idfrontmatter field is left for the add_id.sh flow to assign.scanners=[secret]demonstration in §4.4 and must stay format-valid for trivy to detect it. Push protection was resolved via the used-in-tests allowance.