Skip to content

HDDS-15244. Add secure-cluster /logLevel integration tests - #11024

Open
yandrey321 wants to merge 5 commits into
apache:masterfrom
yandrey321:HDDS-15244
Open

HDDS-15244. Add secure-cluster /logLevel integration tests#11024
yandrey321 wants to merge 5 commits into
apache:masterfrom
yandrey321:HDDS-15244

Conversation

@yandrey321

@yandrey321 yandrey321 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

HDDS-15244 reported that the /logLevel HTTP endpoint is inaccessible in secure
clusters (permanent 403 / SPNEGO replay). Investigation showed the endpoint is
not broken on master:

  • In a secure cluster operators set
    ozone.http.filter.initializers=org.apache.hadoop.security.AuthenticationFilterInitializer
    (see compose/ozonesecure and SecuringOzoneHTTP.md). That initializer registers
    the SPNEGO AuthenticationFilter as authentication.
  • addDefaultServlets() registers /logLevel via addServlet(...), which calls
    addFilterPathMapping("/logLevel", …) — mapping the auth filter onto /logLevel
    exactly once.
  • Result: /logLevel challenges unauthenticated requests (401) and returns 200 to an
    authenticated Ozone admin, gated by hasAdministratorAccess.

This PR:

  1. Adds integration tests locking the (already-correct) behavior in both non-secure
    and secure (SPNEGO/Kerberos + MiniKdc) clusters, covering OM, SCM and DN:
    • TestLogLevelEndpointInsecure — GET/SET /logLevel return 200 and reflect the level change.
    • TestLogLevelEndpointSecure — authenticated admin GET/SET succeed over SPNEGO;
      unauthenticated requests are rejected (401/403). The secure test sets
      ozone.http.filter.initializers=AuthenticationFilterInitializer, mirroring a real
      secure cluster, and restores global login-user / jdk.http.auth.* system state
      around each request.
    • LogLevelEndpointTestUtil — shared helpers for both suites.
  2. Corrects the documentation (Observability.md / Observability.zh.md), which
    stated /logLevel is "not yet supported in secure environment". These endpoints work
    over SPNEGO: run kinit first, the requesting user must be an Ozone administrator, and
    ozone insight negotiates SPNEGO automatically (since HDDS-13883).

No change to ozone insight — SPNEGO client support already exists.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15244

How was this patch tested?

CI: https://github.com/yandrey321/ozone/actions/runs/31817735986/job/94826812059

New integration tests, run locally:

  • TestLogLevelEndpointInsecure — 6/6 pass (GET/SET on OM, SCM, DN).
  • TestLogLevelEndpointSecure — 9/9 pass (admin GET/SET over SPNEGO on OM, SCM, DN;
    unauthenticated requests rejected).
  • Total 15/15, checkstyle:check clean on the touched modules.
  mvn -pl hadoop-ozone/integration-test test \
    -Dtest=TestLogLevelEndpointInsecure,TestLogLevelEndpointSecure

@yandrey321

Copy link
Copy Markdown
Contributor Author

@jojochuang @smengcl please take a look

@jojochuang jojochuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good, didn't realize it's already working

}

static String getLogLevel(String address, String logger) throws Exception {
HttpURLConnection connection =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to make sure the socket connections are closed when end.

@ss77892

ss77892 commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Do we need a test for authenticated but not admin user?

<div class="alert alert-info" role="alert">

Under the hood `ozone insight` uses HTTP endpoints to retrieve the required information (`/conf`, `/prom` and `/logLevel` endpoints). It's not yet supported in secure environment.
Under the hood `ozone insight` uses HTTP endpoints to retrieve the required information (`/conf`, `/prom` and `/logLevel` endpoints). In a secure (Kerberos) cluster these endpoints are protected by SPNEGO: run `kinit` first, and the requesting user must be an Ozone administrator (`ozone.administrators`). `ozone insight` negotiates SPNEGO automatically.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's not exactly right. Admin rights are required only by /logLevel. /conf requires an additional hadoop.security.instrumentation.requires.admin set to true to be that. And /prom accepts any auhenticated user.

<div class="alert alert-info" role="alert">

Under the hood `ozone insight` uses HTTP endpoints to retrieve the required information (`/conf`, `/prom` and `/logLevel` endpoints). It's not yet supported in secure environment.
Under the hood `ozone insight` uses HTTP endpoints to retrieve the required information (`/conf`, `/prom` and `/logLevel` endpoints). In a secure (Kerberos) cluster these endpoints are protected by SPNEGO: run `kinit` first, and the requesting user must be an Ozone administrator (`ozone.administrators`). `ozone insight` negotiates SPNEGO automatically.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like ozone insight config does not support SPNEGO right now. It should use InsightHttpUtils.openConnection just like ozone insight metrics and ozone insight log already do to allow auth with SPNEGO.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we document this behavior for 'ozone insight config' and open a separate JIRA to fix it?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, let's file a JIRA to fix that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants