Skip to content

[Feat] [SDK-399] Add java agent for network telemetry events - #374

Open
buongarzoni wants to merge 35 commits into
masterfrom
feat/SDK-399/add-java-agent-for-network-telemetry-events
Open

[Feat] [SDK-399] Add java agent for network telemetry events#374
buongarzoni wants to merge 35 commits into
masterfrom
feat/SDK-399/add-java-agent-for-network-telemetry-events

Conversation

@buongarzoni

@buongarzoni buongarzoni commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Description of the change

Add Java agent for automatic network telemetry capture

Auto-instruments all major HTTP clients via -javaagent: using ByteBuddy, capturing 4xx/5xx responses as Rollbar telemetry events with no changes at HTTP call sites.

Scope of "no code changes": request code is never touched — no wrappers, no interceptors, no per-call bookkeeping, and nothing to remember when a new HTTP call is added. Setup is a one-time wiring step: the agent JAR on the application classpath, and .telemetryEventTracker(RollbarAgent.getTelemetryTracker()) on the config builder.

That wiring is not automatic by design of the current SDK: ConfigBuilder.build() installs its default RollbarTelemetryEventTracker whenever telemetryEventTracker(...) was not called, and there is no global registry or ServiceLoader hook an agent could claim instead. Making the agent self-installing would require a change to rollbar-java and is tracked separately.

Caution

This module targets JVM-based applications only. Android is not supported — ART does not implement
the java.lang.instrument API required by Java agents. Android users should use the existing
rollbar-android module instead.

The acceptance criteria on the Shortcut story need the same narrowing — "zero application code changes" → "no changes at HTTP call sites; one-time tracker wiring at init".

What's included:

  • New rollbar-java-agent module — shadow JAR with ByteBuddy bundled and relocated
  • Instruments HttpURLConnection, java.net.http.HttpClient, Apache HC 4.x and 5.x
  • URL sanitization (strips credentials, query params, fragment before recording)
  • Deduplication via WeakHashMap to handle re-entrant getResponseCode() calls and dual-advice firing on HttpClient
  • Integration tests using WireMock 3.x for each instrumented client
  • README with installation and manual testing guide

Usage:

  -javaagent:/path/to/rollbar-java-agent.jar
  Rollbar.init(withAccessToken("...")
      .telemetryEventTracker(RollbarAgent.getTelemetryTracker())
      .build());

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Maintenance
  • New release

Related issues

Shortcut stories and GitHub issues (delete irrelevant)

Checklists

Development

  • Lint rules pass locally
  • The code changed/added as part of this pull request has been covered with tests
  • All tests related to the changed code pass in development

Code review

  • This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached
  • "Ready for review" label attached to the PR and reviewers assigned
  • Issue from task tracker has a link to this pull request
  • Changes have been reviewed by at least one other engineer

@linear-code

linear-code Bot commented May 26, 2026

Copy link
Copy Markdown

SDK-399

@buongarzoni

Copy link
Copy Markdown
Collaborator Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Code review skipped — your organization has reached its monthly code review spending cap.

An organization admin can view or raise the cap at claude.ai/admin-settings/claude-code. The cap resets at the start of the next billing period.

Once the cap resets or is raised, comment @claude review on this pull request to trigger a review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ad0b2acf09

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@buongarzoni

Copy link
Copy Markdown
Collaborator Author

@claude review

Comment thread rollbar-java-agent/src/main/java/com/rollbar/agent/AgentTelemetryStore.java Outdated
@brianr
brianr self-requested a review June 15, 2026 22:09
@buongarzoni buongarzoni added this to the v2.4.0 milestone Jun 15, 2026

@brianr brianr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posting review findings from the local review.

Comment thread rollbar-java-agent/build.gradle.kts Outdated
.type(ElementMatchers.named("java.net.HttpURLConnection"))
.transform((b, typeDescription, classLoader, module, protectionDomain) ->
b.visit(Advice.to(GetResponseCodeAdvice.class)
.on(ElementMatchers.named("getResponseCode")))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Capture HttpURLConnection requests that skip getResponseCode

For HttpURLConnection callers that trigger the request with getInputStream() or getErrorStream() and never call getResponseCode(), this is the only advised method, so a 4xx/5xx response (or the IOException thrown by getInputStream() on 4xx) is never recorded. This leaves a common HttpURLConnection usage path outside the promised automatic network-error capture.

Comment thread rollbar-java-agent/src/main/java/com/rollbar/agent/UrlSanitizer.java Outdated

@brianr brianr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on the current head. The inline comments cover four release blockers. Two existing review threads also still apply, so I have not duplicated them: the agent still records Rollbar's own SyncSender failures, and NetworkEventBridge.composeUrl() still treats :// inside a relative URI's query/path as an absolute URI. The former thread is marked resolved even though no suppression guard is present at this head.

// from the authority manually.
String authority = uri.getAuthority();
if (authority != null) {
int at = authority.indexOf('@');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve encoded credentials while locating the userinfo delimiter

URI.getAuthority() returns the decoded authority. For https://user:p%40ss@example.com/path, it yields user:p@ss@example.com; indexOf('@') then keeps ss@example.com, so part of the password is recorded as new userinfo. Please operate on getRawAuthority() / getRawUserInfo() (or otherwise locate the delimiter in the raw authority) before reconstructing the sanitized URL, and add a regression test with percent-encoded @ in userinfo.

Oracle documents the decoding behavior here: https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/net/URI.html#getAuthority()

Comment thread rollbar-java-agent/build.gradle.kts Outdated
}

dependencies {
implementation("net.bytebuddy:byte-buddy:1.14.18")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Support the advertised JVM range

Byte Buddy 1.14.18 supports class files only through Java 23 without experimental mode; Java 24 support starts at 1.15.4 and Java 25+ at 1.17.0. The README promises Java 11 or higher, but CI exercises only 11 and 17, so the agent can silently fail to transform HTTP classes on current Java 24/25/26 runtimes. Please upgrade Byte Buddy and add current-LTS/current-JDK coverage, or explicitly narrow the supported range.

Compatibility table: https://github.com/raphw/byte-buddy#java-version-compatibility

implementation("net.bytebuddy:byte-buddy:1.14.18")
implementation("net.bytebuddy:byte-buddy-agent:1.14.18")
api(project(":rollbar-api"))
implementation(project(":rollbar-java"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep Rollbar SDK classes out of the shaded agent

shadowJar merges runtimeClasspath by default, so this implementation dependency and the rollbar-api api dependency are embedded unrelocated in the fat JAR (along with transitive dependencies such as SLF4J); only Byte Buddy is relocated below. The README also puts the agent alongside the application's Rollbar SDK, creating duplicate com.rollbar.* classes that can pin/override another SDK version or split class identity in containers. Please shade only agent-private dependencies (for example via a dedicated Byte Buddy configuration) and leave the Rollbar API/SDK as ordinary external dependencies.

Shadow's default dependency behavior: https://gradleup.com/shadow/configuration/dependencies/

Comment thread rollbar-java-agent/README.md Outdated
@@ -0,0 +1,202 @@
# Rollbar Java Agent

A zero-code-change Java instrumentation agent that automatically captures HTTP network errors (4xx and 5xx responses) as Rollbar telemetry events.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Align the implementation with the zero-code-change claim

The instrumentation removes changes at HTTP call sites, but installation still requires adding the agent as an application dependency and changing Rollbar.init(...) to install RollbarAgent.getTelemetryTracker() (steps 3–4). The behavior table also says that without this wiring events accumulate but are not sent. If SDK-399 requires genuinely zero application-code changes, the agent needs to integrate its tracker automatically; otherwise this claim should be narrowed to "no HTTP client call-site changes" and the stated problem/acceptance criteria updated.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the two nit-level findings from this run's inline comments, I also checked a few other candidate issues and ruled them out: instrumenting the Rollbar SDK's own SyncSender does not create a feedback loop, the repeated classloader/bridge-lookup boilerplate across the four HttpUrlConnectionInstrumentation advice classes is a style-only duplication (not a bug), the README's "zero-code-change" wording is accurate for the HTTP call-site instrumentation itself, and the HC4/HC5 hierarchy-walk name filter excluding only JDK prefixes is a performance micro-optimization rather than a correctness gap.

Extended reasoning...

This run's findings are both nits (a redirect-misattribution edge case in the JDK HttpClient advice, and an unnecessary-reflection quality nit in HttpUrlConnectionInstrumentation), and the PR has already absorbed a long sequence of substantive P1/P2 fixes from prior review rounds (recursive getResponseCode() guard, doExecute()-based Apache HC4/HC5 unification covering the HttpHost overloads, UrlSanitizer host-corruption and encoded-credential fixes, composeUrl scheme-position bounding, shading only Byte Buddy instead of the Rollbar SDK, Byte Buddy version bump for current JDKs, and preventing the thin jar from clobbering the shaded one). Given this module's complexity (bytecode instrumentation across four HTTP clients) and its security-sensitive URL-sanitization logic, I'm not approving outright, but wanted to record the additional items examined and ruled out this run so they aren't re-explored from scratch.

Comment on lines +129 to +138
if (response != null) {
int statusCode = (Integer) response.getClass().getMethod("statusCode").invoke(response);
if (statusCode >= 400) {
Object uri = request.getClass().getMethod("uri").invoke(request);
String method = (String) request.getClass().getMethod("method").invoke(request);
// response object is the dedup key — unique per send() call, shared between
// HttpClientFacade and HttpClientImpl so only one event is recorded
bridge.getMethod("recordNetworkEvent",
Object.class, String.class, String.class, String.class)
.invoke(null, response, method, uri.toString(), String.valueOf(statusCode));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The JDK HttpClient advice (SendAdvice.onExit sync path and NetworkEventBridge.createAsyncCallback async path) reads the URL/method off the original pre-redirect request argument instead of the response, so when the client is configured with a redirect policy other than the default NEVER (e.g. Redirect.NORMAL), a final-hop 4xx/5xx is recorded with the original request's host/method rather than the actual failing one — unlike the other three instrumented clients, whose instrumentation point already sees the final target. Fix by using response.uri() and response.request().method() instead of the request argument in both SendAdvice.onExit (JavaHttpClientInstrumentation.java:132-138) and the HttpResponse branch of createAsyncCallback (NetworkEventBridge.java).

Extended reasoning...

What's wrong: SendAdvice.onExit (JavaHttpClientInstrumentation.java:129-138) records telemetry using request.getClass().getMethod("uri").invoke(request) and request.getClass().getMethod("method").invoke(request), where request is @Advice.Argument(0) — the original HttpRequest object passed into HttpClient.send(...), not anything derived from the returned response. The async path (SendAsyncAdviceNetworkEventBridge.createAsyncCallback) has the identical pattern: the callback closes over the original request object captured at sendAsync() time and reads uri()/method() off of it in the response.statusCode() >= 400 branch.

Why this is wrong: java.net.http.HttpClient, when configured with a redirect policy other than the default Redirect.NEVER (e.g. HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NORMAL)), follows the entire redirect chain internally inside one send()/sendAsync() call. The javadoc for HttpResponse.uri() says the returned URI "may be different from the request URI if redirection occurred," and HttpResponse.request() returns the actual final HttpRequest (whose method can also change — a 303 converts POST to GET). So when the chain ends in a 4xx/5xx, the response correctly reflects the final hop, but the code reads the pre-redirect request's URI/method instead.

Why nothing else in the code catches this: there is no logic anywhere in SendAdvice/createAsyncCallback that inspects redirect history or consults response.request() — the response is used only for statusCode() and as the WeakHashMap dedup key, never for its own uri()/request().

Step-by-step proof:

  1. App code: HttpClient client = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NORMAL).build();
  2. client.send(HttpRequest.newBuilder(URI.create("https://api.example.com/widgets")).build(), ...).
  3. api.example.com responds 301 → Location: https://cdn.example.com/widgets.
  4. The JDK client follows the redirect internally (still inside the single send() call) and issues the request to cdn.example.com.
  5. cdn.example.com responds 500.
  6. send() returns the final HttpResponse, whose uri() is https://cdn.example.com/widgets and whose request() is the request that was actually sent to cdn.example.com.
  7. SendAdvice.onExit fires with @Advice.Argument(0) request still bound to the original request object built in step 2 (URI https://api.example.com/widgets), and reads request.uri()/request.method() from it.
  8. Recorded telemetry: {method: <original>, url: "https://api.example.com/widgets", status_code: "500"} — attributing the 500 to api.example.com, when the failing dependency was actually cdn.example.com.

Impact: this defeats the purpose of the URL field, which exists specifically to let a developer identify which downstream host actually failed. A developer investigating the error would look at (and possibly page/alert on) the wrong service.

Fix: in SendAdvice.onExit, replace request.uri()/request.method() with response.uri() and response.request().method() (both available via reflection on the HttpResponse/HttpRequest interfaces, consistent with how the rest of the advice already does reflective lookups). The same substitution applies to the HttpResponse branch inside NetworkEventBridge.createAsyncCallback.

Severity: nit, not normal — it requires a non-default, opt-in redirect policy (the JDK's own default is Redirect.NEVER, under which a 3xx is simply not recorded at all) combined with a redirect chain that terminates in an error. The failure mode is misattributed/degraded telemetry, not a crash, exception, or data loss, and the misattributed URL is still a URL the app genuinely requested (just the wrong hop). This is consistent with how the similarly-scoped composeUrl nested-URL host-misattribution finding elsewhere in this PR was rated.

Comment on lines +171 to +178
if (statusCode >= 400) {
Object url = connection.getClass().getMethod("getURL").invoke(connection);
String urlStr = url != null ? url.toString() : "";
String method = (String) connection.getClass()
.getMethod("getRequestMethod").invoke(connection);
bridge.getMethod("recordNetworkEvent",
Object.class, String.class, String.class, String.class)
.invoke(null, connection, method, urlStr, String.valueOf(statusCode));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 GetResponseCodeAdvice.onExit() (and the equivalent code in GetInputStreamAdvice/GetErrorStreamAdvice) reflects into getURL()/getRequestMethod()/getResponseCode() via connection.getClass().getMethod(...).invoke(...), but these are public methods declared directly on the bootstrap-loaded java.net.HttpURLConnection/URLConnection, visible from every classloader — unlike the genuine NetworkEventBridge lookup in the same methods, which does need reflection to cross into the app classloader. Typing @Advice.This as HttpURLConnection and calling the methods directly removes six reflective lookups plus their checked-exception handling from a per-error-response path with identical behavior, since ByteBuddy inlines the advice regardless of the declared parameter type.

Extended reasoning...

What the finding is

GetResponseCodeAdvice.onExit (lines 171-178) reflectively invokes getURL() and getRequestMethod() on the HttpURLConnection instance via connection.getClass().getMethod(name).invoke(connection). The same pattern recurs for getResponseCode() in GetInputStreamAdvice.onExit (line 94) and GetErrorStreamAdvice.onExit (line 125).

All three methods — getResponseCode() and getRequestMethod() on java.net.HttpURLConnection, getURL() on its superclass java.net.URLConnection — are public and declared on bootstrap-loaded java.base classes. There is no classloader gap to bridge: every type this advice is inlined into (HttpURLConnection itself, or a concrete subclass like sun.net.www.protocol.http.HttpURLConnection) is a subtype of the bootstrap class HttpURLConnection, so a direct invokevirtual reference resolves from any classloader. This is fundamentally different from the NetworkEventBridge lookup a few lines below in the same methods, which genuinely must go through Thread.currentThread().getContextClassLoader().loadClass(...) because NetworkEventBridge lives in the application classloader and is invisible from a bootstrap-inlined advice body.

Why the change is safe

Typing @Advice.This as HttpURLConnection instead of Object is valid for every instrumented site: GetResponseCodeAdvice is inlined directly into java.net.HttpURLConnection.getResponseCode() (exact match), and GetInputStreamAdvice/GetErrorStreamAdvice target concrete subtypes, for which HttpURLConnection is always an assignable supertype. ByteBuddy inlines advice bytecode into the target method regardless of the advice parameter's declared type, so this is a purely mechanical substitution — connection.getResponseCode()/getURL()/getRequestMethod() called directly instead of via Method.invoke. getURL() and getRequestMethod() declare no checked exceptions, and getResponseCode()'s IOException is already caught by the surrounding catch (Throwable ignored), so no new exception handling is needed at the call site.

Step-by-step proof (GetResponseCodeAdvice)

  1. A 404 response triggers getResponseCode() to return, statusCode >= 400.
  2. Current code: connection.getClass().getMethod("getURL").invoke(connection) — a reflective lookup + invoke against a bootstrap class the calling code could reference directly.
  3. Replacement: connection.getURL() — same bytecode-inlined call site, same return value, no reflection, no NoSuchMethodException/IllegalAccessException/InvocationTargetException handling needed.
  4. Identical for getRequestMethod(), and for getResponseCode() in the two other advice classes.

Impact

Six reflective getMethod()+invoke() calls are removed from the response/error-handling path (fired on every 4xx/5xx HttpURLConnection response), improving clarity and per-response cost with no behavior change. It is also strictly safer under JPMS than reflecting on an internal JDK implementation class. This is a pure code-quality/efficiency cleanup — nothing about program correctness changes — so it does not block merge.

Comment on lines +85 to +91
@Advice.OnMethodExit(onThrowable = Throwable.class)
public static void onExit(
@Advice.Argument(0) HttpHost target,
@Advice.Argument(1) HttpRequest request,
@Advice.Return HttpResponse response,
@Advice.Thrown Throwable thrown
) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 DoExecuteAdvice types its @Advice.Argument/@Advice.Return parameters as concrete Apache HttpClient types (HttpHost/HttpRequest/HttpResponse in HC4, the hc5 equivalents in HC5) instead of Object + reflection, unlike HttpUrlConnectionInstrumentation and JavaHttpClientInstrumentation which deliberately avoid this. Because Advice.to(DoExecuteAdvice.class) resolves these parameter types via getDeclaredMethods() in the agent's own classloader, whenever Apache HttpClient is loaded by a classloader the agent can't see (Spring Boot fat jars, WAR containers, OSGi), this throws NoClassDefFoundError inside the transform lambda and the advice is silently never woven in — Apache HttpClient telemetry is never recorded in these deployments, with no crash and only a capped stderr line as a symptom.

Extended reasoning...

What's wrong: DoExecuteAdvice in both ApacheHttpClient4Instrumentation.java (lines 85-91) and ApacheHttpClient5Instrumentation.java declares its @Advice.OnMethodExit method with parameters typed directly as library classes — HttpHost, HttpRequest/ClassicHttpRequest, HttpResponse/ClassicHttpResponse. This is a deliberate divergence from the pattern used everywhere else in this PR: HttpUrlConnectionInstrumentation types @Advice.This as Object and JavaHttpClientInstrumentation types @Advice.Argument(0) as Object, precisely so the advice class itself never needs to resolve a type that might not be visible from its own classloader.

The mechanism: Advice.to(DoExecuteAdvice.class) is called inside the AgentBuilder.Transformer lambda registered in ApacheHttpClient4Instrumentation.installIfAvailable/ApacheHttpClient5Instrumentation.installIfAvailable. That lambda runs in the classloader that loaded ApacheHttpClient4Instrumentation itself — the agent's own defining classloader, which under -javaagent is typically the system classloader. To locate the @Advice.OnMethodExit method on DoExecuteAdvice, ByteBuddy reflects over its declared methods via TypeDescription.ForLoadedType.of(advice).getDeclaredMethods(), which for a loaded type delegates to the JVM's own Class.getDeclaredMethods(). That JVM call eagerly resolves every declared method's parameter and return types — this is standard JVM behavior, and multiple verifiers reproduced it directly: a class whose method references a type absent from its own defining classloader loads fine, but getDeclaredMethods() on it throws NoClassDefFoundError, even in an isolated ByteBuddy Advice.to(...) reproduction using the exact same call ByteBuddy makes.

Why it's reachable in production but invisible in tests: the agent's build.gradle.kts marks httpclient/httpclient5 as compileOnly, and shadowJar embeds only the shaded configuration (Byte Buddy), so org.apache.http.* is never inside the agent jar. When HC4/HC5 is loaded by the same classloader as the agent (a flat classpath, which is exactly what Gradle test JVMs use), getDeclaredMethods() succeeds and every WireMock test in this PR passes. But when HC4/HC5 is loaded by a child classloader the agent's classloader can't see — Spring Boot executable jars via LaunchedURLClassLoader, per-WAR servlet container classloaders, or OSGi bundles, all extremely common deployment topologies — the same call throws NoClassDefFoundError: org/apache/http/HttpHost inside the transform lambda. AgentBuilder catches per-type transform exceptions and routes them to RollbarAgent.ErrorReportingListener (capped at 10 stderr lines), so nothing crashes — but no advice is ever woven into that CloseableHttpClient subtype, and Apache HttpClient telemetry is silently never recorded for the lifetime of that classloader.

Step-by-step proof:

  1. App is packaged as a Spring Boot fat jar. -javaagent:rollbar-java-agent.jar puts the agent (and ApacheHttpClient4Instrumentation/DoExecuteAdvice) on the system classloader; org.apache.http.* lives only in BOOT-INF/lib, loaded by the child LaunchedURLClassLoader.
  2. The app constructs a CloseableHttpClient subtype, triggering class loading.
  3. ByteBuddy's registered matcher (hasSuperType(named("...CloseableHttpClient"))) matches, and the .transform(...) lambda fires — running in the agent's classloader context, not the app's.
  4. Inside that lambda, Advice.to(DoExecuteAdvice.class) calls getDeclaredMethods() on DoExecuteAdvice, which must resolve HttpHost/HttpRequest/HttpResponse — but the system classloader cannot see those types (they're only in the child classloader).
  5. NoClassDefFoundError is thrown, caught by AgentBuilder, routed to ErrorReportingListener (a few lines on stderr).
  6. The transform returns unmodified; CloseableHttpClient.doExecute is never instrumented for that classloader.
  7. Every subsequent Apache HttpClient 4xx/5xx response in that app produces zero telemetry, with no further indication anything is wrong.

Why existing code doesn't prevent this: the comment at ApacheHttpClient4Instrumentation.java:64-67 ("Apache HC 4.x runs in the application classloader, so we can reference Rollbar classes directly") only justifies the NetworkEventBridge reference inside the advice body, which does link fine because the child app classloader delegates parent-first to the system classloader holding NetworkEventBridge. It says nothing about the advice method's own parameter types, which are resolved eagerly at weave time in the agent's classloader, not the target's — a completely different resolution point that the comment doesn't address.

Fix: type the advice parameters as Object (matching HttpUrlConnectionInstrumentation/JavaHttpClientInstrumentation) and access HttpHost/HttpRequest/HttpResponse members via reflection from inside the advice body, which executes in the target class's classloader context and therefore can see those types safely. Same fix applies identically to ApacheHttpClient5Instrumentation.

Comment on lines +40 to +49
private static void installInstrumentation(Instrumentation inst) {
// Override ByteBuddy's default which ignores all java.* and javax.* classes,
// so we can instrument JDK HTTP clients (HttpURLConnection, HttpClient).
// We still ignore ByteBuddy's own classes to avoid instrumentation loops.
AgentBuilder builder = new AgentBuilder.Default()
.ignore(ElementMatchers.nameStartsWith("net.bytebuddy.")
.or(ElementMatchers.nameStartsWith("com.rollbar.agent.shaded.")))
.with(new ErrorReportingListener())
.with(AgentBuilder.InitializationStrategy.NoOp.INSTANCE)
.with(AgentBuilder.TypeStrategy.Default.REDEFINE);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 installInstrumentation() (RollbarAgent.java:40-49) never sets .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION), so ByteBuddy stays at the default RedefinitionStrategy.DISABLED and only transforms classes loaded after installOn(inst) runs — it never retransforms classes already loaded. This silently breaks the dynamic-attach path exposed via agentmain (and the manifest's Can-Redefine-Classes/Can-Retransform-Classes: true), since HTTP client classes are almost always already loaded in a running JVM by the time attach happens, so none of the four instrumentations install and zero telemetry is recorded with no error raised.

Extended reasoning...

What the bug is: RollbarAgent.installInstrumentation() builds its AgentBuilder (lines 40-49) with an ignore filter, the ErrorReportingListener, InitializationStrategy.NoOp, and TypeStrategy.Default.REDEFINE — but never calls .with(AgentBuilder.RedefinitionStrategy...). ByteBuddy's AgentBuilder.Default defaults RedefinitionStrategy to DISABLED. Under DISABLED, installOn(inst) registers the ClassFileTransformer with inst.addTransformer(transformer, /* canRetransform */ false) — the transformer only sees types as they are freshly loaded from that point forward. It never iterates over and retransforms classes the JVM already has loaded at the moment installOn runs. Note TypeStrategy.Default.REDEFINE (which is set) is a different, independent setting — it controls how a matched type's bytecode is rewritten, not whether already-loaded types get revisited — so it does not compensate for the missing RedefinitionStrategy.\n\nThe code path that triggers it: RollbarAgent exposes a public agentmain(String, Instrumentation) (lines 36-38) specifically for dynamic attach to an already-running JVM via VirtualMachine.loadAgent(...), and build.gradle.kts's manifest sets both Agent-Class and Can-Redefine-Classes/Can-Retransform-Classes: true — JVM-level permissions that exist for exactly this scenario. But in a real running application at the moment of attach, java.net.HttpURLConnection, java.net.http.HttpClient, and any already-used Apache HttpClient classes are, in virtually every case, already loaded. With RedefinitionStrategy left at DISABLED, none of the four installIfAvailable/install calls in installInstrumentation can retransform those already-loaded classes, so no advice is ever woven into them.\n\nWhy nothing else catches it: ErrorReportingListener.onError is the only failure-surfacing mechanism in this code, but it only fires when a transform attempt is made and fails to apply — here, no transform is even attempted on the already-loaded classes, so onError never fires. The agent's agentmain returns normally, giving every outward signal of successful attachment while silently receives zero events for the rest of the process's life.\n\nStep-by-step proof:\n1. A long-running application is already executing, having triggered classloading of sun.net.www.protocol.http.HttpURLConnection (or 's impl classes) well before any Rollbar tooling attaches.\n2. An operator (or tooling) dynamically attaches this agent via VirtualMachine.attach(pid).loadAgent(jarPath), invoking RollbarAgent.agentmain(args, inst).\n3. installInstrumentation(inst) builds the AgentBuilder and calls HttpUrlConnectionInstrumentation.install(builder, inst) (and the other three installIfAvailable calls), each ending in .installOn(inst).\n4. Because RedefinitionStrategy was never set, installOn calls inst.addTransformer(transformer, false)false meaning "do not retransform currently loaded classes."\n5. The already-loaded HttpURLConnection/HttpClient/Apache HC classes are never revisited by the JVM; no transform is attempted, so ErrorReportingListener.onError never fires.\n6. The application continues making HTTP calls through those already-loaded, un-instrumented classes. Every 4xx/5xx response goes completely unrecorded — AgentTelemetryStore stays empty for the process's entire remaining lifetime, with no log line or exception anywhere indicating the failure.\n\nHow to fix: add .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION) to the builder in installInstrumentation(). The manifest already grants the required Can-Retransform-Classes: true JVM permission, so this is a pure one-line fix with no other changes needed; advice inlining here only rewrites method bodies, which retransformation supports without any schema/field changes.\n\nSeverity: the documented and tested path is -javaagent:/premain, where HTTP classes are loaded lazily after premain installs the transformer — this is proven by the passing WireMock integration test suite. Dynamic attach via agentmain is not documented in the README, even though the code (agentmain) and manifest (Agent-Class, Can-Redefine-Classes/Can-Retransform-Classes) both advertise it as a supported entry point. Since merging without this fix does not break the primary, documented feature, this should not block the PR — but it is a genuine, reproducible defect on an entry point the module explicitly exposes and grants JVM permissions for, and is worth a one-line fix (or removing the agentmain/Agent-Class surface if dynamic attach isn't actually meant to be supported yet).

Comment on lines +100 to +117
}

if (response != null && request != null) {
int statusCode = response.getStatusLine().getStatusCode();
if (statusCode >= 400) {
// The host-based overloads carry the target separately from a request whose URI may be
// just a path, so rejoin the two rather than reading the request URI alone.
String base = target != null ? target.toURI() : null;
String requestUri = request.getRequestLine() != null
? request.getRequestLine().getUri() : null;
NetworkEventBridge.recordNetworkEvent(
response,
request.getRequestLine().getMethod(),
NetworkEventBridge.composeUrl(base, requestUri),
String.valueOf(statusCode)
);
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The doExecute(HttpHost, request, context) advice reads its recorded host/URI off the original @Advice.Argument(0)/(1) parameters, but Apache HttpClient (both HC4 and HC5) follows redirects internally via RedirectExec, which reassigns only its own local currentRequest/currentRoute and never mutates the objects doExecute is holding. So when a request to host A 3xx-redirects to host B and host B returns a 4xx/5xx, the recorded telemetry pairs the final status code with host A instead of the failing host B. This is a pre-existing design limitation, unrelated to any change introduced by this PR — it stems from choosing doExecute() as the single instrumentation point, which is otherwise the right choice for overload coverage.

Extended reasoning...

What the bug is. DoExecuteAdvice.onExit in ApacheHttpClient4Instrumentation.java (lines 100-117) binds @Advice.Argument(0) target and @Advice.Argument(1) request — the exact parameters doExecute(HttpHost, HttpRequest, HttpContext) was originally invoked with — and uses them, together with @Advice.Return response, to build the recorded URL via NetworkEventBridge.composeUrl(target.toURI(), request.getRequestLine().getUri()). ApacheHttpClient5Instrumentation.java's DoExecuteAdvice has the identical shape, reading request.getUri()/request.getMethod() off its own doExecute argument.

Why this is wrong. InternalHttpClient.doExecute() (verified via javap on httpclient-4.5.14) wraps the incoming request into a fresh HttpRequestWrapper and hands it to the exec chain (execChain.execute(route, wrapper, context)); it never reassigns its own target/request locals afterward. RedirectExec, the outermost element of that chain, is what actually follows a redirect: on each hop it reassigns only its own local currentRequest/currentRoute (via HttpRequestWrapper.wrap(redirect)), then returns the final response from its loop. It never touches the target/request objects that doExecute itself is holding. So for a redirect chain from host A to host B ending in a 4xx/5xx, doExecute() returns the true final (host B) response, but its own target/request parameters — the ones the advice reads — still describe host A. HC5's InternalHttpClient/RedirectExec follow the same wrap-and-reassign-local pattern, so ApacheHttpClient5Instrumentation has the same issue.

Reachability. This is on the default path: HttpClients.createDefault() enables DefaultRedirectStrategy, so HC4/HC5 follow redirects out of the box for GET/HEAD (and 307/308 for any method). Any followed redirect that ends in an error — cross-host, http→https, apex→www, a CDN hop — records the pre-redirect host/URL paired with the final status code.

Why nothing else in the code catches it. The advice never consults HttpClientContext (available as @Advice.Argument(2), and updated by RedirectExec to the final HTTP_TARGET_HOST/HTTP_REQUEST attributes), and ClassicHttpResponse/HttpResponse carry no back-reference to the request actually sent — so there is no way to recover the final target from the response alone, unlike java.net.http.HttpResponse, which does expose response.request().

Step-by-step proof.

  1. client.execute(new HttpGet("http://a.example.com/widgets")) where a.example.com 301-redirects to https://b.example.com/widgets.
  2. doExecute(target=a.example.com, request=GET /widgets, context) is invoked; target/request are bound as this method's own arguments.
  3. RedirectExec.execute() follows the 301 by reassigning its own local currentRequest/currentRoute to b.example.com, issues the redirected request, and gets back a 500.
  4. doExecute() returns that 500 HttpResponse — but its target/request parameters are unchanged, still a.example.com/GET /widgets.
  5. DoExecuteAdvice.onExit fires with target=a.example.com, request=GET /widgets, response=500, records {method: GET, url: "http://a.example.com/widgets", status_code: "500"}.
  6. The telemetry attributes the failure to a.example.com, when the request that actually failed went to b.example.com.

Impact and fix. This degrades (misattributes) the recorded URL/host for a redirected error — the status code is still correct, and no crash/data loss occurs, so it is a nit rather than a blocking issue. A fix would read the final target/request from HttpClientContext's HTTP_TARGET_HOST/HTTP_REQUEST attributes (updated by the exec chain on each redirect hop) instead of from doExecute's own arguments. This is distinct from the already-reported JavaHttpClientInstrumentation redirect finding (comment 2026-08-10T21:29:36Z): that finding's aside that HC4/HC5 "already see the final target" is incorrect, and its suggested fix (response.request()) does not apply here since Apache's ClassicHttpResponse has no such back-reference.

Comment on lines +160 to +168
if (thrown != null) {
Boolean recorded = (Boolean) bridge
.getMethod("markAsRecorded", Object.class).invoke(null, thrown);
if (recorded) {
String msg = thrown.getMessage() != null
? thrown.getMessage() : thrown.getClass().getName();
bridge.getMethod("recordError", String.class).invoke(null, msg);
}
return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: the reflective markAsRecorded/recordError block is copy-pasted verbatim in three advice sites — HttpUrlConnectionInstrumentation.GetResponseCodeAdvice.onExit (lines 160-168), JavaHttpClientInstrumentation.SendAdvice.onExit, and SendAsyncAdvice.onExit — each doing two separate reflective Method lookups plus the same null-message fallback. A single NetworkEventBridge.recordThrowable(Throwable) helper would collapse each site to one reflective invoke, with no behavior change.

Extended reasoning...

What's duplicated: the same 5-line pattern — reflectively call markAsRecorded(thrown), and if it returns true, build a null-safe message (thrown.getMessage() != null ? thrown.getMessage() : thrown.getClass().getName()) and reflectively call recordError(message) — is copy-pasted verbatim across three advice classes: HttpUrlConnectionInstrumentation.GetResponseCodeAdvice.onExit (lines 160-168), JavaHttpClientInstrumentation.SendAdvice.onExit (~118-126), and JavaHttpClientInstrumentation.SendAsyncAdvice.onExit (~65-73). Each site performs two independent reflective getMethod()+invoke() calls (one for markAsRecorded, one for recordError) plus the identical message-fallback branch.\n\nThe same shape also shows up, non-reflectively, in the two Apache HC4/HC5 DoExecuteAdvice classes and in NetworkEventBridge.createAsyncCallback's thrown-exception branch — but the three reflective copies alone are enough to justify a shared helper.\n\nWhy this happened: each advice class is inlined into a different bootstrap/JDK class (HttpURLConnection, HttpClient's send/sendAsync) and has to cross the classloader gap to reach NetworkEventBridge, which lives in the application classloader. Because the crossing itself requires reflection, it was natural to write the whole mark+record sequence reflectively at each call site rather than factoring it into the bridge — but the branch and string logic don't need to be reflective at all; only the single entry-point call into does.\n\nThe fix: add one method to NetworkEventBridge — e.g. public static void recordThrowable(Throwable thrown) — that does the markAsRecorded check, builds the null-safe message, and calls recordError internally (all in the app classloader, no reflection needed for that part). Each of the three advice sites then reduces from five duplicated lines plus two reflective Method objects down to a single reflective invoke:\n\njava\nbridge.getMethod("recordThrowable", Throwable.class).invoke(null, thrown);\n\n\nStep-by-step proof this is safe and behavior-preserving:\n1. Today, GetResponseCodeAdvice.onExit does: look up markAsRecorded, invoke it with thrown → if true, look up recordError, build the message, invoke it.\n2. With the helper, NetworkEventBridge.recordThrowable(thrown) runs the identical two-step check but as plain Java inside the bridge class (which already has direct access to markAsRecorded/recordError since they're static methods on the same class).\n3. The advice site now looks up and invokes only recordThrowable, passing thrown through unchanged.\n4. Since recordThrowable's internal logic is byte-for-byte the same branch/message-building code that used to live at the call site, the recorded telemetry (message content, dedup behavior via markAsRecorded) is identical — only the amount of reflection and duplicated code changes.\n\nImpact: this is a pure code-quality/reuse cleanup — three (arguably five, counting the HC4/HC5 direct-call sites) copies of the same logic collapse to one, each reflective call site shrinks from ~8 lines and two Method lookups to one, and there's no behavior change. It doesn't block merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants