docs: add RPC forwarder troubleshooting guide

[kejan2514]

Summary

Adds a troubleshooting guide for follower nodes experiencing generic eth_sendRawTransaction forwarding errors.

The guide covers:

• verifying the configured --rpc.forwarder
• testing the upstream RPC directly
• checking systemd environment differences
• checking TLS / proxy behavior
• validating eth_chainId
• enabling focused debug logging
• collecting useful diagnostics before reporting a forwarder issue

This is intended to help operators debug cases similar to #59 where transaction forwarding fails with a generic error sending request message.

Related: #59

[osr21]

Verified the mechanics against the repo and against #59 (where I've been one of the participants in the debugging thread):

• The error string and log lines match #59 verbatim (-32603: error sending request for url, rpc::eth / alloy_rpc_client debug targets — the exact modules that appear in the reporter's logs), so the RUST_LOG recipe in §4 is correctly targeted, not generic advice.
• Unit name arc-execution matches what running-an-arc-node.md actually installs (/etc/systemd/system/arc-execution.service, SyslogIdentifier=arc-execution), so the systemctl commands compose with the existing guide.
• The §5 read-path/write-path split is a genuinely useful distinction that the current docs don't state anywhere: a follower can serve verified reads indefinitely while forwarding is broken.
• No real collision with #192 (README-level Windows/setup notes — different scope and file).

The main gap: the guide's checklist is the part of #59 that already failed to find the bug. The reporter had verified essentially all of §§1–3 and 5 before the thread got anywhere — curl worked from the same host and under the service environment, TLS/DNS/proxy were clean — and the error persisted. What actually moved #59 forward is absent from the guide:

1. The idle-gap repro. The failure correlates with connection-pool staleness, not reachability: reads polling every few seconds → immediate write succeeds; reads then 30–90 s idle → write fails. A troubleshooting doc for this failure class should include that timing test — it's cheap, deterministic in our experience, and it distinguishes "your network is broken" (§§1–3 territory) from "the forwarder client reused a dead pooled connection" (the live hypothesis in #59, with h2/ALPN ruled out as a cause and pool_max_idle_per_host(0) probes as the proposed discriminator).
2. The curl --http1.1 vs default negative control from the #59 thread, which separates protocol-negotiation issues from pooling issues.
3. A pointer to #59 itself in "Related documentation" — an operator who lands on this page with this exact symptom should find the live investigation, not just the generic checklist that its reporter already exhausted.

None of this requires waiting for #59 to resolve — the guide can honestly say "if all checks above pass and the failure correlates with idle time, you are likely hitting the pooled-connection issue tracked in #59" and stop there. With §§1–5 as-is plus an added §6 along those lines, this becomes the doc that would actually have saved #59's reporter several weeks of elimination. Happy to review again after that addition; the bones here are good and the command hygiene (secrets warnings, service-account reproduction) is above average for a troubleshooting doc.

[kejan2514]

Thanks for the detailed review. I’ve added the idle-gap reproduction, the curl --http1.1 comparison, and a direct pointer to #59. The troubleshooting checklist was also updated to capture those results. Happy to adjust further if needed.

[osr21]

Re-verified the revision (c9ea98b) against the #59 thread — all three additions landed accurately:

• §6 idle-gap repro matches the observed behavior exactly: continuous reads → immediate write succeeds; 30–90 s idle → write fails. Importantly, you framed it as evidence gathering ("more consistent with reuse of a stale pooled connection") rather than a settled diagnosis — the right level of confidence while #59 is still open. If the root cause lands (e.g. via a pool_max_idle_per_host change or a client fix), this section will need a one-line update noting which versions are affected, but that's future work, not this PR's.
• The --http1.1 negative control is correctly positioned as distinguishing protocol negotiation from pool staleness, which is exactly the split it served in the investigation — both public forwarders negotiate h2 fine, so a divergence here would point somewhere new rather than at the known hypothesis.
• The #59 pointers (inline in §6 plus Related documentation) mean an operator landing here with this symptom finds the live thread instead of re-running an exhausted checklist — this was the biggest gap and it's fully closed.
• The two new §7 bug-report bullets (idle-gap behavior, curl protocol comparison) are a nice touch: future #59-class reports will arrive with the discriminating evidence already collected instead of the reachability checks that don't discriminate.

With §6 in place, the guide now covers both halves: §§1–5 eliminate the environment, §6 characterizes what's left. This is the doc that would have saved #59's reporter weeks. LGTM.