google.auth.exceptions.TransportError is not retried despite HttpRetryOptions

[emileferreira]

Is this a client library issue or a product issue?

This is a google-genai client library issue. A transient Google Auth transport
failure bypasses the retry policy even when HttpRetryOptions is explicitly
configured.

Environment

• Python: 3.12
• Reproduced with google-genai==2.12.1 and google-auth==2.55.2
• Confirmed against google-genai==2.18.1, google-auth==2.56.0, and current main
• Vertex AI with service-account Application Default Credentials

Description

When Vertex credentials need refreshing, google-genai calls
credentials.refresh() before dispatching the model request. Network failures
from that refresh are wrapped by Google Auth as
google.auth.exceptions.TransportError.

The complete _async_request_once() operation, including credential refresh,
is already wrapped by the configured Tenacity policy. However, retry_args()
only retries selected APIError status codes and HTTPX timeout/connect errors.
It does not include Google Auth's equivalent transport exception, so the first
refresh failure terminates the request regardless of the configured attempt
count.

Network-free reproduction

import httpx
import tenacity

from google.auth.exceptions import TransportError
from google.genai._api_client import retry_args
from google.genai.types import HttpRetryOptions


def attempts_for(error):
  attempts = 0

  def fail():
    nonlocal attempts
    attempts += 1
    raise error

  retrying = tenacity.Retrying(
      **retry_args(
          HttpRetryOptions(
              attempts=3,
              initial_delay=0.001,
              max_delay=0.001,
              jitter=0.001,
          )
      )
  )
  try:
    retrying(fail)
  except Exception:
    pass
  return attempts


print(attempts_for(httpx.ConnectError('synthetic')))
print(attempts_for(TransportError('synthetic')))

Actual output:

3
1

Expected output:

3
3

An end-to-end reproduction with flaky credentials and a mocked model transport
also produces one credential refresh attempt, zero model HTTP requests, and an
immediately propagated TransportError. After including TransportError in
the predicate, the second credential refresh succeeds and exactly one model
HTTP request is dispatched.

Root cause

The current predicate is:

lambda e: (isinstance(e, errors.APIError) and e.code in retriable_codes)
or isinstance(e, _HTTPX_TRANSIENT_EXC)

google.auth.exceptions.TransportError matches neither branch.

Prior art

• httpx.TimeoutException is not retried despite HttpRetryOptions being configured #2337 reported the equivalent omission for HTTPX transport exceptions.
• fix(retry): retry on httpx.TimeoutException with HttpRetryOptions #2345 added httpx.TimeoutException and httpx.ConnectError to this predicate.
• google.auth.exceptions.TransportError not being retried google-auth-library-python#737 concluded that client-library retry
  predicates should account for google.auth.exceptions.TransportError.
• Google Cloud Storage and BigQuery include TransportError in their retry
  predicates.

Suggested fix

Treat google.auth.exceptions.TransportError as retryable when
HttpRetryOptions is enabled, while continuing to exclude permanent
authentication failures such as RefreshError.

The aiohttp request branches currently perform one separate hard-coded retry
for TransportError. If the shared Tenacity predicate is extended, that path
may eventually benefit from consolidation to avoid nested retry policies. That
existing aiohttp behaviour is separate from the credential-refresh failure
above, which occurs before model dispatch.

[kkorpal]

Hi @emileferreira, thanks for the incredibly clear reproduction script! I'm taking this directly to the engineering team. Really appreciate you flagging this.

[emileferreira]

My pleasure, @kkorpal. I'm closing my PR and letting your team handle it from here.