Skip to content

fix: map a failed outbound RPC instead of passing its status to the caller - #50

Merged
AlexeyShalaev merged 1 commit into
masterfrom
fix/48-mask-outbound-rpc-error
Oct 16, 2026
Merged

AlexeyShalaev merged 1 commit into
masterfrom
fix/48-mask-outbound-rpc-error

Conversation

@AlexeyShalaev

Copy link
Copy Markdown
Member

Summary

AsyncExceptionHandlerInterceptor re-raised every grpc.RpcError untouched. grpc.aio.AbortError is not an RpcError (AbortError -> BaseError -> Exception), and the only RpcError subclass grpcio defines is the client-side grpc.aio.AioRpcError. So the clause never passed through a status this RPC chose; it only let through failures of calls the handler made to other services, and grpc.aio answered the caller with UNKNOWN and the downstream error's repr (status, details, debug string) as details.

The clause is gone; the AbortError one stays. A failed outbound call now goes the same way as any other exception:

  • status from find_mapped_status: INTERNAL with the default map, since RpcError is not in it;
  • details from detail_factory: "Internal server error" by default;
  • logger.exception with the traceback, so the downstream status and details are still in the log.

Because it goes through the map, {grpc.RpcError: grpc.StatusCode.UNAVAILABLE} in error_status_map reports every downstream failure as UNAVAILABLE (or whatever the service wants). No new option.

The issue's script before and after:

before: status: UNKNOWN   details: Unexpected <class 'grpc.aio._call.AioRpcError'>: <AioRpcError ... details = "ledger dsn=postgres://ledger:s3cret@10.0.3.7/ledger" ...>
after:  status: INTERNAL  details: Internal server error

Considered and rejected

  • Forwarding the downstream status by default. It describes this server's call, not the caller's request: a downstream INVALID_ARGUMENT or UNAUTHENTICATED means this server sent a bad request or holds bad credentials, and passing it on blames the caller. INTERNAL is the default; a service that knows a particular downstream code is the caller's business catches the AioRpcError in the handler and aborts with it, and the guide says so.
  • Changing AsyncSentryInterceptor, which also skips grpc.RpcError. Capturing every downstream outage as a new event is a separate decision. The failure is already logged at ERROR with its traceback by the exception handler. Left as it is; the observability guide now says where such failures go.

Compatibility

This changes what callers receive, but only in one case: a response that was UNKNOWN with the downstream repr becomes INTERNAL with the safe detail. The old status carried no information (grpc.aio always answered UNKNOWN for these), so nothing could branch on it; what it did carry was the downstream details. Released as a fix:, not a breaking change.

Interceptors outside the handler (metrics, request logger, tracing) now record INTERNAL, which is what the caller actually gets. Before, they recorded the downstream code from exc.code() while the caller got UNKNOWN. Tracing still records the original AioRpcError on the span through the AbortError's __context__.

A custom detail_factory now also receives AioRpcError instances. It already had to handle arbitrary exceptions (any unmapped type reaches it), so a factory that echoes str(exc) leaks here exactly as it leaks for any other exception.

servicewright

servicewright#86 is the same bug in UnhandledErrorInterceptor, which is its own implementation on top of this kit's AsyncServerInterceptor base, not on AsyncExceptionHandlerInterceptor. Both now behave the same way: an outbound RpcError is logged with its traceback and answered with a masked INTERNAL, a deliberate AbortError passes through. The wording on the wire stays each library's own (Internal server error here; internal_error plus x-error-code in servicewright, which has no counterpart in this kit).

Tests

  • Unit: an outbound AioRpcError (downstream INTERNAL and PERMISSION_DENIED) aborts with INTERNAL / "Internal server error" and logs the original exception with its traceback; grpc.RpcError in error_status_map picks the status. rpc_error is dropped from the pass-through parametrization.
  • Integration: a second real server whose handler calls the test server's AbortDirectly (PERMISSION_DENIED, "handler-abort"); its caller gets INTERNAL / "Internal server error", and its metrics record INTERNAL.
  • With master's exception_handler.py put back, all four new tests fail (the integration one with UNKNOWN != INTERNAL).
  • ruff check, ruff format --check, mypy grpc_server_kit: clean. pytest --cov: 441 passed, 97.60% coverage.

Type of change

  • Bug fix
  • Documentation update

Checklist

  • Tests added or updated
  • make check passes locally (ruff + mypy)
  • Conventional-commit title (the changelog is generated by Release Please — do not edit it by hand)
  • Documentation updated (if the public API changed), docs/agents.md included

Related issues

Fixes #48

…aller

AsyncExceptionHandlerInterceptor re-raised every grpc.RpcError untouched.
grpc.aio.AbortError is not an RpcError, so that clause only ever let through
grpc.aio.AioRpcError from calls the handler made, and grpc.aio answered the
caller with UNKNOWN and the downstream error's repr as details.

Such an error now goes through the status map like any other exception:
INTERNAL with the safe detail by default, logged with its traceback.
Mapping grpc.RpcError in error_status_map picks another status.

Fixes #48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AsyncExceptionHandlerInterceptor lets a downstream grpc.RpcError through: the caller gets UNKNOWN with the other service's details

1 participant