This documentation is part of the "Projects with Books" initiative at zenOSmosis.

The source code for this project is available on GitHub.

Error Handling

Relevant source files

Purpose and Scope

This document describes the error handling architecture in the rust-muxio system. It covers error types at each layer (framing, RPC, transport), error propagation mechanisms, and patterns for handling failures in distributed RPC calls. For information about RPC service definitions and method dispatch, see RPC Framework. For transport-specific connection management, see Transport State Management.

Error Type Hierarchy

The rust-muxio system defines errors at three distinct layers, each with its own error type. These types compose to provide rich error context while maintaining clean layer separation.

Sources:

graph TB
    subgraph "Application Layer Errors"
        RpcServiceError["RpcServiceError\n(muxio-rpc-service)"]
RpcVariant["Rpc(RpcServiceErrorPayload)"]
TransportVariant["Transport(io::Error)"]
CancelledVariant["Cancelled"]
end
    
    subgraph "RPC Error Codes"
        NotFound["NotFound\nMETHOD_ID not registered"]
Fail["Fail\nHandler returned error"]
System["System\nInternal failure"]
Busy["Busy\nResource unavailable"]
end
    
    subgraph "Framing Layer Errors"
        FrameEncodeError["FrameEncodeError\n(muxio core)"]
FrameDecodeError["FrameDecodeError\n(muxio core)"]
CorruptFrame["CorruptFrame"]
end
    
 
   RpcServiceError --> RpcVariant
 
   RpcServiceError --> TransportVariant
 
   RpcServiceError --> CancelledVariant
    
 
   RpcVariant --> NotFound
 
   RpcVariant --> Fail
 
   RpcVariant --> System
 
   RpcVariant --> Busy
    
 
   TransportVariant --> FrameEncodeError
 
   TransportVariant --> FrameDecodeError
    
 
   FrameEncodeError --> CorruptFrame
 
   FrameDecodeError --> CorruptFrame

RpcServiceError

RpcServiceError is the primary error type exposed to application code. It represents failures that can occur during RPC method invocation, from method lookup through response decoding.

Error Variants

Variant	Description	Typical Cause
`Rpc(RpcServiceErrorPayload)`	Remote method handler returned an error	Handler logic failure, validation error, or internal server error
`Transport(io::Error)`	Low-level transport or encoding failure	Network disconnection, frame corruption, or serialization failure
`Cancelled`	Request was cancelled before completion	Connection dropped while request was pending

RpcServiceErrorPayload

The RpcServiceErrorPayload structure provides detailed information about server-side errors:

RpcServiceErrorCode

Code	Usage	Example Scenario
`NotFound`	Method ID not registered on server	Client calls method that server doesn't implement
`Fail`	Handler explicitly returned an error	Business logic validation failed (e.g., "item does not exist")
`System`	Internal server panic or system failure	Handler panicked or server encountered critical error
`Busy`	Server cannot accept request	Resource exhaustion or rate limiting

Sources:

Framing Layer Errors

The framing layer defines two error types for low-level binary protocol operations. These errors are typically wrapped in RpcServiceError::Transport before reaching application code.

FrameEncodeError

Occurs when encoding RPC requests or responses into binary frames fails. Common causes:

Frame data exceeds maximum allowed size
Corrupt internal state during encoding
Memory allocation failure

FrameDecodeError

Occurs when decoding binary frames into RPC messages fails. Common causes:

Malformed or truncated frame data
Protocol version mismatch
Corrupt frame headers
Mutex poisoning in shared state

Sources:

Error Propagation Through RPC Layers

The following diagram shows how errors flow from their point of origin through the system layers to the application code:

Sources:

sequenceDiagram
    participant App as "Application Code"
    participant Caller as "RpcCallPrebuffered\ntrait"
    participant Client as "RpcServiceCallerInterface\nimplementation"
    participant Dispatcher as "RpcDispatcher"
    participant Transport as "WebSocket or\nother transport"
    participant Server as "RpcServiceEndpointInterface"
    participant Handler as "Method Handler"
    
    Note over App,Handler: Success Path (for context)
    App->>Caller: method::call(args)
    Caller->>Client: call_rpc_buffered()
    Client->>Dispatcher: call()
    Dispatcher->>Transport: emit frames
    Transport->>Server: receive frames
    Server->>Handler: invoke handler
    Handler->>Server: Ok(response_bytes)
    Server->>Transport: emit response
    Transport->>Dispatcher: read_bytes()
    Dispatcher->>Client: decode response
    Client->>Caller: Ok(result)
    Caller->>App: Ok(output)
    
    Note over App,Handler: Error Path 1: Handler Failure
    App->>Caller: method::call(args)
    Caller->>Client: call_rpc_buffered()
    Client->>Dispatcher: call()
    Dispatcher->>Transport: emit frames
    Transport->>Server: receive frames
    Server->>Handler: invoke handler
    Handler->>Server: Err("validation failed")
    Server->>Transport: response with Fail code
    Transport->>Dispatcher: read_bytes()
    Dispatcher->>Client: decode RpcServiceError::Rpc
    Client->>Caller: Err(RpcServiceError::Rpc)
    Caller->>App: Err(RpcServiceError::Rpc)
    
    Note over App,Handler: Error Path 2: Transport Failure
    App->>Caller: method::call(args)
    Caller->>Client: call_rpc_buffered()
    Client->>Dispatcher: call()
    Dispatcher->>Transport: emit frames
    Transport--xClient: connection dropped
    Dispatcher->>Dispatcher: fail_all_pending_requests()
    Dispatcher->>Client: RpcStreamEvent::Error
    Client->>Caller: Err(RpcServiceError::Cancelled)
    Caller->>App: Err(RpcServiceError::Cancelled)

Dispatcher Error Handling

The RpcDispatcher is responsible for correlating requests with responses and managing error conditions during stream processing.

Mutex Poisoning Policy

The dispatcher uses a Mutex to protect the shared request queue. If this mutex becomes poisoned (i.e., a thread panicked while holding the lock), the dispatcher implements a "fail-fast" policy:

This design choice prioritizes safety over graceful degradation. A poisoned queue indicates partial state mutation, and continuing could lead to:

Incorrect request/response correlation
Data loss or duplication
Undefined behavior in dependent code

Sources:

src/rpc/rpc_dispatcher.rs:85-118

Stream Error Events

When decoding errors occur during stream processing, the dispatcher generates RpcStreamEvent::Error events:

These events are delivered to registered response handlers, allowing them to detect and react to mid-stream failures.

Sources:

src/rpc/rpc_dispatcher.rs:187-207

graph TB
 
   A["fail_all_pending_requests()"] --> B["Take ownership of\nresponse_handlers map"]
B --> C["Iterate over all\npending request IDs"]
C --> D["Create synthetic\nRpcStreamEvent::Error"]
D --> E["Invoke handler\nwith error event"]
E --> F["Handler wakes\nawaiting Future"]
F --> G["Future resolves to\nRpcServiceError::Cancelled"]

Connection Failure Cleanup

When a transport connection is dropped, all pending requests must be notified to prevent indefinite waiting. The fail_all_pending_requests() method handles this:

This ensures that all client code waiting for responses receives a timely error indication rather than hanging indefinitely.

Sources:

src/rpc/rpc_dispatcher.rs:427-457

graph TB
 
   Start["call(client, input)"] --> Encode["encode_request(input)"]
Encode -->|io::Error| EncodeErr["Return RpcServiceError::Transport"]
Encode -->|Success| CreateReq["Create RpcRequest"]
CreateReq --> CallBuffered["call_rpc_buffered()"]
CallBuffered -->|RpcServiceError| ReturnErr1["Return error directly"]
CallBuffered -->|Success| Unwrap["Unwrap nested result"]
Unwrap --> CheckInner["Check inner Result"]
CheckInner -->|Ok bytes| Decode["decode_response(bytes)"]
CheckInner -->|Err RpcServiceError| ReturnErr2["Return RpcServiceError"]
Decode -->|io::Error| DecodeErr["Wrap as Transport error"]
Decode -->|Success| Success["Return decoded output"]

Error Handling in RpcCallPrebuffered

The RpcCallPrebuffered trait implementation demonstrates the complete error handling flow from client perspective:

The nested result structure (Result<Result<T, io::Error>, RpcServiceError>) separates transport-level errors from decoding errors:

Outer Result: Transport or RPC-level errors (RpcServiceError)
Inner Result: Decoding errors after successful transport (io::Error from deserialization)

Sources:

extensions/muxio-rpc-service-caller/src/prebuffered/traits.rs:49-98

extensions/muxio-tokio-rpc-client/tests/prebuffered_integration_tests.rs:205-240

Mock Client Error Injection

Unit tests use mock clients to inject specific error conditions:

Sources:

extensions/muxio-rpc-service-caller/tests/prebuffered_caller_tests.rs:135-177

Error Code Mapping

The following table shows how different failure scenarios map to RpcServiceErrorCode values:

Scenario	Code	Typical Message	Detected By
Method ID not in registry	`NotFound`	"Method not found"	Server endpoint
Handler returns `Err(String)`	`System`	Handler error message	Server endpoint
Handler panics	`System`	"Method has panicked"	Server endpoint (catch_unwind)
Business logic failure	`Fail`	Custom validation message	Handler implementation
Transport disconnection	N/A (Cancelled variant)	N/A	Dispatcher on connection drop
Frame decode error	N/A (Transport variant)	Varies	Framing layer
Serialization failure	N/A (Transport variant)	"Failed to encode/decode"	Bitcode layer

Sources:

Best Practices

For Service Implementations

UseFail for Expected Errors: Return Err("descriptive message") from handlers for expected failure cases like validation errors or missing resources.
Let System Handle Panics : If a handler panics, the server automatically converts it to a System error. No explicit panic handling is needed.
Provide Descriptive Messages : Error messages are transmitted to clients and should contain enough context for debugging without exposing sensitive information.

For Client Code

Match on Error Variants : Distinguish between recoverable errors (Rpc with Fail code) and fatal errors (Cancelled, Transport).
Handle Connection Loss : Be prepared for Cancelled errors and implement appropriate reconnection logic.
Don't Swallow Transport Errors : Transport errors indicate serious issues like protocol corruption and should be logged or escalated.

For Testing

Test Both Success and Failure Paths : Every RPC method should have tests for successful calls and expected error conditions.
Verify Error Codes : Match on specific RpcServiceErrorCode values rather than just checking is_err().
Test Connection Failures : Simulate transport disconnection to ensure proper cleanup and error propagation.

Sources:

Dismiss

Refresh this wiki

Enter email to refresh

rust-muxio Documentation

Error Handling

Purpose and Scope

Error Type Hierarchy

RpcServiceError

Error Variants

RpcServiceErrorPayload

RpcServiceErrorCode

Framing Layer Errors

FrameEncodeError

FrameDecodeError

Error Propagation Through RPC Layers

Dispatcher Error Handling

Mutex Poisoning Policy

Stream Error Events

Connection Failure Cleanup

Error Handling in RpcCallPrebuffered

Testing Error Conditions

Handler Failures

Method Not Found

Mock Client Error Injection

Error Code Mapping

Best Practices

For Service Implementations

For Client Code

For Testing

Keyboard shortcuts

rust-muxio Documentation