This documentation is part of the "Projects with Books" initiative at zenOSmosis.

The source code for this project is available on GitHub.

Transport and Framing Errors

Loading…

Transport and Framing Errors

Relevant source files

This page documents low-level transport and framing errors in the muxio system. These errors occur at the binary protocol layer, connection management layer, and dispatcher coordination layer. For application-level RPC service errors (method not found, invalid parameters, handler exceptions), see RPC Service Errors.

Scope : This page covers FrameDecodeError, FrameEncodeError, connection failures, transport state transitions, dispatcher mutex poisoning, and cleanup mechanisms when connections drop unexpectedly.

Overview of Transport and Framing Error Types

The muxio system defines errors at multiple layers of the transport stack. Each layer reports failures using specific error types that propagate upward through the system.

graph TB
    subgraph "Low-Level Frame Errors"
        FDE["FrameDecodeError"]
FEE["FrameEncodeError"]
end
    
    subgraph "Connection Errors"
        CE["io::Error\nConnectionRefused\nConnectionReset"]
TE["Transport Errors\nWebSocket failures\nNetwork timeouts"]
end
    
    subgraph "Dispatcher Coordination Errors"
        MP["Mutex Poisoning\nPoisonError"]
PC["Pending Call Failures\nReadAfterCancel"]
end
    
    subgraph "RPC Layer Errors"
        RSE["RpcServiceError\nTransport variant"]
end
    
 
   FDE -->|wrapped in| RSE
 
   CE -->|converted to| RSE
 
   TE -->|triggers| PC
 
   MP -->|panics| PANIC["System Panic"]
PC -->|error events to| RSE
    
    style PANIC fill:#ffcccc

Error Type Hierarchy

Sources :

Framing Protocol Errors

FrameDecodeError

FrameDecodeError represents failures when parsing incoming binary frames. These errors occur in the RpcSession decoder when frame headers are malformed, stream state is inconsistent, or data is corrupted.

Variant	Description	When It Occurs
`CorruptFrame`	Frame header is invalid or stream state is inconsistent	Malformed binary data, protocol violation
`ReadAfterCancel`	Attempt to read from a cancelled stream	Connection dropped mid-stream, explicit cancellation
`UnexpectedEnd`	Stream ended prematurely without End frame	Transport closed unexpectedly
Other variants	(Implementation-specific)	Various protocol violations

Sources :

FrameEncodeError

FrameEncodeError represents failures when encoding outbound frames. These are less common than decode errors since encoding is deterministic, but can occur when stream state is invalid or resources are exhausted.

Variant	Description	When It Occurs
`CorruptFrame`	Internal state inconsistency	Invalid encoder state, logic error
Other variants	(Implementation-specific)	Resource exhaustion, invalid input

Sources :

Connection and Transport Errors

Connection Establishment Failures

When a client attempts to connect to a non-existent or unreachable server, the connection fails immediately with an io::Error of kind ConnectionRefused.

Connection Flow with Error :

Sources :

Connection Drop During Operation

When a connection drops while RPC calls are in flight, the client must:

Detect the disconnection
Fail all pending requests
Notify state change handlers
Prevent new requests from being sent

Disconnect Detection and Handling :

Sources :

Dispatcher Error Handling

Mutex Poisoning

The RpcDispatcher uses a Mutex<VecDeque<(u32, RpcRequest)>> to track pending inbound responses. If a thread panics while holding this mutex, the mutex becomes “poisoned” and all subsequent lock attempts return Err(PoisonError).

Design Decision : The dispatcher treats mutex poisoning as a critical, unrecoverable error and panics immediately rather than attempting recovery.

Rationale (from src/rpc/rpc_dispatcher.rs:85-97):

If the lock is poisoned, it likely means another thread panicked while holding the mutex. The internal state of the request queue may now be inconsistent or partially mutated. Continuing execution could result in incorrect dispatch behavior, undefined state transitions, or silent data loss. This should be treated as a critical failure and escalated appropriately.

Mutex Poisoning Detection :

Poisoning Sites :

Location	Purpose	Panic Behavior
src/rpc/rpc_dispatcher.rs:104-118	`init_catch_all_response_handler`	Panics if queue lock is poisoned
src/rpc/rpc_dispatcher.rs:367-370	`read_bytes` queue access	Returns `FrameDecodeError::CorruptFrame`

Sources :

Failing Pending Requests on Disconnect

When a transport connection drops, the RpcDispatcher::fail_all_pending_requests() method ensures that all in-flight RPC calls are notified of the failure. This prevents deadlocks where application code waits indefinitely for responses that will never arrive.

Cleanup Sequence :

Implementation Details (src/rpc/rpc_dispatcher.rs:422-456):

Take Ownership : std::mem::take(&mut self.rpc_respondable_session.response_handlers) moves all handlers out of the map, leaving it empty
Synthetic Error : Creates RpcStreamEvent::Error { frame_decode_error: error, ... } for each handler
Invoke Handlers : Calls each handler with the error event, waking any awaiting futures
Memory Safety : Handlers are dropped after invocation, preventing leaks

Sources :

graph TB
    subgraph "Transport Layer"
        T1["TCP/IP Error\nio::Error"]
T2["WebSocket Error\ntungstenite::Error"]
end
    
    subgraph "Framing Layer"
        F1["FrameDecodeError\nRpcSession::read_bytes()"]
F2["FrameEncodeError\nRpcStreamEncoder"]
end
    
    subgraph "RPC Protocol Layer"
        R1["RpcDispatcher::read_bytes()\nReturns Vec<u32> or Error"]
R2["RpcDispatcher::call()\nReturns Encoder or Error"]
end
    
    subgraph "RPC Service Layer"
        S1["RpcServiceError::Transport\nWrapped frame errors"]
S2["RpcServiceCallerInterface::call_rpc_prebuffered()\nReturns Result"]
end
    
    subgraph "Application Layer"
        A1["Application Code\nReceives Result<T, RpcServiceError>"]
end
    
 
   T1 -->|Connection drops| T2
 
   T2 -->|Stream error in ws_receiver.next| F1
 
   F1 -->|propagated via read_bytes| R1
 
   R1 -->|FrameDecodeError| S1
    
 
   F2 -->|Encode failure| R2
 
   R2 -->|FrameEncodeError| S1
    
 
   S1 --> S2
 
   S2 --> A1
    
    style T1 fill:#ffe6e6
    style F1 fill:#fff0e6
    style S1 fill:#e6f7ff

Error Propagation Through Layers

Errors flow upward through the muxio layer stack, with each layer translating or wrapping errors as appropriate for its abstraction level.

Error Flow Diagram

Sources :

Error Handling in Stream Processing

Per-Stream Error Events

When a stream encounters a decode error, the RpcSession emits an RpcStreamEvent::Error event to the registered handler. This allows stream-specific error handling without affecting other concurrent streams.

Error Event Structure :

Handler Processing (src/rpc/rpc_dispatcher.rs:187-206):

Sources :

Catch-All Response Handler

The dispatcher installs a catch-all handler to process incoming response events that don’t have a specific registered handler. This handler is responsible for error logging and queue management.

Handler Registration (src/rpc/rpc_dispatcher.rs:98-209):

Sources :

src/rpc/rpc_dispatcher.rs:98-209

stateDiagram-v2
    [*] --> Connecting: RpcClient::new() called
    Connecting --> Connected : WebSocket handshake complete
    Connecting --> [*]: Connection error\n(io::Error returned)
    
    Connected --> Disconnecting : WebSocket error detected
    Connected --> Disconnecting : shutdown_async() called
    
    Disconnecting --> Disconnected : is_connected.swap(false)
    Disconnected --> HandlerNotified : Call state_change_handler
    HandlerNotified --> RequestsFailed : fail_all_pending_requests()
    RequestsFailed --> [*] : Cleanup complete
    
    note right of Connected
        Heartbeat pings sent
        RPC calls processed
    end note
    
    note right of RequestsFailed
        All pending RPC calls
        resolved with errors
    end note

Connection State Management

The RpcClient tracks connection state using an AtomicBool (is_connected) and notifies application code via state change handlers.

State Transition Diagram

State Change Handler Contract :

State	When Called	Guarantees
`RpcTransportState::Connected`	Immediately after `set_state_change_handler()` if connected	Client is ready for RPC calls
`RpcTransportState::Disconnected`	On connection drop, explicit shutdown, or client Drop	All pending requests have been failed

Sources :

Recovery and Cleanup Strategies

Automatic Cleanup on Drop

The RpcClient implements Drop to ensure graceful shutdown when the client is destroyed:

Drop Implementation (extensions/muxio-tokio-rpc-client/src/rpc_client.rs:42-52):

Limitations : The synchronous Drop trait cannot await async operations, so fail_all_pending_requests() is only called when shutdown_async() is explicitly invoked by background tasks detecting errors.

Sources :

extensions/muxio-tokio-rpc-client/src/rpc_client.rs:42-77

Manual Disconnect Handling

Applications can register state change handlers to implement custom cleanup logic:

Best Practices :

Always handleDisconnected: Assume all in-flight RPC calls have failed
Avoid blocking operations : Handler is called synchronously from disconnect detection path
Use channels for async work : Spawn tasks rather than awaiting in the handler

Sources :

extensions/muxio-tokio-rpc-client/tests/transport_state_tests.rs:112-126

Error Handling Patterns in Tests

Testing Connection Failures

The test suite validates error handling through various scenarios:

Connection Refusal Test (extensions/muxio-tokio-rpc-client/tests/transport_state_tests.rs:15-31):

Attempts connection to unused port
Verifies io::Error with ErrorKind::ConnectionRefused
Confirms no panic or hang

Disconnect During Operation Test (extensions/muxio-tokio-rpc-client/tests/transport_state_tests.rs:167-292):

Establishes connection
Spawns RPC call
Closes server connection
Verifies pending call fails with error containing “cancelled stream” or “Transport error”

sequenceDiagram
    participant Test as Test Code
    participant Server as Mock Server
    participant Client as RpcClient
    participant RPC as Pending RPC Call
    
    Test->>Server: Start listener
    Test->>Client: RpcClient::new()
    Test->>RPC: Spawn Echo::call() in background
    Test->>Test: Sleep to let RPC become pending
    
    Note over RPC: RPC call waiting in\ndispatcher.response_handlers
    
    Test->>Server: Signal to close connection
    Server->>Client: Close WebSocket
    Client->>Client: Detect error in ws_receiver
    Client->>Client: shutdown_async()
    Client->>Client: fail_all_pending_requests()
    Client->>RPC: Emit Error event
    RPC-->>Test: Return Err(RpcServiceError)
    
    Test->>Test: Assert error contains\n"cancelled stream"

Test Pattern :

Sources :

Summary Table: Error Types and Handling

Error Type	Layer	Cause	Handling Strategy
`FrameDecodeError::CorruptFrame`	Framing	Malformed binary data	Log error, drop stream
`FrameDecodeError::ReadAfterCancel`	Framing	Stream cancelled	Propagate to RPC layer
`FrameEncodeError`	Framing	Encoder state error	Return error to caller
`io::Error::ConnectionRefused`	Transport	Server not reachable	Return from `new()`
`tungstenite::Error`	Transport	WebSocket failure	Trigger `shutdown_async()`
`PoisonError<T>`	Dispatcher	Thread panic with lock	PANIC (critical failure)
`RpcServiceError::Transport`	RPC Service	Wrapped lower-level error	Return to application

Sources :

Keyboard shortcuts

rust-muxio Documentation