This documentation is part of the "Projects with Books" initiative at zenOSmosis.

The source code for this project is available on GitHub.

Serialization with Bitcode

Relevant source files

Purpose and Scope

This document explains how the bitcode library is used for binary serialization in the rust-muxio RPC system. It covers the serialization architecture, the integration with service definitions, encoding strategies for different payload sizes, and performance characteristics.

For information about defining custom RPC services that use this serialization, see Creating Service Definitions. For information about compile-time method identification, see Method ID Generation.

Overview

The rust-muxio system uses bitcode version 0.6.6 as its primary serialization format for encoding and decoding RPC request parameters and response values. Bitcode is a compact, binary serialization library that integrates with Serde, providing efficient serialization with minimal overhead compared to text-based formats like JSON.

Sources: Cargo.lock:158-168

Why Bitcode

The system uses bitcode for several key reasons:

Feature	Benefit
Binary Format	Significantly smaller payload sizes compared to JSON or other text formats
Serde Integration	Works seamlessly with Rust's standard serialization ecosystem via `#[derive(Serialize, Deserialize)]`
Type Safety	Preserves Rust's strong typing across the network boundary
Performance	Fast encoding and decoding with minimal CPU overhead
Cross-Platform	Identical binary representation on native and WASM targets

The compact binary representation is particularly important for the multiplexed stream architecture, where multiple concurrent RPC calls share a single WebSocket connection. Smaller payloads mean lower latency and higher throughput.

Sources: Cargo.lock:158-168 high-level architecture diagrams

Integration with RpcMethodPrebuffered

Bitcode serialization is accessed through the RpcMethodPrebuffered trait, which defines four core methods for encoding and decoding:

Sources: Service definition pattern from integration tests

graph TB
    subgraph "RpcMethodPrebuffered Trait"
        TRAIT[RpcMethodPrebuffered]
        
        ENC_REQ["encode_request(Input)\n→ Result&lt;Vec&lt;u8&gt;, io::Error&gt;"]
DEC_REQ["decode_request(&[u8])\n→ Result&lt;Input, io::Error&gt;"]
ENC_RES["encode_response(Output)\n→ Result&lt;Vec&lt;u8&gt;, io::Error&gt;"]
DEC_RES["decode_response(&[u8])\n→ Result&lt;Output, io::Error&gt;"]
TRAIT --> ENC_REQ
 
       TRAIT --> DEC_REQ
 
       TRAIT --> ENC_RES
 
       TRAIT --> DEC_RES
    end
    
    subgraph "Bitcode Library"
        BITCODE_ENC["bitcode::encode()"]
BITCODE_DEC["bitcode::decode()"]
end
    
    ENC_REQ -.uses.-> BITCODE_ENC
    DEC_REQ -.uses.-> BITCODE_DEC
    ENC_RES -.uses.-> BITCODE_ENC
    DEC_RES -.uses.-> BITCODE_DEC
    
    subgraph "Service Implementation"
        ADD_SERVICE["Add Method\nInput: Vec&lt;f64&gt;\nOutput: f64"]
MULT_SERVICE["Mult Method\nInput: Vec&lt;f64&gt;\nOutput: f64"]
ECHO_SERVICE["Echo Method\nInput: Vec&lt;u8&gt;\nOutput: Vec&lt;u8&gt;"]
end
    
    ADD_SERVICE -.implements.-> TRAIT
    MULT_SERVICE -.implements.-> TRAIT
    ECHO_SERVICE -.implements.-> TRAIT

Serialization Flow

The following diagram shows how data transforms from typed Rust values to binary bytes and back during a complete RPC roundtrip:

sequenceDiagram
    participant AppClient as "Application Code\n(Client)"
    participant EncReq as "encode_request\nbitcode::encode"
    participant Transport as "WebSocket Transport\nBinary Frames"
    participant DecReq as "decode_request\nbitcode::decode"
    participant AppServer as "Application Code\n(Server)"
    participant EncRes as "encode_response\nbitcode::encode"
    participant DecRes as "decode_response\nbitcode::decode"
    
    AppClient->>EncReq: Vec<f64> input
    Note over EncReq: Serialize to bytes
    EncReq->>Transport: Vec<u8> (binary)
    
    Transport->>DecReq: Vec<u8> (binary)
    Note over DecReq: Deserialize from bytes
    DecReq->>AppServer: Vec<f64> input
    
    Note over AppServer: Process request
    
    AppServer->>EncRes: f64 output
    Note over EncRes: Serialize to bytes
    EncRes->>Transport: Vec<u8> (binary)
    
    Transport->>DecRes: Vec<u8> (binary)
    Note over DecRes: Deserialize from bytes
    DecRes->>AppClient: f64 output

This flow ensures that application code works with native Rust types on both sides, while bitcode handles the conversion to and from the binary wire format.

Sources: extensions/muxio-rpc-service-caller/src/prebuffered/traits.rs:49-97 extensions/muxio-tokio-rpc-client/tests/prebuffered_integration_tests.rs:36-43

Encoding Strategy for Large Payloads

The system implements an intelligent encoding strategy that adapts based on payload size. This is necessary because RPC header frames have size limitations imposed by the underlying transport protocol.

graph TB
    INPUT["Typed Input\n(e.g., Vec&lt;f64&gt;)"]
ENCODE["encode_request()"]
BYTES["Encoded Bytes\nVec&lt;u8&gt;"]
SIZE_CHECK{"Size ≥\nDEFAULT_SERVICE_MAX_CHUNK_SIZE\n(64 KB)?"}
SMALL["Place in\nrpc_param_bytes\n(Header Field)"]
LARGE["Place in\nrpc_prebuffered_payload_bytes\n(Chunked Stream)"]
REQ["RpcRequest\nmethod_id + params/payload"]
DISPATCHER["RpcDispatcher\nMultiplexing Layer"]
INPUT --> ENCODE
 
   ENCODE --> BYTES
 
   BYTES --> SIZE_CHECK
    
 
   SIZE_CHECK -->|No| SMALL
 
   SIZE_CHECK -->|Yes| LARGE
    
 
   SMALL --> REQ
 
   LARGE --> REQ
    
 
   REQ --> DISPATCHER

This dual-path strategy is implemented in extensions/muxio-rpc-service-caller/src/prebuffered/traits.rs:55-65:

Small Payload Path

When encoded arguments are less than 64 KB (the value of DEFAULT_SERVICE_MAX_CHUNK_SIZE):

Bytes are placed in RpcRequest::rpc_param_bytes
Transmitted in the initial header frame
Most efficient for typical RPC calls

Large Payload Path

When encoded arguments are 64 KB or larger :

Bytes are placed in RpcRequest::rpc_prebuffered_payload_bytes
Automatically chunked by RpcDispatcher
Streamed as multiple frames after the header
Handles arbitrarily large arguments

The server-side endpoint interface contains corresponding logic to check both fields when extracting request parameters.

Sources: extensions/muxio-rpc-service-caller/src/prebuffered/traits.rs:30-72 [muxio-rpc-service constants](https://github.com/jzombie/rust-muxio/blob/fcb45826/muxio-rpc-service constants)

Practical Example: Add Method

Here's how serialization works in practice for the Add RPC method:

Client-Side Encoding

Server-Side Decoding and Processing

Client-Side Response Handling

Sources: extensions/muxio-tokio-rpc-client/tests/prebuffered_integration_tests.rs:36-43 extensions/muxio-tokio-rpc-client/tests/prebuffered_integration_tests.rs:82-88

Large Payload Handling

The system is tested with payloads up to 200x the chunk size (approximately 12.8 MB) to ensure that large data sets can be transmitted reliably:

The RpcDispatcher in the multiplexing layer handles chunking transparently. Neither the application code nor the serialization layer needs to be aware of this streaming behavior.

Sources: extensions/muxio-tokio-rpc-client/tests/prebuffered_integration_tests.rs:155-203 extensions/muxio-wasm-rpc-client/tests/prebuffered_integration_tests.rs:230-312

Error Handling

Bitcode serialization can fail for several reasons:

Encoding Errors

Error Type	Cause	Example
Type Mismatch	Data type doesn't implement `Serialize`	Non-serializable field in struct
Encoding Failure	Bitcode internal error	Extremely nested structures

These are returned as io::Error from encode_request and encode_response methods.

Decoding Errors

Error Type	Cause	Example
Invalid Format	Corrupted bytes or version mismatch	Network corruption
Type Mismatch	Server/client type definitions don't match	Field added without updating both sides
Truncated Data	Incomplete transmission	Connection interrupted mid-stream

These are returned as io::Error from decode_request and decode_response methods.

All serialization errors are converted to RpcServiceError::Transport by the caller interface, ensuring consistent error handling throughout the RPC stack.

Sources: extensions/muxio-rpc-service-caller/src/prebuffered/traits.rs:87-96

graph LR
    subgraph "Service Definition Crate"
        TYPES["Shared Type Definitions\nstruct with Serialize/Deserialize"]
end
    
    subgraph "Native Client/Server"
        NATIVE_ENC["bitcode::encode()\nx86_64 / ARM64"]
NATIVE_DEC["bitcode::decode()\nx86_64 / ARM64"]
end
    
    subgraph "WASM Client"
        WASM_ENC["bitcode::encode()\nWebAssembly"]
WASM_DEC["bitcode::decode()\nWebAssembly"]
end
    
    subgraph "Wire Format"
        BYTES["Binary Bytes\n(Platform-Independent)"]
end
    
    TYPES -.defines.-> NATIVE_ENC
    TYPES -.defines.-> NATIVE_DEC
    TYPES -.defines.-> WASM_ENC
    TYPES -.defines.-> WASM_DEC
    
 
   NATIVE_ENC --> BYTES
 
   WASM_ENC --> BYTES
 
   BYTES --> NATIVE_DEC
 
   BYTES --> WASM_DEC

Cross-Platform Compatibility

One of bitcode's key advantages is that it produces identical binary representations across different platforms:

This ensures that:

Native Tokio clients can communicate with native Tokio servers
WASM clients can communicate with native Tokio servers
Any client can communicate with any server, as long as they share the same service definitions

The cross-platform compatibility is validated by integration tests that run identical test suites against both Tokio and WASM clients.

Sources: extensions/muxio-tokio-rpc-client/tests/prebuffered_integration_tests.rs extensions/muxio-wasm-rpc-client/tests/prebuffered_integration_tests.rs high-level architecture diagrams

Performance Characteristics

Bitcode provides several performance advantages:

Encoding Speed

Bitcode uses a zero-copy architecture where possible, minimizing memory allocations during serialization. For primitive types and simple structs, encoding is extremely fast.

Decoding Speed

The binary format is designed for fast deserialization. Unlike text formats that require parsing, bitcode can often deserialize directly into memory with minimal transformations.

Payload Size Comparison

Format	Example Payload	Size
JSON	`[1.0, 2.0, 3.0]`	~19 bytes
Bitcode	`[1.0, 2.0, 3.0]`	~8 bytes (depends on encoding)

The exact size savings vary by data type and structure, but bitcode consistently produces smaller payloads than text-based formats.

Memory Usage

Bitcode minimizes temporary allocations during encoding and decoding. The library is designed to work efficiently with Rust's ownership model, often allowing zero-copy operations for borrowed data.

Sources: Cargo.lock:158-168 (bitcode dependency), general bitcode library characteristics

Version Compatibility

The service definition crate specifies the exact bitcode version (0.6.6) used across the entire system. This is critical for compatibility:

Shared Dependencies : All crates use the same bitcode version via the workspace
Binary Stability : Bitcode maintains binary compatibility within minor versions
Migration Path : Upgrading bitcode requires updating all clients and servers simultaneously

To ensure compatibility, the service definition crate that implements RpcMethodPrebuffered should be shared as a compiled dependency, not duplicated across codebases.

Sources: Cargo.lock:158-168 workspace structure from high-level diagrams

Summary

The bitcode serialization layer provides:

Efficient binary encoding for request/response data
Transparent integration via RpcMethodPrebuffered trait
Automatic chunking for large payloads via smart routing
Cross-platform compatibility between native and WASM targets
Type safety through compile-time Rust types

Application code remains abstraction-free, working with native Rust types while bitcode handles the low-level serialization details. The integration with the RPC framework ensures that serialization errors are properly propagated as RpcServiceError::Transport for consistent error handling.

Sources: All sections above

Dismiss

Refresh this wiki

Enter email to refresh

Keyboard shortcuts

rust-muxio Documentation