Hot questions for Using Cap'n Proto in rust

Question:

I need to send an RPC with the Rust Cap'n Proto API that exceeds the default message traversal limit. When I try to send the message, I get the following error:

remote exception: <class \'capnp.lib.capnp.KjException
\'>:capnp/arena.c++:130: failed: Exceeded message traversal limit.
See capnp::ReaderOptions.

If I were reading a message using capnp::serialize::read_message, I could provide a ReaderOptions struct specifying a new traversal limit.

However, I am making and sending a request like in the calculator example instead of reading a message directly. How can I set the traversal limit for my request message?


Answer:

You can set it when you construct the VatNetwork:

pub fn new<U>(
    input_stream: T,
    output_stream: U,
    side: Side,
    receive_options: ReaderOptions,
) -> VatNetwork<T>
where
    U: Write + 'static,

https://github.com/capnproto/capnproto-rust/issues/124#issuecomment-468877758

Question:

How should one use Cap'n Proto for an application's mutable state similar to how Protobuf gets used? Is there a garbage collector?

Kenton Varda confirmed in his comparison of Cap'n Proto, FlatBuffers, and SBE that Cap'n Proto uses arena allocators internally to messages. A single message would grow without bound if one edits it over an extended period, say due to being written to disk and reloaded.

Are there any garbage collectors for Cap'n Proto to rearange the message and reclaim any wasted space? Would a garbage collector be the optimal approach? If not, or if not exists, then what is the recommended approach?

I'm actually writing a Rust program that must only save encrypted data anyways. I'm therefore okay with recopying the whole message structure, but I'm curious about the options more widely.


Answer:

The only way to reclaim wasted space is to copy the message into a new MessageBuilder. Only the "used" parts will be copied. This effectively is "GC" -- many of the best GC algorithms actually move data, which is what you'd be doing here.

There is no practical way to implement non-moving GC of arena-allocated Cap'n Proto messages.

I am considering extending the Cap'n Proto code generator in C++ to also generate a set of classes appropriate for representing the same data structures on the heap, such that you can modify the structure over time. Converting between the heap representation and the arena representation will require a copy, of course. But, this isn't yet implemented and I don't have any timeline. (The Rust implementation would likely get a similar update.)