Hot questions for Using Cap'n Proto in types

Question:

I am playing around with Rust's capnproto library. Because Rust can infer types in some situations, I can do things like this:

let mut message = ::capnp::message::Builder::new_default();

Without having to know the type of message. If I want to pass a reference to message into a function, I now need to know what message is to let the function know what to expect.

Is there a convenient way to do this in general?

So far I have done the following:

let testing: () = message;

which fails with the compiler error:

error[E0308]: mismatched types
   --> src/main.rs:197:18
    |
197 |                 let temp: () = message;
    |                           ^^^^^^^ expected (), found struct `capnp::message::Builder`

But when I type annotate my function as follows:

fn example_fn(message: capnp::message::Builder) {...}

I get an error like:

error[E0243]: wrong number of type arguments: expected 1, found 0
  --> src/main.rs:72:32
   |
72 | fn dump_capnp_to_file(message: capnp::message::Builder, filename: &str) {
   |                                ^^^^^^^^^^^^^^^^^^^^^^^ expected 1 type argument

error: aborting due to previous error

I'm very new to Rust coming from a C++ background; sorry if this a rookie question!


Answer:

Rust will not infer types in function parameter position. This is by design, as the Rust language FAQ states:

Why aren't function signatures inferred?

In Rust, declarations tend to come with explicit types, while actual code has its types inferred. There are several reasons for this design:

  • Mandatory declaration signatures help enforce interface stability at both the module and crate level.

  • Signatures improve code comprehension for the programmer, eliminating the need for an IDE running an inference algorithm across an entire crate to be able to guess at a function’s argument types; it’s always explicit and nearby.

  • Mechanically, it simplifies the inference algorithm, as inference only requires looking at one function at a time.

Since capnp::message::Builder<A> takes a type parameter A, you need to qualify the type of the parameter by giving A a value:

fn dump_capnp_to_file(message: capnp::message::Builder<SomeType>, filename: String) {
//                                                    ^^^^^^^^^^

or else make your function also generic so it can accept any type A:

fn dump_capnp_to_file<A>(message: capnp::message::Builder<A>, filename: String) {
//                   ^^^                                 ^^^
Putting bounds on A

If you take the last option, you might want additional trait bounds to allow you to do different things with message inside the function. For example, you might want to send message to another thread, which requires that Builder<A> implement Send. Builder has the following impl (reference):

impl <A> Send for Builder<A> where A: Send + Allocator

which means that Builder<A> can implement Send, but only when A implements Send and Allocator. You can make that your own bound (requirement) on A:

fn dump_capnp_to_file<A>(message: capnp::message::Builder<A>, filename: String)
    where A: Send + Allocator
{
    // multi-threaded code...
}

Alternatively (and maybe slightly better), bound Builder<A> with Send directly:

fn dump_capnp_to_file<A>(message: capnp::message::Builder<A>, filename: String)
    where capnp::message::Builder<A>: Send

Then you will only be able to call dump_capnp_to_file on a Builder that implements Send.

Question:

I'm using Cap'n proto to send and retrieve messages between multiple clients and my websocket server.

Since I have only one websocket channel to send and receive data and various type of messages can be sent, I need a way to distinguish between then when trying to decode it.

How can this be done correctly in Cap'n proto?

I think this question is language agnostic, but if a language is needed, I have my server in Rust and clients in Rust, Go and Javascript, so a solution that works in all these languages would be much appreciated.


Answer:

The best thing to do is create an outer struct which is a union of all the possible types. For example, if you have types Foo, Bar, and Baz, define a type like:

struct Outer {
  union {
    foo @0 :Foo;
    bar @1 :Bar;
    baz @2 :Baz;
  }
}

How to access a union depends on the language, but usually there is a which() method that returns an enum value specifying which field is filled in, and then you use the regular getter methods for a nested struct.

Note that both ends have to use this Outer type. You can't serialize a message with a root type of Foo and then parse it as an Outer -- the message has to be created with type Outer. In general there is no way to distinguish different Cap'n Proto types based on the bytes alone.

Question:

Below is a code snippet in Python that stores the IP prefixes in a radix tree and then associates IP and ASNs in a dictionary, if the IP belongs to a prefix.

I would like to find out all different ASNs for a particular prefix. More details are provided below:

#rtree is a radix tree which has prefixes stored.
rtree = radix.Radix()    
with open(path-to-prefix-file,'r') as fp:
    for line in fp:
        rnode = rtree.add(line)  # Eg, Prefixes like "192.168.2.0/24"
        rnode.data["count"]= 0
...        
# The code has several lines here for processing a capnproto - skipping them.

rnode.data[IP]=asn_complete  # I read a Capnproto buffer and store IP and asn_complete

...

for rnode in rtree:
    seen_list = []  # defining a list to store different val, i.e.,asn_complete values
    if rnode.data["count"] > 1:
            """  Iterate through the rnode.data dictionary """
            for ip,val in rnode.data.iteritems():
                    if val not in seen_list:  # Condition is always satisfied!!
                            seen_list.append(val) 

For eg: val has the following value from the protobuf in several iterations:

[<capnp list reader [15169]>, <capnp list reader [1239]>, <capnp list reader [4837]>]

When I print out the seen_list:

[[<capnp list reader [15169]>, <capnp list reader [1239]>, <capnp list reader [4837]>], [<capnp list reader [15169]>, <capnp list reader [1239]>, <capnp list reader [4837]>], [<capnp list reader [15169]>, <capnp list reader [1239]>, <capnp list reader [4837]>],....]

Clearly val is in seen_list; but, if val not in seen_list: is always true and val gets appended to seen_list so many times. I don't understand why the condition always returns true. Is it because of the type of object stored in seen_list?


Answer:

At present, Cap'n Proto readers do not support any sort of "equality" comparison. Partly this is because it's unclear what equality should mean: should it be by identity (two readers are equal if they point at exactly the same object) or should it be by value (they are equal if they point at objects with equivalent content)?

In any case, in requires an implementation of __eq__ to test for equality, and in the case of Cap'n Proto there is no such implementation. Probably, what ends up happening is that Python is comparing the wrapper objects by identity -- and as new wrapper objects keep being created, these comparisons are always false.

In order to get what you want, you'll probably need to convert the Cap'n Proto objects fully into plain Python objects which are properly comparable.