Hot questions for Using Cap'n Proto in list

Question:

I have capnproto definition like this:

struct School {
  name @0 :Text;
  address @1 :Address;
  foundation @2 :Date;
  emailAddresses @3 :List(Text);
}

I would like to set the emailAddresses field in a builder with code similar to this (but this won't compile):

static School::Builder random_School() {
  capnp::MallocMessageBuilder msg;
  School::Builder result = msg.initRoot<School>();
  result.setName(rand_str(36));
  result.setAddress(random_Address());
  result.setFoundation(random_Date());
  result.initEmailAddresses(item_count);
  for (size_t i = 0; i < item_count; ++i) {
    result.getEmailAddresses()[i] = rand_str(37); // rand_str returns std::string
  }
  return result;
}

What is the correct way to do this?


Answer:

According to the capnproto documentation in the Lists section, you should use builder.set(index, value).

result.getEmailAddresses().set(i, rand_str(37));

I guess it should compile now.

Question:

According to the CapnProto documentation: (NOTE: I am using the C++ version)

For List where Foo is a non-primitive type, the type returned by operator[] and iterator::operator*() is Foo::Reader (for List::Reader) or Foo::Builder (for List::Builder). The builder’s set method takes a Foo::Reader as its second parameter.

While using "set" seems to work fine for non-primitive types: Other stack overflow question for primitives only

There does not appear to be a "set" function for automatically generated lists of non-primitives. Did my CapnProto generation fail in some way, or is there another method for setting elements in a list of non-primitives?


Answer:

There is a "set" method, but it is called setWithCaveats():

destListBuilder.setWithCaveats(index, sourceStructReader)

This is to let you know that there are some obscure problems with setting an element of a struct list. The problem stems from the fact that struct lists are not represented as a list of pointers as you might expect, but rather they are a "flattened" series of consecutive structs, all of the same size. This implies that space for all the structs in the list is allocated at the time that you initialize the list. So, when you call setWithCaveats(), the target space is already allocated previously, and you're copying the source struct into that space.

This presents a problem in the face of varying versions: the source struct might have been constructed using a newer version of the protocol, in which additional fields were defined. In this case, it may actually be larger than expected. But, the destination space was already allocated, based on the protocol version you compiled with. So, it's too small! Unfortunately, there's no choice but to discard the newly-defined fields that we don't know about. Hence, data may be lost.

Of course, it may be that in your application, you know that the struct value doesn't come from a newer version, or that you don't care if you lose fields that you don't know about. In this case, setWithCaveats() will do what you want.

If you want to be careful to preserve unknown fields, you may want to look into the method capnp::Orphanage::newOrphanConcat(). This method can concatenate a list of lists of struct readers into a single list in such a way that no data is lost -- the target list is allocated with each struct's size equal to the maximum of all input structs.

auto orphanage = Orphanage::getForMessageContaining(builder);
auto orphan = orphanage.newOrphanConcat({list1Reader, list2Reader});
builder.adoptListField(kj::mv(orphan));