Hot questions for Using Cap'n Proto in java

Question:

I am using a TCP Client/Server to send Cap'n Proto messages from C++ to Java.

Sometimes the receiving buffer may be overfilled or underfilled and to handle these cases we need to know the message size.

When I check the size of the buffer in Java I get 208 bytes, however calling

MyModel.MyMessage.STRUCT_SIZE.total()

returns 4 (not sure what unit of measure is being used here).

I notice that 4 divides into 208, 52 times. But I don't know of a significant conversion factor using 52.

How do I check the message size in Java?


Answer:

MyMessage.STRUCT_SIZE represents the constant size of that struct itself (measured in 8-byte words), but if the struct contains non-trivial fields (like Text, Data, List, or other structs) then those take up space too, and the amount of space they take is not constant (e.g. Text will take space according to how long the string is).

Generally you should try to let Cap'n Proto directly write to / read from the appropriate ByteChannels, so that you don't have to keep track of sizes yourself. However, if you really must compute the size of a message ahead of time, you could do so with something like:

ByteBuffer[] segments = message.getSegmentsForOutput();
int total = (segments.length / 2 + 1) * 8;  // segment table
for (ByteBuffer segment: segments) {
  total += segment.remaining();
}
// now `total` is the total number of bytes that will be
// written when the message is serialized.

On the C++ size, you can use capnp::computeSerializedSizeInWords() from serialize.h (and multiply by 8).

But again, you really should structure your code to avoid this, by using the methods of org.capnproto.Serialize with streaming I/O.

Question:

I have a project requiring the use of Cap'n Proto for Java. I have a Linux system and I've successfully installed the Cap'n Proto schema compiler as described here:

https://capnproto.org/install.html

Now following the installation instructions as described here: https://dwrensha.github.io/capnproto-java/index.html, I can't seem to figure out how to generate the capnpc-java plugin. The instructions here seem unclear:

You will need to install the latest release of the Cap’n Proto schema compiler. Then, running make should build capnpc-java.

Running make from where? I did that from the $WORKDIR/capnproto-java directory but that doesn't work.

The other approach I did was to follow the cmake instructions from from $WORKDIR/capnproto-java/cmake/README.md but that didn't work either. According to that README.md:

mkdir build cd build cmake -DCAPNP_PKG_PATH=[path of Capnproto pkgconfig directory (with capnp.pc)] [path of CMakeLists.txt]

Where I set CAPNP_PKG_PATH to be the path of the $WORKDIR/capnproto/c++/pkgconfig directory, and the [path of CMakeLists.txt] to $WORKDIR/capnproto, where $WORKDIR is a directory on my system.

Can anyone help? I'm not an expert on make system, Makefiles, or cmake. Where does the capnpc-java created? When I do a which capnpc-java, nothing is showing up (I do however, can successfully which capnp, which is located in /usr/local/bin/)


Answer:

Looks like I figured it out after some trial and error. It's not straightforward from the README, but at the end of the day, I got the capnpc-java built. This is the approach I did:

  1. Per https://capnproto.org/install.html, follow the instructions From Git
  2. Go to $WORKDIR/capnproto-java/cmake. Follow the instructions in the README.md inside that cmake directory, under Using cmake.
  3. When specifying -DCAPNP_PKG_PATH, specify the path from Step 1, but make sure to include capnp.pc. For example: cmake -DCAPNP_PKG_PATH=$WORKDIR/capnproto/c++/pkgconfig/capnp.pc $WORKDIR/capnproto/CMakeLists.txt
  4. cd to $WORKDIR/capnproto and run make -j6
  5. cd to $WORKDIR/capnproto-java and run make
  6. capnpc-java is generated in $WORKDIR/capnproto-java. Copy this to /usr/local/bin

I'm assuming this works. I haven't run the capnp compile yet, but at least this answered my original question.

Question:

In Java Client I get a

com.google.protobuf.ByteString capn_object_bytes =   response.getCapnObject(); 

from C++ Server, and I want read the Capn Object from protobuf.ByteString

@Kenton Varda


Answer:

Use ByteString#asReadOnlyByteBuffer() to get a ByteBuffer, which you can then read into Cap'n Proto:

MessageReader message =
    org.capnproto.Serialize.read(capn_object_bytes.asReadOnlyByteBuffer());