Hot questions for Using Neural networks in tensorflow lite

Question:

I'm looking at the Tensorflow tf.nn.quantized_conv2d function and I'm wondering what exactly the qint8, etc. dataypes are, particularly if they are the datatypes used for the "fake quantization nodes" in tf.contrib.quantize or are actually stored using 8 bits (for qint8) in memory.

I know that they are defined in tf.dtypes.DType, but that doesn't have any information about what they actually are.


Answer:

These are the data types of the output Tensor of the function, tf.quantization.quantize(). This corresponds to the Argument, T of the function.

Mentioned below is the underlying code, which converts/quantizes a Tensor from one Data Type (e.g. float32) to another (tf.qint8, tf.quint8, tf.qint32, tf.qint16, tf.quint16).

out[i] = (in[i] - min_range) * range(T) / (max_range - min_range)
if T == qint8: out[i] -= (range(T) + 1) / 2.0

Then, they can be passed to functions like tf.nn.quantized_conv2d, etc.., whose input is a Quantized Tensor, explained above.

TLDR, to answer your question in short, they are actually stored 8 bits (for qint8) in memory.

You can find more information about this topic in the below links:

https://www.tensorflow.org/api_docs/python/tf/quantization/quantize

https://www.tensorflow.org/api_docs/python/tf/nn/quantized_conv2d

https://www.tensorflow.org/lite/performance/post_training_quantization

If you feel this answer is useful, kindly accept this answer and/or up vote it. Thanks.

Question:

The following code snippet creates an 8-bit quantized TF Lite model, and replacing QUANTIZED_UINT8 with FLOAT creates a 32-bit model. Is there any flag that creates a 16-bit quantized model? I've searched the TF Lite documentation but I couldn't find any documentation on the list of possible flags. Does anyone know how to do this?

~/tensorflow/bazel-bin/tensorflow/contrib/lite/toco/toco \
  --input_file=$(pwd)/model.pb \
  --input_format=TENSORFLOW_GRAPHDEF \
  --output_format=TFLITE \
  --output_file=$(pwd)/model.lite --inference_type=QUANTIZED_UINT8 \
  --input_type=QUANTIZED_UINT8 --input_arrays=conv2d_1_input \
  --default_ranges_min=0.0 --default_ranges_max=1.0 \
  --output_arrays=average_pooling2d_2/AvgPool --input_shapes=1024,32,32,2

Answer:

Currently, the only quantized type that TFLite supports in 8 bits. See here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/toco/types.proto#L27

This is because for existing quantized models, 8 bits was found sufficient, but this may change. If you have a model that needs more bits for quantization, it may be worthwhile to create an tensorflow issue describing your use case.

Question:

I tried use Tensorflow Lite, but it has lots of limitations, it doesn't have batch normalization operation, and even with simple operations it gave a very strange result to the same data tested with Keras. It means with keras everything works, with tensorflow lite, the result is completely wrong. So I need something to execute the .pb file on Android.


Answer:

You can use the TensorFlowInferenceInterface to make predictions using a .pb file. First, place the .pb file in your app's assets folder.

  1. In your build.gradle(Module: app) file, add the following dependency, implementation 'org.tensorflow:tensorflow-android:1.11.0'
  2. Then initialize TensorFlowInferenceInterface, if your model files name is "model.pb" then , TensorFlowInferenceInterface tensorFlowInferenceInterface = new TensorFlowInferenceInterface(context.getAssets() , "file:///android_asset/model.pb") ;
  3. tensorFlowInferenceInterface.feed( INPUT_NAME , inputs , 1, 28, 28); where INPUT_NAME is the name of your input layer. 1 , 50 are the input dimensions.

  4. tensorFlowInferenceInterface.run( new String[]{ OUTPUT_NAME } ); where OUTPUT_NAME is the name of your output layer.

  5. float[] outputs = new float[ nuymber_of_classes ]; tensorFlowInferenceInterface.fetch( OUTPUT_NAME , outputs ) ;

outputs are the float values which are predicted from your model.

Here's the full code :

TensorFlowInferenceInterface tensorFlowInferenceInterface = new 
TensorFlowInferenceInterface(context.getAssets() , "file:///android_asset/model.pb");
tensorFlowInferenceInterface.feed( INPUT_NAME , inputs , 1, 28, 28);
tensorFlowInferenceInterface.run( new String[]{ OUTPUT_NAME } );
float[] outputs = new float[ nuymber_of_classes ];
tensorFlowInferenceInterface.fetch( OUTPUT_NAME , outputs ) ;