Hot questions for Using Neural networks in android

Question:

I'm working on and android app that can digitize numbers from a paper. I use native OpenCV code to find the numbers on the image. After that I want to use OpenCV's dnn module to recognise the number. A nice tutorial on creating the neural net can be found here: https://www.youtube.com/watch?v=kFWKdLOxykE The mnist_convnet_graph.pbtxt begins with this:

node {
  name: "conv2d_1_input"
  op: "Placeholder"
  attr {
    key: "dtype"
    value {
    type: DT_FLOAT
  }
  }
  attr {
    key: "shape"
    value {
      shape {
        dim {
          size: -1
        }
        dim {
          size: 28
        }
        dim {
          size: 28
        }
        dim {
          size: 1
        }
      }
    }
  }
}

So the input is a 28x28 grayscale image. In the tutorial java code is used to use the neural net. However, I would like to use it in C++, because of the speed. I successfully load the model with cv::dnn::Net Dnn.readNetFromTensorflow(String model, String config); and pass the object to the NDK side. I create the input for the neural net with the following:

// The part of the image, we are interested in.
Rect roi(static_cast<int>(w), static_cast<int>(h),
             static_cast<int>(w), static_cast<int>(h));
Mat cropped(image_gray, roi);
// Resize image to 28x28.
Mat resized;
cv::resize(cropped, resized, Size(28,28));

After that, the forwarding should work:

const double IN_SCALE_FACTOR = 0.003921; // 1.0/255.0
Mat blob = dnn::blobFromImage(resized, IN_SCALE_FACTOR, Size(28,28));
net.setInput(blob);
Mat detections = net.forward();

where net is the passed cv::dnn::Net object. But the net.forward() command fails and gives: OpenCV(3.4.5) Error: Assertion failed (inputs.size() == requiredOutputs) in virtual bool cv::dnn::experimental_dnn_34_v11::DataLayer::getMemoryShapes(const std::vector >&, int, std::vector >&, std::vector >&) const, file /build/3_4_pack-android/opencv/modules/dnn/src/dnn.cpp, line 681

I also tried:

  • cropping the rgb image
  • Mat blob = dnn::blobFromImage(resized, 1.0f, Size(28,28));
  • Not using blobFromImage, but net.setInput(resized); instead

but none of these led to the solutions. Anyone has a solutions for this? Any suggestion or idea will be appreciated.


Answer:

Okay, I managed to solve my problem. Firstly, I realized that the .pb and .pbtxt files are in the wrong directory and getting 2 Failed to upload a file information log. After putting the files into the right directory, I faced with problem: error: (-215:Assertion failed) const_layers.insert(std::make_pair(name, li)).second in function 'void cv::dnn::experimental_dnn_34_v11::{anonymous}::addConstNodes(opencv_tensorflow::GraphDef&, std::map<cv::String, int>&, std::set<cv::String>&)' As Dmitry Kurtaev suggested here, I removed the .pbtxt from the Dnn.readNetFromTensorflow. After that I got error: OpenCV(3.4.5) Error: Unspecified error (Can't create layer "flatten_1/Shape" of type "Shape") in cv::Ptr<cv::dnn::experimental_dnn_34_v11::Layer> cv::dnn::experimental_dnn_34_v11::LayerData::getLayerInstance(), file /build/3_4_pack-android/opencv/modules/dnn/src/dnn.cpp, line 513

This led me to a link, which I found in Dmitry Kurtaev's comment here. After doing the suggested modifications (removing Const nodes, modifying and removing flatten nodes) on the .pbtxt file, finally I got no errors and got successfully running the neural net.

Note: Adding K.backend.set_learning_phase(0) before creating the model, also can be useful.

Question:

I'm very new to Neural Network's, but for a Project of mine they seem to fit. The application should run on a Android phone in the end. My idea is to use TenserFlow, but I'm not sure if its a fit.

I have following Situation, My Input is a Set of Images (the order of them should not have any impact on the output). The Set size is not fixed, but in most cases lower then 10. My output for the whole set is just a binary categorisation (Pass/Fail).

I will have a Convoluted Neural Network, which calculates two outputs, an weight and a pass/fail value. Each Image is supplied seperately to this CNN, the resulting values are then aggregated into a final pass/fail value by using a weighted arithmetic mean.

My Question is, can I train such a network with TensorFlow?

I do not have the values for the CNN outputs in my training data, but only the outputs after the aggregation. Is this possible in general with a gradient oriented Framework or do I have to use a Genetic Algorithm aproach for that.


Answer:

You can definitely do this with tensorflow. After you've done the intro tutorials, you should look at the CNN tutorial to learn how to implement a convolutional neural network in tensorflow. All of the heavy lifting is already taken care of. All you have to do is use the tf.nn.conv2d() method to make the convolutional layer, and then use one of the pooling and normalization ops and so on. If you're unfamiliar with what this means, you should read up on it, but in a nutshell, there are three unique components to a CNN. The convolutional layer scans a window through the image that looks for certain patterns and its activations are recorded in the output as a grid. It's important for learning to lower the dimensionality of the data, and that's what the pooling layer does; it takes the output of the convolutional layer and reduces its size. The normalization layer than normalizes this because having normalized data tends to improve learning.

If you only have the aggregate outputs, then you need to think of a way of coming up with reasonable proxy outputs for individual images. One thing you could do is just use the aggregate output as labels for each individual image in a set and use gradient descent to train.

Question:

Hey I created a neural network using python, this network can recognize handwritten digits. I want to use this in my android app. The android app will take a picture of a hand written digit, and send it to the neural network, the neural network will figure out the digit and send it back to the app. How would I do this? I looked at Google Cloud Platform, but I am confused. I want to know how I will send the picture from the app to my python neural network, and send the output back.


Answer:

Make yourself familiar with the concepts of REST (RestEasy, Jersey,...), create a REST server with an endpoint that can receive a JSON string which contains your base64 picture. The app converts the picture with base64 and sends it to the REST server. Unencode it in your REST server, delegate it to the python script, convert the result back to JSON and send it back to the app. The app itself receives the JSON and gets the data from it.

This is how I would do it with WildFly server and RestEasy:

//app side 
PictureInputBo pictureInputBo = new PictureInputBo([your binary]); //encodes it to base64

//This is working Java code and can be used.
//It will automatically convert to JSON and sent to the "convert" endpoint of the server
ResteasyClient client = new ResteasyClientBuilder().build();
ResteasyWebTarget target = client.target("http://yourserver:8080/your-webservice/rest/convert/");
Response response = target.request().accept(MediaType.APPLICATION_JSON).post(Entity.entity(pictureInputBo, "application/json;charset=UTF-8;version=1"));


//Server side..."Converter Endpoint", this is also working Java code.
//it will automatically converted back to the Java object "PictureInputBo" by RestEasy
@POST
@Path("/convert")
@Consumes(MediaType.APPLICATION_JSON)
public Response convertPicture(@NotNull(message = "Must not be null") @Valid PictureInputBo inputBo)
{
    //Here you pass your business object to the converter service which processes the data
    //(pass it to python or whatever) 
    PictureOutputBo result = converterService.convert(inputBo);

    //Resteasy converts it back to JSON and responds it to the app.
    return Response.ok().entity(result).build();
}


//Back in your app.
check if response.getStatus() == 200) //HTTP Status OK
PictureOutputBo pictureOutputBo = response.readEntity(PictureOutputBo.class);

Question:

Known that I'm very new in Machine learning.

I was thinking about a real world example of using Machine Learning and Neural network in an application and I want to try it with a mobile application who can handle image recognition with the front camera after make an image of something(A cat for exemple).

I really need advice of tools to use to rapidly make a prototype of this application with a python backend that I will call via rest.

Thanks in advance.


Answer:

I suggest if you are new to the machine learning algorithms, that you use an API from Google or Microsoft and get in touch with the flow and how it works .. Once you understand what are the inputs and outputs, you can try to replace the API for you own neural net, try to train it properly and collect results ..

Machine learning is not an easy concept and if you start big, there is a good chance that you'll get discouraged before you finish building it ... The API will provide you with a functional prototype very quickly and thus help you stay motivated to pursue it more ..

But to answer your question more directly, TensorFlow by Google is probably the most sophisticated tool for machine learning in general right now..

There is an excellent course for deep learning with TensorFlow made by Google on Udacity ..