Hot questions for Using Neural networks in recursion

Question:

I ran a Neural Network model for MNIST classification and received error-

RecursionError: maximum recursion depth exceeded

I checked some of the issues on stackoverflow and tried to increase the recursion limit to 1500 but did not work. How should I increase the limit? An how do I know what limit will cause stack overflow?

I followed the tutorial from here

I have Anaconda 3.5 distribution on my windows 10 machine.

The full code is here-

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist= input_data.read_data_sets("/tmp/data/", one_hot=True)

n_nodes_hl1 = 500
n_nodes_hl2 = 500
n_nodes_hl3 =500

n_classes = 10
batch_size = 100

#height x weight
x = tf.placeholder('float', [None, 784])
y = tf.placeholder('float')

def neural_network_model(data):

    hidden_1_layer= {'weights': tf.Variable(tf.random_normal([784, n_nodes_hl1])),
                 'biases': tf.Variable(tf.random_normal([n_nodes_hl1]))
                 }
    hidden_2_layer = {'weights': tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])),
                  'biases': tf.Variable(tf.random_normal([n_nodes_hl2]))
                  }
    hidden_3_layer = {'weights': tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])),
                  'biases': tf.Variable(tf.random_normal([n_nodes_hl3]))
                  }
    output_layer = {'weights': tf.Variable(tf.random_normal([n_nodes_hl3,n_classes])),
                  'biases': tf.Variable(tf.random_normal([n_classes]))
                }

#our model= (input_data x weights) + biases

    l1 = tf.add(tf.matmul(data, hidden_1_layer['weights']), hidden_1_layer['biases'])
    l1 = tf.nn.relu(l1)

    l2 = tf.add(tf.matmul(l1, hidden_2_layer['weights']), hidden_2_layer['biases'])
    l2 = tf.nn.relu(l2)

    l3 = tf.add(tf.matmul(l2, hidden_3_layer['weights']), hidden_3_layer['biases'])
    l3 = tf.nn.relu(l3)

    output = tf.matmul(l3, output_layer['weights']) + output_layer['biases']

    return output


def train_neural_network(x):
    prediction = train_neural_network(x)
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(prediction,y))
    optimizer= tf.train.AdamOptimizer().minimize(cost) #default learning rate for adamoptimizer= 0.001

    hm_epochs = 5
    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        for epoch in range(hm_epochs):
            epoch_loss = 0
            for _ in range(int(mnist.train.num_examples / batch_size)):
                epoch_x, epoch_y = mnist.train.next_batch(batch_size)
                _, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y})
                epoch_loss += c

    print(('Epoch', epoch), ('completed out of', hm_epochs), ('loss:', epoch_loss))

    correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
    print(('Accuracy:', accuracy.eval({x: mnist.test.images, y: mnist.test.labels})))

train_neural_network(x)

Answer:

I don't know what the exact code is supposed to be, but I'm quite sure the following lines are wrong:

def train_neural_network(x):
    prediction = train_neural_network(x)

This will cause an infinite recursion, and increasing the recursion limit will not solve the problem.

Question:

Let's say I built a nested model like this:

from keras.models import Sequential, Model
from keras.layers.core import Input, Dense

model_1 = Sequential()
model_1.add(Dense(...))
model_1.add(Dense(...))

input_2 = Input(...)
output_2 = Dense(...)(input_2)
model_2 = Model(inputs=input_2, outputs=output_2) 

model = Sequential()
model.add(model_1)
model.add(model_2)

How can I transform this recursively into a "flat" model, that does not contain any Model or Sequential layers.

Since model_1 and model_2 might have been trained in advance the parameters should be conserved during the transformation.


Answer:

I had a similar problem, and I got a working solution, but this doesn't seem very elegant.

The basic idea is to iterate through the layers of the sub-models and add them to the overall model one by one rather than adding the entire sub-models.

model = Sequential()

for layer1 in model1.layers:
    model.add(layer1)

for layer2 in model2.layers:
    model.add(layer2)

If the model already includes nested models, it is possible to iterate over them via:

model_flat = Sequential()

for layer_nested in model.get_layer('nested_model').layers:
    model_flat.add(layer_nested)