## Hot questions for Using Neural networks in tensorflow estimator

Question:

I'm using `tf.estimator`

in TensorFlow 1.4 and `tf.estimator.train_and_evaluate`

is great but I need early stopping. What's the prefered way of adding that?

I assume there is some `tf.train.SessionRunHook`

somewhere for this. I saw that there was an old contrib package with a `ValidationMonitor`

that seemed to have early stopping, but it doesn't seem to be around anymore in 1.4. Or will the preferred way in the future be to rely on `tf.keras`

(with which early stopping is really easy) instead of `tf.estimator/tf.layers/tf.data`

, perhaps?

Answer:

Good news! `tf.estimator`

now has early stopping support on master and it looks like it will be in 1.10.

estimator = tf.estimator.Estimator(model_fn, model_dir) os.makedirs(estimator.eval_dir()) # TODO This should not be expected IMO. early_stopping = tf.contrib.estimator.stop_if_no_decrease_hook( estimator, metric_name='loss', max_steps_without_decrease=1000, min_steps=100) tf.estimator.train_and_evaluate( estimator, train_spec=tf.estimator.TrainSpec(train_input_fn, hooks=[early_stopping]), eval_spec=tf.estimator.EvalSpec(eval_input_fn))

Question:

I am attempting to use TensorFlow's Estimators. In the documentation the following code is used to train and evaluate the network.

# Fit nn.fit(x=training_set.data, y=training_set.target, steps=5000) # Score accuracy ev = nn.evaluate(x=test_set.data, y=test_set.target, steps=1) loss_score = ev["loss"] print("Loss: %s" % loss_score)

The whole training set is passed in, but we have `steps=5000`

. Does this mean that only the first 5000 examples from the set are considered?

What does the `batch_size`

parameter mean in this context, and how does it interact with `steps`

?

Thanks!

Answer:

**batch_size** is the number of examples processed at once. TF pushes all of those through one forward pass (in parallel) and follows with a back-propagation on the same set. This is one *iteration*, or *step*.

The **steps** parameter tells TF to run 5000 of these iterations to train the model.

One **epoch** is treating each example in the training set exactly once. For instance, if you have one million examples and a batch size of 200, then you need 5000 steps to one epoch: 200 * 5.000 = 1.000.000

Does that clear up the terminology?

Question:

(Complete novice at python, machine learning, and TensorFlow)

I am attempting to adapt the TensorFlow Linear Model Tutorial from their offical documentation to the Abalone dataset featured on the ICU machine learning repository. The intent is to guess the rings(age) of an abalone from the other given data.

When running the below program I get the following:

File "/home/lawrence/tensorflow3.5/lib/python3.5/site-packages/tensorflow /python/ops/lookup_ops.py", line 220, in lookup (self._key_dtype, keys.dtype)) TypeError: Signature mismatch. Keys must be dtype <dtype: 'string'>, got <dtype: 'int32'>.

The error is being thrown in lookup_ops.py at line 220 and is documented as being thrown when:

Raises: TypeError: when `keys` or `default_value` doesn't match the table data types.

From debugging `parse_csv()`

it seems to be the case that all the tensors are created with the correct type.

Could you please explain what is going wrong? I believe I am following the tutorial code logic and cannot figure this out.

Source Code:

import tensorflow as tf import shutil _CSV_COLUMNS = [ 'sex', 'length', 'diameter', 'height', 'whole_weight', 'shucked_weight', 'viscera_weight', 'shell_weight', 'rings' ] _CSV_COLUMN_DEFAULTS = [['M'], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0]] _NUM_EXAMPLES = { 'train': 3000, 'validation': 1177, } def build_model_columns(): """Builds a set of wide feature columns.""" # Continuous columns sex = tf.feature_column.categorical_column_with_hash_bucket('sex', hash_bucket_size=1000) length = tf.feature_column.numeric_column('length', dtype=tf.float32) diameter = tf.feature_column.numeric_column('diameter', dtype=tf.float32) height = tf.feature_column.numeric_column('height', dtype=tf.float32) whole_weight = tf.feature_column.numeric_column('whole_weight', dtype=tf.float32) shucked_weight = tf.feature_column.numeric_column('shucked_weight', dtype=tf.float32) viscera_weight = tf.feature_column.numeric_column('viscera_weight', dtype=tf.float32) shell_weight = tf.feature_column.numeric_column('shell_weight', dtype=tf.float32) base_columns = [sex, length, diameter, height, whole_weight, shucked_weight, viscera_weight, shell_weight] return base_columns def build_estimator(): """Build an estimator appropriate for the given model type.""" base_columns = build_model_columns() return tf.estimator.LinearClassifier( model_dir="~/models/albones/", feature_columns=base_columns, label_vocabulary=_CSV_COLUMNS) def input_fn(data_file, num_epochs, shuffle, batch_size): """Generate an input function for the Estimator.""" assert tf.gfile.Exists(data_file), ( '%s not found. Please make sure you have either run data_download.py or ' 'set both arguments --train_data and --test_data.' % data_file) def parse_csv(value): print('Parsing', data_file) columns = tf.decode_csv(value, record_defaults=_CSV_COLUMN_DEFAULTS) features = dict(zip(_CSV_COLUMNS, columns)) labels = features.pop('rings') return features, labels # Extract lines from input files using the Dataset API. dataset = tf.data.TextLineDataset(data_file) if shuffle: dataset = dataset.shuffle(buffer_size=_NUM_EXAMPLES['train']) dataset = dataset.map(parse_csv) # We call repeat after shuffling, rather than before, to prevent separate # epochs from blending together. dataset = dataset.repeat(num_epochs) dataset = dataset.batch(batch_size) iterator = dataset.make_one_shot_iterator() features, labels = iterator.get_next() return features, labels def main(unused_argv): # Clean up the model directory if present shutil.rmtree("/home/lawrence/models/albones/", ignore_errors=True) model = build_estimator() # Train and evaluate the model every `FLAGS.epochs_per_eval` epochs. for n in range(40 // 2): model.train(input_fn=lambda: input_fn( "/home/lawrence/abalone.data", 2, True, 40)) results = model.evaluate(input_fn=lambda: input_fn( "/home/lawrence/abalone.data", 1, False, 40)) # Display evaluation metrics print('Results at epoch', (n + 1) * 2) print('-' * 60) for key in sorted(results): print('%s: %s' % (key, results[key])) if __name__ == '__main__': tf.logging.set_verbosity(tf.logging.INFO) tf.app.run(main=main)

Here is the classification of the columns of the dataset from abalone.names:

Name Data Type Meas. Description ---- --------- ----- ----------- Sex nominal M, F, [or] I (infant) Length continuous mm Longest shell measurement Diameter continuous mm perpendicular to length Height continuous mm with meat in shell Whole weight continuous grams whole abalone Shucked weight continuous grams weight of meat Viscera weight continuous grams gut weight (after bleeding) Shell weight continuous grams after being dried Rings integer +1.5 gives the age in years

Dataset entries appear in this order as common separated values with a new line for a new entry.

Answer:

You've done almost everything right. The problem is with the definition of an estimator.

The task is to predict the `Rings`

column, which is an integer, so it looks like a *regression* problem. But you've decided to do a *classification* task, which is also valid:

def build_estimator(): """Build an estimator appropriate for the given model type.""" base_columns = build_model_columns() return tf.estimator.LinearClassifier( model_dir="~/models/albones/", feature_columns=base_columns, label_vocabulary=_CSV_COLUMNS)

By default, `tf.estimator.LinearClassifier`

assumes binary classification, i.e., `n_classes=2`

. In your case, it's obviously not true - that's the first bug. You've also set `label_vocabulary`

, which tensorflow interprets as a set of possible values in the label column. That's why it expects `tf.string`

dtype. Since `Rings`

is an integer, you simply don't need `label_vocabulary`

at all.

Combining it all together:

def build_estimator(): """Build an estimator appropriate for the given model type.""" base_columns = build_model_columns() return tf.estimator.LinearClassifier( model_dir="~/models/albones/", feature_columns=base_columns, n_classes=30)

I suggest you also try `tf.estimator.LinearRegressor`

, which will probably be more accurate.

Question:

I'm trying to create a simple one-layer/one-unit nn with tensorflow custom estimators that will be able to compute logical AND operation, but I've got a trouble with sigmoid activation -- I want to set threshold

here is my code

x = np.array([ [0, 0], [1, 0], [0, 1], [1, 1] ], dtype=np.float32) y = np.array([ [0], [0], [0], [1] ]) def sigmoid(val): res = tf.nn.sigmoid(val) isGreater = tf.greater(res, tf.constant(0.5)) return tf.cast(isGreater, dtype=tf.float32) def model_fn(features, labels, mode, params): predictions = tf.layers.dense(inputs=features, units=1, activation=sigmoid) if mode == tf.estimator.ModeKeys.PREDICT: return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions) loss = tf.losses.sigmoid_cross_entropy(labels, predictions) optimizer = tf.train.GradientDescentOptimizer(0.5) train_op = optimizer.minimize(loss, global_step=tf.train.get_global_step()) return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op) nn = tf.estimator.Estimator(model_fn=model_fn) input_fn = tf.estimator.inputs.numpy_input_fn(x=x, y=y, shuffle=False, num_epochs=None) nn.train(input_fn=input_fn, steps=500)

BUT this throws an error

ValueError: No gradients provided for any variable, check your graph for ops that do not support gradients, between variables ["<tf.Variable 'dense/kernel:0' shape=(2, 1) dtype=float32_ref>", "<tf.Variable 'dense/bias:0' shape=(1,) dtype=float32_ref>"] and loss Tensor("sigmoid_cross_entropy_loss/value:0", shape=(), dtype=float32).

How can I fix this? Please help..

Another question I've got - why Tensorflow does not have inbuilt threshold for sigmoid activation? Isn't it one of the most needed things for binary classification (with sigmoid/tanh)?

Answer:

There is a built-in sigmoid activation, which is `tf.nn.sigmoid`

.

However when you create a network you should never use an activation on the last layer. You need to provide unscaled logits to the layer, like this:

predictions = tf.layers.dense(inputs=features, units=1, activation=None) loss = tf.losses.sigmoid_cross_entropy(labels, predictions)

Otherwise, with your custom sigmoid your predictions will be either `0`

or `1`

and there is no gradient available for this.

Question:

Im doing a neural network in tensorflow and Im using softmax_cross_entropy to calculate the loss, I'm doing tests and note that it never gives a value of zero, even if I compare the same values, this is my code

labels=[1,0,1,1] with tf.Session() as sess: onehot_labels=tf.one_hot(indices=labels,depth=2) logits=[[0.,1.],[1.,0.],[0.,1.],[0.,1.]] print(sess.run(onehot_labels)) loss=tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels,logits=logits) print(sess.run(loss))

I obtain this

[[0. 1.] [1. 0.] [0. 1.] [0. 1.]] 0.31326166

Why is not zero??

Answer:

Matias's post is correct. The following code gives the same result as your code

labels=[1,0,1,1] with tf.Session() as sess: onehot_labels=tf.one_hot(indices=labels,depth=2) logits=[[0.,1.],[1.,0.],[0.,1.],[0.,1.]] print(sess.run(onehot_labels)) probabilities = tf.nn.softmax(logits=logits) # cross entropy loss = -tf.reduce_sum(onehot_labels * tf.log(probabilities)) / len(labels) print(sess.run(loss))