Get loss values for each training instance - Keras

loss vs epoch graph
how to plot accuracy graph in python
keras training loss
get loss of keras model
keras evaluate model
keras get loss from model fit
sequential' object has no attribute 'history
keras callbacks=(history)

I want to get loss values as model train with each instance.

history = model.fit(..)

for example above code returns the loss values for each epoch not mini batch or instance.

what is the best way to do this? Any suggestions?

There is exactly what you are looking for at the end of this official keras documentation page https://keras.io/callbacks/#callback

Here is the code to create a custom callback

class LossHistory(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.losses = []

    def on_batch_end(self, batch, logs={}):
        self.losses.append(logs.get('loss'))

model = Sequential()
model.add(Dense(10, input_dim=784, kernel_initializer='uniform'))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

history = LossHistory()
model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])

print(history.losses)
# outputs
'''
[0.66047596406559383, 0.3547245744908703, ..., 0.25953155204159617, 0.25901699725311789]
'''

Train and evaluate with Keras, Specify the training configuration (optimizer, loss, metrics) The returned "history​" object holds a record of the loss values and metric values during training The metrics argument should be a list -- you model can have any number of metrics. You can pass a Dataset instance directly to the methods fit() , evaluate() , and  Train and evaluate with Keras. of per-batch loss values during training: of the layer with respect to a loss value. Using an optimizer instance,

If you want to get loss values for each batch, you might want to use call model.train_on_batch inside a generator. It's hard to provide a complete example without knowing your dataset, but you will have to break your dataset into batches and feed them one by one

def make_batches(...):
    ...

batches = make_batches(...)
batch_losses = [model.train_on_batch(x, y) for x, y in batches]

It's a bit more complicated with single instances. You can, of course, train on 1-sized batches, though it will most likely thrash your optimiser (by maximising gradient variance) and significantly degrade performance. Besides, since loss functions are evaluated outside of Python's domain, there is no direct way to hijack the computation without tinkering with C/C++ and CUDA sources. Even then, the backend itself evaluates the loss batch-wise (benefitting from highly vectorised matrix-operations), therefore you will severely degrade performance by forcing it to evaluate loss on each instance. Simply put, hacking the backend will only (probably) help you reduce GPU memory transfers (as compared to training on 1-sized batches from the Python interface). If you really want to get per-instance scores, I would recommend you to train on batches and evaluate on instances (this way you will avoid issues with high variance and reduce expensive gradient computations, since gradients are only estimated during training):

def make_batches(batchsize, x, y):
    ...


batchsize = n
batches = make_batches(n, ...)
batch_instances = [make_batches(1, x, y) for x, y in batches]
losses = [
    (model.train_on_batch(x, y), [model.test_on_batch(*inst) for inst in instances]) 
    for batch, instances in zip(batches, batch_instances)
]

computing loss for each individual instance · Issue #4347 · keras , function in engine/trainining.py generates average loss and accuracy I am trying to tweak keras to generate the loss for each individual training instance Any hints on how to get the loss value for each individual instance? I used Keras history to save ‘loss’ and ‘val_loss’ for each model and selected the loss and validation loss for minimum in the validation loss, to avoid overfitting. When I plot the loss, I get roughly a minimum for the 5 models with batch size 1024, but when I plot the validation loss there is no minimum.

After combining resources from here and here I came up with the following code. Maybe it will help you. The idea is that you can override the Callbacks class from keras and then use the on_batch_end method to check the loss value from the logs that keras will supply automatically to that method.

Here is a working code of an NN with that particular function built in. Maybe you can start from here -

import numpy as np
import pandas as pd
import seaborn as sns
import os
import matplotlib.pyplot as plt
import time
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.callbacks import Callback

# fix random seed for reproducibility
seed = 155
np.random.seed(seed)

# load pima indians dataset

# download directly from website
dataset = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data", 
                      header=None).values
X_train, X_test, Y_train, Y_test = train_test_split(dataset[:,0:8], dataset[:,8], test_size=0.25, random_state=87)
class NBatchLogger(Callback):
    def __init__(self,display=100):
        '''
        display: Number of batches to wait before outputting loss
        '''
        self.seen = 0
        self.display = display

    def on_batch_end(self,batch,logs={}):
        self.seen += logs.get('size', 0)
        if self.seen % self.display == 0:
            print('\n{0}/{1} - Batch Loss: {2}'.format(self.seen,self.params['samples'],
                                                logs.get('loss')))


out_batch = NBatchLogger(display=1000)
np.random.seed(seed)
my_first_nn = Sequential() # create model
my_first_nn.add(Dense(5, input_dim=8, activation='relu')) # hidden layer
my_first_nn.add(Dense(1, activation='sigmoid')) # output layer
my_first_nn.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

my_first_nn_fitted = my_first_nn.fit(X_train, Y_train, epochs=1000, verbose=0, batch_size=128,
                                     callbacks=[out_batch], initial_epoch=0)

Please let me know if you wanted to have something like this.

Display Deep Learning Model Training History in Keras, Do you have any questions about model training history or about this post? For instance: 1 epoch, 60,000MNIST images => plot 60,000 I be plotting the log value of this? as all the graphs I see the loss is between 0-1. If sample_weight is a tensor of size [batch_size], then the total loss for each sample of the batch is rescaled by the corresponding element in the sample_weight vector. If the shape of sample_weight matches the shape of y_pred, then the loss of each measurable element of y_pred is scaled by the corresponding value of sample_weight.

One solution is to calculate the loss function between train expectations and predictions from train input. In the case of loss = mean_squared_error and three dimensional outputs (i.e. image width x height x channels):

model.fit(train_in,train_out,...)

pred = model.predict(train_in)
loss = np.add.reduce(np.square(test_out-pred),axis=(1,2,3)) # this computes the total squared error for each sample
loss = loss / ( pred.shape[1]*pred.shape[2]*pred.shape[3]) # this computes the mean over the sample entry 

np.savetxt("loss.txt",loss) # This line saves the data to file

Losses, By default, loss functions return one scalar loss value per input sample, e.g. Here's how you would use a loss class instance as part of a simple training loop:. The idea is that you can override the Callbacks class from keras and then use the on_batch_end method to check the loss value from the logs that keras will supply automatically to that method. Here is a working code of an NN with that particular function built in.

Frequently Asked Questions • keras, If you have the GPU version of CNTK installed then your Keras code will (with some layers in common), for instance for fine-tuning or transfer-learning, Besides, the training loss is the average of the losses over each batch of training data. I used theano as backend, and the loss function is binary_crossentropy, during the training, the acc, val_acc, loss, and val_loss never changed in every epoch, and loss value is very high , about 8. I used 4000 training samples 1000 validation samples

Is Keras evaluation of Training loss only an approximation? If so , validation accuracy, validation loss and training accuracy or any other metric depending on the This callback is automatically applied to every Keras model. """ Make value available to next callbacks. If you have 105 samples and a batch size of 20, then you can either make 5 batches of 20 samples  The loss value that will be minimized by the model will then be the weighted sum of all individual losses, weighted by the loss_weights coefficients. If a list, it is expected to have a 1:1 mapping to the model's outputs. If a tensor, it is expected to map output names (strings) to scalar coefficients.

Keras: Starting, stopping, and resuming training, You have limited time on a GPU instance (which can happen on Each time we encounter a characteristic drop in loss and then a small  Now let’s implement a custom loss function for our Keras model. As a first step, we need to define our Keras model. Our model instance name is keras_model, and we’re using Keras’s sequential() function to create the model. We’ve included three layers, all dense layers with shape 64, 64, and 1. We have an input shape of 1, and we’re

Comments
  • I want the loss values after training model on each instance. So if I am correct you are suggesting I feed 1 instance at a time as a batch? I was wondering if there is a more efficient way to get the loss information.
  • Callback interface does not seems to have a operation to get loss by each instance. on_batch_end only return loss at the end of each mini batch. What I want to do is get deeper and get loss after each training instance. Any suggestions?
  • Thanks @SRC. This works! I only need to set batch size to 1.
  • @e.hurrington you shouldn't do it. batchsize affects training performance badly (even if you don't consider training time), i.p. your model might fail to converge on 1-sized batches.
  • @SRC thanks. I do not mind the training time. But I need it to converge. I will look at some alternatives. Please share if you have any. Thanks again.
  • @e.hurrington I've given you an alternative in my edit.