Hot questions for Using Neural networks in pickle

Question:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

dataset = pd.read_csv("Churn_Modelling.csv")
X = dataset.iloc[:,3:13].values
Y = dataset.iloc[:,13:].values

from sklearn.preprocessing import OneHotEncoder,LabelEncoder,StandardScaler

enc1=LabelEncoder()
enc2=LabelEncoder()
X[:,1] = enc1.fit_transform(X[:,1])
X[:,2] = enc2.fit_transform(X[:,2])

one = OneHotEncoder(categorical_features=[1])
X=one.fit_transform(X).toarray()

X = X[:,1:]

from sklearn.model_selection import train_test_split
Xtrain,Xtest,Ytrain,Ytest = train_test_split(X,Y,random_state=0,test_size=0.2)

scale = StandardScaler()
scale.fit_transform(Xtrain)
scale.transform(Xtest)

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score 
from keras.models import Sequential
from keras.layers import Dense

def func1():
    net = Sequential()
    net.add(Dense(input_dim=11,units=6,activation="relu",kernel_initializer='uniform'))
    net.add(Dense(units=6,activation="relu",kernel_initializer='uniform'))
    net.add(Dense(units=1,activation="sigmoid",kernel_initializer='uniform'))
    net.compile(optimizer='adam',metrics=['accuracy'],loss='binary_crossentropy')

    return net

classfier = KerasClassifier(build_fn=func1(),batch_size=10, epochs=100)
cross = cross_val_score(estimator=classfier, X=Xtrain, y=Ytrain, cv=10 , n_jobs=-1)

Throws the error:

Traceback (most recent call last):

  File "<ipython-input-7-e80e82960eb9>", line 1, in <module>
    cross = cross_val_score(estimator=classfier, X=Xtrain, y=Ytrain, cv=10 , n_jobs=-1)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\site-packages\sklearn\model_selection\_validation.py", line 342, in cross_val_score
    pre_dispatch=pre_dispatch)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\site-packages\sklearn\model_selection\_validation.py", line 206, in cross_validate
    for train, test in cv.split(X, y, groups))

  File "C:\Users\Joish\Anaconda3\envs\project\lib\site-packages\sklearn\externals\joblib\parallel.py", line 779, in __call__
    while self.dispatch_one_batch(iterator):

  File "C:\Users\Joish\Anaconda3\envs\project\lib\site-packages\sklearn\externals\joblib\parallel.py", line 620, in dispatch_one_batch
    tasks = BatchedCalls(itertools.islice(iterator, batch_size))

  File "C:\Users\Joish\Anaconda3\envs\project\lib\site-packages\sklearn\externals\joblib\parallel.py", line 127, in __init__
    self.items = list(iterator_slice)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\site-packages\sklearn\model_selection\_validation.py", line 206, in <genexpr>
    for train, test in cv.split(X, y, groups))

  File "C:\Users\Joish\Anaconda3\envs\project\lib\site-packages\sklearn\base.py", line 62, in clone
    new_object_params[name] = clone(param, safe=False)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\site-packages\sklearn\base.py", line 53, in clone
    return copy.deepcopy(estimator)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "C:\Users\Joish\Anaconda3\envs\project\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)

TypeError: can't pickle _thread.lock objects

How do I solve this?


Answer:

Change this line:

classfier = KerasClassifier(build_fn=func1, batch_size=10, epochs=100, verbose=0)

Note that func1 is not called. From the documentation:

build_fn: callable function or class instance

The build_fn should construct, compile and return a Keras model, which will then be used to fit/predict. One of the following three values could be passed to build_fn:

  1. A function

  2. An instance of a class that implements the __call__ method

  3. None. This means you implement a class that inherits from either KerasClassifier or KerasRegressor. The __call__ method of the present class will then be treated as the default build_fn.

Question:

The following code is throwing an error: TypeError: can't pickle _thread.lock objects

I can see that it likely has to do with passing the previous method in as a function in def fit(self, c_m). But I think this is correct via the documentations: https://keras.io/scikit-learn-api/

I may be making a rookie mistake if anyone sees the error in my code I would appreciate help.

np.random.seed(7)
y_dic = []

class NN:
    def __init__(self):
        self.X = None
        self.y = None
        self.model = None

    def clean_data(self):
        seed = 7
        np.random.seed(seed)
        dataset = pd.read_csv('/Users/isaac/pca_rfe_tsne_comparisons/Vital_intrusions.csv', delimiter=',', skiprows=0)
        dataset = dataset.iloc[:,1:6]
        self.X = dataset.iloc[:, 1:5]
        Y = dataset.iloc[:, 0]

        for y in Y:
            if y >= 8:
                y_dic.append(1)
            else:
                y_dic.append(0)
        self.y = y_dic

        self.X = np.asmatrix(stats.zscore(self.X, axis=0, ddof=1))
        self.y = to_categorical(self.y)


    def create_model(self):
        self.model = Sequential()
        self.model.add(Dense(4, input_dim=4, activation='relu'))
        self.model.add(Dense(4, activation='relu'))
        self.model.add(Dense(2, activation='sigmoid'))
        self.model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
        pass

    def fit(self, c_m):
        model = KerasClassifier(build_fn=c_m, verbose=0)
        batch_size = [10, 20, 40, 60, 80, 100]
        epochs = [10, 50, 100]
        param_grid = dict(batch_size=batch_size, epochs=epochs)
        grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1)
        pdb.set_trace()
        grid_result = grid.fit(self.X, self.y)
        return (grid_result)

    def results(self, grid_results):
        print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
        means = grid_result.cv_results_['mean_test_score']
        stds = grid_result.cv_results_['std_test_score']
        params = grid_result.cv_results_['params']
        for mean, stdev, param in zip(means, stds, params):
            print("%f (%f) with: %r" % (mean, stdev, param))


def main():
    nn = NN()
    nn.clean_data()
    nn.create_model()
    grid_results = nn.fit(nn.create_model)
    nn.results(grid_results)

if __name__ == "__main__":
    main()

Ok, a follow up to this. Thanks for your comments @MarcinMo┼╝ejko. You were right about this. There were more errors I should mention. In def fit(), I wrote model = KerasClassifier, not self.model=Keras Classifier. I wanted to mention that incase anyone is looking at the code. I'm now getting a new error on the same line:

AttributeError: 'NoneType' object has no attribute 'loss'.

I can track this back to scikit_learn.py:

loss_name = self.model.loss
        if hasattr(loss_name, '__name__'):
            loss_name = loss_name.__name__
        if loss_name == 'categorical_crossentropy' and len(y.shape) != 2:
            y = to_categorical(y) 

I'm not sure how to solve this as I set the loss term in self.model.compile. I tried changing it to binary_crossentropy but that had no effect. any further thoughts?


Answer:

The problem lies in this line of code:

grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1)

Unfortunately - for now, keras is not supporting applying pickle to your model which is needed for sklearn to apply multiprocessing (here you may read the discussion on this). In order to make this code work you should set:

grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)

Question:

I have about 0.8 million images of 256x256 in RGB, which amount to over 7GB.

I want to use them as training data in a Convolutional Neural Network, and want to put them in a cPickle file, along with their labels.

Now, this is taking a lot of memory, to the extent that it needs to swap with my hard drive memory, and almost consume it all.

Is this is a bad idea?

What would be the smarter/more practical way to load into CNN or pickle them without causing too much memory issue?

This is what the code looks like

import numpy as np
import cPickle
from PIL import Image
import sys,os

pixels = []
labels = []
traindata = []
data=[]


for subdir, dirs, files in os.walk('images'):
        curdir=''
        for file in files:
                if file.endswith(".jpg"):
                        floc=str(subdir)+'/'+str(file)
                        im= Image.open(floc)
                        pix=np.array(im.getdata())
                        pixels.append(pix)
                        labels.append(1)
pixels=np.array(pixels)
labels=np.array(labels)
traindata.append(pixels)
traindata.append(labels)
traindata=np.array(traindata)
.....# do the same for validation and test data
.....# put all data and labels into 'data' array
cPickle.dump(data,open('data.pkl','wb'))

Answer:

Is this is a bad idea?

Yes, indeed.

You are trying to load 7GB of compressed image data into memory all at once (about 195 GB for 800k 256*256 RGB files). This will not work. You have to find a way to update your CNN image-by-image, saving the state as you go along.

Also consider how large your CCN parameter set will be. Pickle is not designed for large amounts of data. If you need to store GB worth of neural net data, you're much better off using a database. If the neural net parameter set is only a few MB, pickle will be fine, though.

You might also want to take a look at the documentation for pickle.HIGHEST_PROTOCOL, so you are not stuck with an old and unoptimized pickle file format.

Question:

I am trying to run an example of a LSTM recurrent neural network that is presented in this git: https://github.com/mesnilgr/is13.

I've installed theano and everything and when I got to the point of running the code, I've noticed the data was not being downloaded, so I've opened an issue on the github (https://github.com/mesnilgr/is13/issues/12) and this guy came up with a solution that consisted in:

1-get the data from the dropbox link he provides. 2- change the code of the 'load.py' file to download, and read the data properly.

The only issue is that the data in the dropbox folder(https://www.dropbox.com/s/3lxl9jsbw0j7h8a/atis.pkl?dl=0) is not a compacted .gz file as, I suppose, was the data from the original repository. So I dont have enough skill to change the code in order to do with the uncompressed data exaclty what it would do with the compressed one. Can someone help me?

The modification suggested and the changes I've done are described on the issue I've opened on the git(https://github.com/mesnilgr/is13/issues/12).


Answer:

It looks like your code is using

gzip.open(...)

But if the file is not gzipped then you probably just need to remove the gzip. prefix and use

open(...)

Question:

How can I create a tuple which contains two "numpy.ndarray"s? If I do so, can I update the ndarrays inside the tuple?

I working on a python code implementing some neural network, in order to make it work with my own input data. The goal is to read an image file and return an output based on what neural network outputs. I have taken the code from an excellent online book on Neural Networks written by Michael Nielsen (link).

The original code reads from a famous data-set known as MNIST, stored as a file which is output of python's pickle function. Now, the input data from the dataset is read by a data=Pickle.Load() call. Pickle loads dataset's data When this line is executed, the data variable is loaded as a "Tuple of length 2" containing two "Numpy ndArrays of length 10000",each ndarray element being another ndarray of length 784 (which are in fact values of a 28x28 image).

Now, the problem is I am trying to override the input so that instead of reading from a pickled file, it reads from a normal image file. The pickled file originally contained N image files. I am trying to crop N image files from my single image file and restructure my data so that it would be compatible with the original format.

I successfully read my image file, cropped a window of my image, created the innermost ndarray that contains 784 pixel values (a single image). But how can I create an ndarray that contains each new image that I crop, and after that put them into a tuple? Is there a way to restructure my data in a single numpy command?


Answer:

I'm making a lot of guesses as to what you want, but here's sample creating a tuple of arrays:

An 'image', raveled to the (784,) shape:

In [1027]: img=np.ones((28,28),int)
In [1028]: imgr = img.ravel()
In [1029]: imgr.shape
Out[1029]: (784,)

'stack' a bunch of copies of this image. np.concatenate is also handy for joining arrays, but np.array automatically joins them on a new axis.

In [1030]: imgs=np.array([imgr.copy(),imgr.copy()*2,imgr.copy()*3])
In [1031]: imgs
Out[1031]: 
array([[1, 1, 1, ..., 1, 1, 1],
       [2, 2, 2, ..., 2, 2, 2],
       [3, 3, 3, ..., 3, 3, 3]])
In [1032]: imgs.shape
Out[1032]: (3, 784)

Making a tuple is trivial (basic Python), just (x, y) syntax.

In [1033]: tup=(imgs, imgs.copy())
In [1034]: tup
Out[1034]: 
(array([[1, 1, 1, ..., 1, 1, 1],
        [2, 2, 2, ..., 2, 2, 2],
        [3, 3, 3, ..., 3, 3, 3]]), array([[1, 1, 1, ..., 1, 1, 1],
        [2, 2, 2, ..., 2, 2, 2],
        [3, 3, 3, ..., 3, 3, 3]]))

Strictly speaking a tuple is immutable; you can't replace elements as you would a list. But since the elements are arrays, they can be changed 'in place'.

In [1035]: tup[0][:]*=2
In [1036]: tup
Out[1036]: 
(array([[2, 2, 2, ..., 2, 2, 2],
        [4, 4, 4, ..., 4, 4, 4],
        [6, 6, 6, ..., 6, 6, 6]]), array([[1, 1, 1, ..., 1, 1, 1],
        [2, 2, 2, ..., 2, 2, 2],
        [3, 3, 3, ..., 3, 3, 3]]))

So here I've scaled the first array by 2, leaving the other as is

to scale one of the images of the 1st array, I could use:

In [1038]: tup[0][1,:] += 5
In [1039]: tup
Out[1039]: 
(array([[2, 2, 2, ..., 2, 2, 2],
        [9, 9, 9, ..., 9, 9, 9],
        [6, 6, 6, ..., 6, 6, 6]]), array([[1, 1, 1, ..., 1, 1, 1],
        [2, 2, 2, ..., 2, 2, 2],
        [3, 3, 3, ..., 3, 3, 3]]))