Hot questions for Using Neural networks in tensorflow datasets

Top 10 Python Open Source / Neural networks / tensorflow datasets

Question:

I'm running some experiments with neural networks in TensorFlow. The release notes for the latest version say DataSet is henceforth the recommended API for supplying input data.

In general, when taking numeric values from the outside world, the range of values needs to be normalized; if you plug in raw numbers like length, mass, velocity, date or time, the resulting problem will be ill-conditioned; it's necessary to check the dynamic range of values and normalize to the range (0,1) or (-1,1).

This can of course be done in raw Python. However, DataSet provides a number of data transformation features and encourages their use, on the theory that the resulting code will not only be easier to maintain, but run faster. That suggests there should also be a built-in feature for normalization.

Looking over the documentation at https://www.tensorflow.org/programmers_guide/datasets however, I'm not seeing any mention of such. Am I missing something? What is the recommended way to do this?


Answer:

My understanding of tensorflow datasets main idea tells me that complex pre-procesing is not directly applicable, because tf.data.Dataset is specifically designed to stream very large amounts of data, more precisely tensors:

A Dataset can be used to represent an input pipeline as a collection of elements (nested structures of tensors) and a "logical plan" of transformations that act on those elements.

The fact that tf.data.Dataset operates with tensors means that obtaining any particular statistic over the data, such as min or max, requires a complete tf.Session and at least one run through the whole pipeline. The following sample lines:

iterator = dataset.make_one_shot_iterator()
batch_x, batch_y = iterator.get_next()

... which are designed to provide the next batch fast, no matter of the size of the dataset, would stop the world until the first batch is ready, if the dataset is responsible for pre-processing. That's why the "logical plan" includes only local transformations, which ensures the data can be streamed and, in addition, allows to do transformations in parallel.

This doesn't mean it's impossible to implement the normalization with tf.data.Dataset, I feel like it's never been designed to do so and, as a result, it will look ugly (though I can't be absolutely sure of that). However, note that batch-normalization fits into this picture perfectly, and it's one of the "nice" options I see. Another option is do simple pre-processing in numpy and feed the result of that into tf.data.Dataset.from_tensor_slices. This doesn't make the pipeline much more complicated, but doesn't restrict you from using tf.data.Dataset at all.

Question:

I am new to TensorFlow. I looked for examples on implementation of multi layer perceptron using tensorflow, but i am getting examples only on MNIST image data sets, apart from MNIST can i able to build the Neural Network model using same optimization and cost functions and train the data which is in number format,Means, Can I train my own number dataset using tensorflow.

Is there any example for training the new dataset?.


Answer:

Finally i got it. Building , Training and minimizing cost / loss of an Artificial Neural Network using Single Layer Perceptron with tensorflow, numpy , matplotlib packages. Data is used in the form of array instead of MNIST. Here is the code.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
learning_rate = 0.0008
training_epochs = 2000
display_step = 50
# taking input as array from numpy package and converting it into tensor
inputX = np.array([[  2,   3],
                  [  1,   3]])
inputY = np.array([[  2,   3],
                  [  1,   3]])
x = tf.placeholder(tf.float32, [None, 2])
y_ = tf.placeholder(tf.float32, [None, 2])

W = tf.Variable([[0.0,0.0],[0.0,0.0]])
b = tf.Variable([0.0,0.0])

layer1 = tf.add(tf.matmul(x, W), b)
y = tf.nn.softmax(layer1)

cost = tf.reduce_sum(tf.pow(y_-y,2))

optimizer =tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)

init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)

avg_set = []
epoch_set = []

for i in range(training_epochs):
   sess.run(optimizer, feed_dict = {x: inputX, y_:inputY})

   #log training
   if i % display_step == 0:
       cc = sess.run(cost, feed_dict = {x: inputX, y_:inputY})
       #check what it thinks when you give it the input data
       print(sess.run(y, feed_dict = {x:inputX}))


       print("Training step:", '%04d' % (i), "cost=", "{:.9f}".format(cc))
       avg_set.append(cc)
       epoch_set.append(i + 1)

print("Optimization Finished!")
training_cost = sess.run(cost, feed_dict = {x: inputX, y_: inputY})
print("Training cost = ", training_cost, "\nW=", sess.run(W),
      "\nb=", sess.run(b))
plt.plot(epoch_set,avg_set,'o',label = 'SLP Training phase')
plt.ylabel('cost')
plt.xlabel('epochs')
plt.legend()
plt.show()

Later by adding hidden layers it can be also implemented with Multi Layer Perceptron

Question:

I'm loading a dataset with multiple input images. The input image paths should only be decoded at batch time, in order to handle a large dataset.

The data set is N image path inputs and M float outputs. The images for each input have different resolutions.

Data is ([img_input_1.png, img_input_2.png, ...], [0.65, 0.7, 0.8])

The model is using the Keras functional api in symbolic mode.

Here is the most recently EDITED code

from itertools import zip_longest

def read_image(path, shape):
    try:
        image = tf.io.read_file(path)
        image = tf.image.decode_png(image)
        image = tf.image.resize(image, [shape[1],shape[2]])
        image /= 255.0
        return image
    except:
        print('ERROR: preprocess_image: bad path', path)    

def load_image(x, y, shp):
    pout = [(k, x[k]) for k in x.keys()]
    l1   = tf.convert_to_tensor(list(x))
    l2   = tf.convert_to_tensor(list(x.values()))

    pl = tf.map_fn(
        lambda args: (read_image(args[0], shp), args[1]), [l1, l2], dtype=(tf.float32, tf.float32)
    )
    pl = {path: (pl[0][i], pl[1][i]) for i, path in enumerate(x)}
    return (pl,y)

def dataset_prep(json_data, seq, batch_size):
    # LOAD DATA FROM JSON
    x,y = json_parse_x_y(json_data[seq])
    xx  = [*zip_longest(*x)] # NOTE: goes from variable sized input to {'input_N':...}
    yy  = [*zip_longest(*y)]

    # GET SHAPES (hard coded atm)
    lns = [[len(xxx)] for xxx in xx]
    rzs = [[24,512,1],[96,512,1]] # TEMP TODO! grab grom [(v['h'],v['w'],v['c']) for v in xx]
    shp = [*zip_longest(*[lns,rzs])]
    shp = [list(s) for s in shp]
    shp = [[*itertools.chain.from_iterable(s)] for s in shp]

    xd  = dict([[ "input_{}".format(i+1),np.array(y)] for i,y in [*enumerate(xx)]])
    yd  = dict([["output_{}".format(i+1),np.array(y)] for i,y in [*enumerate(yy)]])

    ds  = tf.data.Dataset.from_tensor_slices((xd, yd))

    ds  = ds.shuffle(10000)
    ds  = ds.repeat()
    ds  = ds.map(map_func=lambda x,y: load_image(x, y, shp), num_parallel_calls=AUTOTUNE)
    ds  = ds.batch(batch_size) if batch_size else ds 
    ds  = ds.prefetch(AUTOTUNE) 
    return ds

This is the error I'm getting:

Traceback (most recent call last):
    File "/home/me/.local/bin/wavfeat", line 11, in <module>
        load_entry_point('wavfeat==0.1.0', 'console_scripts', 'wavfeat')()
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/__main__.py", line 91, in main
        analysis_batch_sql(obj)
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 50, in analysis_batch_sql
        qy = [*map(lambda c: run_elm(c[0], c[1]), ch)]
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 50, in <lambda>
        qy = [*map(lambda c: run_elm(c[0], c[1]), ch)]
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 23, in run_elm
        out = fn(input, elm)
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 196, in one_sec_onset_train
        return train(input, elm)
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 182, in train
        ts = dataset_prep(jd, 'train', bc)
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 123, in dataset_prep
        ds  = ds.map(map_func=lambda x,y: load_image(x, y, shp), num_parallel_calls=AUTOTUNE)
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1146, in map
        self, map_func, num_parallel_calls, preserve_cardinality=True)
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3264, in __init__
        use_legacy_function=use_legacy_function)
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2591, in __init__
        self._function = wrapper_fn._get_concrete_function_internal()
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1366, in _get_concrete_function_internal
        *args, **kwargs)
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1360, in _get_concrete_function_internal_garbage_collected
        graph_function, _, _ = self._maybe_define_function(args, kwargs)
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1648, in _maybe_define_function
        graph_function = self._create_graph_function(args, kwargs)
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1541, in _create_graph_function
        capture_by_value=self._capture_by_value),
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 716, in func_graph_from_py_func
        func_outputs = python_func(*func_args, **func_kwargs)
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2585, in wrapper_fn
        ret = _wrapper_helper(*args)
    File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2530, in _wrapper_helper
        ret = func(*nested_args)
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 123, in <lambda>
        ds  = ds.map(map_func=lambda x,y: load_image(x, y, shp), num_parallel_calls=AUTOTUNE)
    File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_data_loader.py", line 91, in load_image
        print("x['input_1'].values(): ", x['input_1'].values())
AttributeError: 'Tensor' object has no attribute 'values'

What am I doing that is preventing the paths from being loaded?

EDIT:

Attempting pandrey's fix, I'm getting input errors. Here is the data before from_tensor_slices and ds.map and then after:

pre_from_tensor_slices x:  {'input_1': array(['/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/7388_39216_30--id=7388__sql_table=oac_1__sql_idx=405167__pitch=30__onset=39216.png',
       '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/2447_864_27--id=2447__sql_table=oac_1__sql_idx=415458__pitch=27__onset=864.png',
       '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/2386_20208_38--id=2386__sql_table=oac_1__sql_idx=433248__pitch=38__onset=20208.png',
       ...,
       '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/6261_24528_57--id=6261__sql_table=oac_1__sql_idx=449753__pitch=57__onset=24528.png',
       '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/3727_22944_31--id=3727__sql_table=oac_1__sql_idx=407620__pitch=31__onset=22944.png',
       '/media/me/sp_data/sp_data/datasets/chr_01/one_sec_onset_11_oac-leg/1668_7056_60--id=1668__sql_table=oac_1__sql_idx=381152__pitch=60__onset=7056.png'],
      dtype='<U162'), 'input_2': array(['/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/7388_39216_30--id=7388__sql_table=oac_1__sql_idx=405167__pitch=30__onset=39216.png',
       '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/2447_864_27--id=2447__sql_table=oac_1__sql_idx=415458__pitch=27__onset=864.png',
       '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/2386_20208_38--id=2386__sql_table=oac_1__sql_idx=433248__pitch=38__onset=20208.png',
       ...,
       '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/6261_24528_57--id=6261__sql_table=oac_1__sql_idx=449753__pitch=57__onset=24528.png',
       '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/3727_22944_31--id=3727__sql_table=oac_1__sql_idx=407620__pitch=31__onset=22944.png',
       '/media/me/sp_data/sp_data/datasets/mel_01/one_sec_onset_11_oac-leg/1668_7056_60--id=1668__sql_table=oac_1__sql_idx=381152__pitch=60__onset=7056.png'],
      dtype='<U162')}
pre_from_tensor_slices y:  {'output_1': array([0.817, 0.018, 0.421, ..., 0.511, 0.478, 0.147])}
_________________________
y:  {'output_1': <tf.Tensor 'args_2:0' shape=() dtype=float64>}
x:  {'input_1': <tf.Tensor 'args_0:0' shape=() dtype=string>, 'input_2': <tf.Tensor 'args_1:0' shape=() dtype=string>}
x.values():  dict_values([<tf.Tensor 'args_0:0' shape=() dtype=string>, <tf.Tensor 'args_1:0' shape=() dtype=string>])
x['input_1']:  Tensor("args_0:0", shape=(), dtype=string)

Running x['input_1'].values() throws an error: 'Tensor' object has no attribute 'values'

I get an error situated around map_fn

  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 284, in _constant_impl
    allow_broadcast=allow_broadcast))
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 455, in make_tensor_proto
    raise ValueError("None values not supported.")
ValueError: None values not supported.

EDIT 2

Attempting the latest I get the following error

Traceback (most recent call last):
  File "/home/me/.local/bin/wavfeat", line 11, in <module>
    load_entry_point('wavfeat==0.1.0', 'console_scripts', 'wavfeat')()
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/__main__.py", line 91, in main
    analysis_batch_sql(obj)
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 50, in analysis_batch_sql
    qy = [*map(lambda c: run_elm(c[0], c[1]), ch)]
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 50, in <lambda>
    qy = [*map(lambda c: run_elm(c[0], c[1]), ch)]
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/analysis_run_csv.py", line 23, in run_elm
    out = fn(input, elm)
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 216, in one_sec_onset_train
    return train(input, elm)
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 203, in train
    vs = validation_prep(jd, 'validation', bc)
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_onset.py", line 176, in validation_prep
    ds  = ds.map(map_func=load_and_preprocess_from_path_label, num_parallel_calls=AUTOTUNE)
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1146, in map
    self, map_func, num_parallel_calls, preserve_cardinality=True)
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3264, in __init__
    use_legacy_function=use_legacy_function)
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2591, in __init__
    self._function = wrapper_fn._get_concrete_function_internal()
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1366, in _get_concrete_function_internal
    *args, **kwargs)
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1360, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1648, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1541, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 716, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2585, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2530, in _wrapper_helper
    ret = func(*nested_args)
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_data_loader.py", line 47, in load_and_preprocess_from_path_label
    pl   = dict([(pk, tf.map_fn(load_and_preprocess_image, po, dtype=tf.float32)) for pk,po in pout])
  File "/home/me/.local/lib/python3.6/site-packages/wavfeat/one_sec_data_loader.py", line 47, in <listcomp>
    pl   = dict([(pk, tf.map_fn(load_and_preprocess_image, po, dtype=tf.float32)) for pk,po in pout])
  File "/home/me/.local/lib/python3.6/site-packages/tensorflow/python/ops/map_fn.py", line 214, in map_fn
    raise ValueError("elems must be a 1+ dimensional Tensor, not a scalar")
ValueError: elems must be a 1+ dimensional Tensor, not a scalar

Answer:

Add-on: not using dict structures

This is a full code (save for defining json_parse_x_y and declaring AUTOTUNE) to achieve what you are attempting without using dict structures.

I tested that make_dataset works (see example below), so if you encounter an issue it should be due to a specification error regarding load_tensors.

from itertools import zip_longest

import tensorflow as tf

# additionnally, `json_parse_x_y` must be defined
# and `AUTOTUNE` must be declared (in my example, I set it to 2)


def read_image(path, shape):
    """Read an image of givent filepath and tensor shape.

    Return a float tensor of given shape.
    """
    try:
        image = tf.io.read_file(path)
        image = tf.image.decode_png(image)
        image = tf.image.resize(image, [shape[1], shape[2]])
        image /= 255.0
        return image
    except:
        raise FileNotFoundError("preprocess_image: bad path '%s'" % path)


def load_images(paths, shapes):
    """Load an ensemble of images (associated with a single sample).

    paths  : rank-1 string Tensor
    shapes : list of images' shapes (same length as `paths`)

    Return a tuple of float tensors containing the loaded images.
    """
    return tuple((
        read_image(paths[i], shapes[i])
        for i in range(len(shapes))
    ))


def load_tensors(json_data, seq):
    """Load images descriptors from a json dump.

    Return a tuple containing:
        * a rank-2 tensor containing lists of image paths (str)
        * a rank-2 tensor containing resolution values (float)
        * a list of image shapes, of same length as the rank-2
          tensor's second axis
    """
    x,y = json_parse_x_y(json_data[seq])
    xx  = [*zip_longest(*x)] # NOTE: goes from variable sized input to {'input_N':...}
    yy  = [*zip_longest(*y)]

    # GET SHAPES (hard coded atm)
    lns = [[len(xxx)] for xxx in xx]
    rzs = [[24,512,1],[96,512,1]] # TEMP TODO! grab grom [(v['h'],v['w'],v['c']) for v in xx]
    shp = [*zip_longest(*[lns,rzs])]
    shp = [list(s) for s in shp]
    shp = [[*itertools.chain.from_iterable(s)] for s in shp]
    return (xx, yy, shp)


def make_dataset(xx, yy, shp, batch_size):
    """Build a Dataset instance containing loaded images.

    xx, yy, shp : see the specification of `load_tensors`'s outputs
    batch_size  : batch size to set on the Dataset

    Return a Dataset instance where each batched sample is a tuple
    containing two elements: first, a tuple containing N loaded images'
    rank-3 tensors; second, a rank-1 tensor containing M float values.
    (to be clear: batching adds a dimension to all those tensors)
    """
    data = tf.data.Dataset.from_tensor_slices((xx, yy))
    data = data.shuffle(10000)
    data = data.map(lambda x, y: (load_images(x, shapes), y))
    data = data.repeat()
    data = data.batch(batch_size) if batch_size else data
    data = data.prefetch(AUTOTUNE) 
    return data


def dataset_prep(json_data, seq, batch_size):
    """Full pipeline to making a Dataset from json."""
    xx, yy, shapes = load_tensors(json_data, seq)
    return make_dataset(xx, yy, shapes)

Example, using "hand-made' values ; all images are actually this classic image, of shape [512, 512, 3].

import numpy as np
import tensorflow as tf

# import previous code

# Here, N = 2, and I make 2 samples.
x = tf.convert_to_tensor(np.array([
    ['image_1a.png', 'image_1b.png'],
    ['image_2a.png', 'image_2b.png']
]))
shapes = [[1, 512, 512], [1, 512, 512]]  # images are initially [512, 512, 3]
# Here, M = 3, and I make 2 samples. Values are purely random.
y = tf.convert_to_tensor(np.array([
    [.087, .92, .276],
    [.242, .37, .205]
]))

# This should work.
data = make_dataset(x, y, shapes, batch_size=1)
# Output signature is <PrefetchDataset shapes:
#     (((None, 512, 512, None), (None, 512, 512, None)), (None, 3)),
#     types: ((tf.float32, tf.float32), tf.float64)
# >
# Where the first None is actually `batch_size`
# and the second is, in this case, 3.

Answer to the current question:

Okay, the problem you are now encountering is that the revised load_image function does not fit the specifications of the Dataset, hence the exception raising. Please find below a full edited code that seems to work (I ran a test using custom images on my computer, with xd / yd dict initialized to look like your reported x and y in-dataset tensors). It is not pretty, and I would personally advise to drop the dict structures, but it works:

from itertools import zip_longest

def read_image(path, shape):
    try:
        image = tf.io.read_file(path)
        image = tf.image.decode_png(image)
        image = tf.image.resize(image, [shape[1],shape[2]])
        image /= 255.0
        return image
    except:
        raise FileNotFoundError("preprocess_image: bad path '%s'" % path)

# CHANGED: load_image is actually useless

def dataset_prep(json_data, seq, batch_size):
    # LOAD DATA FROM JSON
    x,y = json_parse_x_y(json_data[seq])
    xx  = [*zip_longest(*x)] # NOTE: goes from variable sized input to {'input_N':...}
    yy  = [*zip_longest(*y)]

    # GET SHAPES (hard coded atm)
    lns = [[len(xxx)] for xxx in xx]
    rzs = [[24,512,1],[96,512,1]] # TEMP TODO! grab grom [(v['h'],v['w'],v['c']) for v in xx]
    shp = [*zip_longest(*[lns,rzs])]
    shp = [list(s) for s in shp]
    shp = [[*itertools.chain.from_iterable(s)] for s in shp]

    xd  = dict([[ "input_{}".format(i+1),np.array(y)] for i,y in [*enumerate(xx)]])
    yd  = dict([["output_{}".format(i+1),np.array(y)] for i,y in [*enumerate(yy)]])

    ds  = tf.data.Dataset.from_tensor_slices((xd, yd))
    ds  = ds.shuffle(10000)

    # CHANGED: the following line, to run images import (also moved epeat instruction later)
    ds  = ds.map(
        lambda x, y: (
            {key: read_image(path, shp[i]) for i, (key, path) in enumerate(x.items())},
            y
        ),
        num_parallel_calls=AUTOTUNE
    )
    ds  = ds.repeat()
    ds  = ds.batch(batch_size) if batch_size else ds 
    ds  = ds.prefetch(AUTOTUNE) 
    return ds

Initial answer (before question edit):

I will only deal with the exception raised by load_image in this answer, but there might be additional work to perform on the rest - I did not test for that, not having a convenient dataset at hand.

The exception message is actually quite explicit: you are passing a scalar element (e.g. n in [(k, tf.map_fn(lambda x: read_image(x, shp), n, dtype=tf.float32)) for k,n in pout]) as elems argument to tf.map_fn, when it expects a tensor (or (possibly nested) list or tuple of tensors), as clearly specified in its documentation.

You are also using tf.map_fn the wrong way in the quoted line of code, because basically you are mixing it up with a python intention list, when you should use either one or the other.

With intention list (also replacing the useless previous lines of the load_image function): pl = {path: (load_image(path, shp), res) for path, res in x.items()}

With tf.map_fn:

# Read all images, return two tensors, one with images, the other with resolutions.
# (so, resolutions inclusion in this is actually useless and should be redesigned)
pl = tf.map_fn(
    lambda args: (read_image(args[0], shp), args[1]),
    [tf.convert_to_tensor(list(x)), tf.convert_to_tensor(list(x.values()))],
    dtype=(tf.float32, tf.float32)
)
# If you really, really want to return a dict, but is it an optimal design?
pl = {path: (pl[0][i], pl[1][i]) for i, path in enumerate(x)}

I do not know whether returning a dict specified in this way is optimal (or even compatible) with Dataset instantiation, however if the rest of your code is working, this should do the trick.

At any rate, if you want to iterate over a dict, go ahead and use either the first version or a modified version of the second one (which may have the advantage of parallelizing images reading).

I hope this helps :-)

Question:

Here is the piece of code

def train(x):
    prediction = cnn(x)
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=prediction, labels=y))
    optimizer = tf.train.AdadeltaOptimizer().minimize(cost)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())

        for epoch in xrange(num_epochs):
            epoch_loss = 0
            for _ in xrange(int(1020/batch_size)):
                epoch_x, epoch_y = train_iterator.get_next()
                _, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y})
                epoch_loss += c

            print('Epoch {} completed out of {} - loss {}'.format(epoch + 1, num_epochs, epoch_loss))

the line of the error is this

_, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y})

I read the data from a tfrecord file using this code

def read_image_dataset_tfrecordfile(filenames, color=False, resize=False, width=100, height=100):

    def parser(record):
        keys_to_features = {
            "image": tf.FixedLenFeature([], tf.string),
            "label": tf.FixedLenFeature([], tf.int64)
        }
        parsed = tf.parse_single_example(record, keys_to_features)
        image = tf.decode_raw(parsed["image"], tf.uint8)
        image = tf.cast(image, tf.float32)
        if resize:
            if color:
                image = tf.reshape(image, shape=[width, height, 3])
            else:
                image = tf.reshape(image, shape=[width, height, 1])
        label = tf.cast(parsed["label"], tf.int32)
        label = tf.one_hot(label, 17)

        return {'image': image}, label

    dataset = tf.data.TFRecordDataset(filenames)
    dataset = dataset.map(parser)

    return dataset

I printed for example one image and its label here

[[59.],
        [94.],
        [79.],
        ...,
        [41.],
        [42.],
        [43.]],

       [[56.],
        [86.],
        [91.],
        ...,
        [43.],
        [41.],
        [33.]],

       [[53.],
        [69.],
        [63.],
        ...,
        [56.],
        [59.],
        [51.]]], dtype=float32)}, array([0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
      dtype=float32))

The image is float32 as the input of my network. You can see

x = tf.placeholder(tf.float32, [None, 10000])
def cnn(x):
    weights = {
        'W_conv1': tf.Variable(tf.random_normal([5, 5, 1, 16])),
        'W_conv2': tf.Variable(tf.random_normal([5, 5, 16, 16])),
        'W_conv3': tf.Variable(tf.random_normal([5, 5, 16, 32])),
        'W_conv4': tf.Variable(tf.random_normal([5, 5, 32, 32])),
        'W_fc': tf.Variable(tf.random_normal([24 * 24 * 32, 1024])),
        'out': tf.Variable(tf.random_normal([1024, n_classes]))
    }

    biases = {
        'b_conv1': tf.Variable(tf.random_normal([16])),
        'b_conv2': tf.Variable(tf.random_normal([16])),
        'b_conv3': tf.Variable(tf.random_normal([32])),
        'b_conv4': tf.Variable(tf.random_normal([32])),
        'b_fc': tf.Variable(tf.random_normal([1024])),
        'b_out': tf.Variable(tf.random_normal([n_classes]))
    }

    x = tf.reshape(x, [-1, 100, 100, 1])

    conv1 = tf.nn.relu(tf.nn.conv2d(x, weights['W_conv1'], strides=[1, 1, 1, 1], padding='SAME') + biases['b_conv1'])
    conv2 = tf.nn.relu(tf.nn.conv2d(conv1, weights['W_conv2'], strides=[1, 1, 1, 1], padding='SAME') +
                       biases['b_conv2'])
    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

    conv3 = tf.nn.relu(tf.nn.conv2d(conv2, weights['W_conv3'], strides=[1, 1, 1, 1], padding='SAME') +
                       biases['b_conv3'])
    conv4 = tf.nn.relu(tf.nn.conv2d(conv3, weights['W_conv4'], strides=[1, 1, 1, 1], padding='SAME') +
                       biases['b_conv4'])
    conv4 = tf.nn.max_pool(conv4, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

    fc = tf.reshape(conv4, [-1, 24 * 24 * 32])
    fc = tf.nn.relu(tf.matmul(fc, weights['W_fc']) + biases['b_fc'])
    fc = tf.nn.dropout(fc, dropout_rate)

    out = tf.matmul(fc, weights['out']) + biases['b_out']

    return out

I'm using the same network that I used with MNIST dataset from TensorFlow examples. The weights and biases are float so my input must be float right? With the MNIST dataset worked all like a charm but now it gives me this error and I can't figure out why.

edit 1

Traceback (most recent call last):
  File "/Users/user/PycharmProjects/ProveTF/main.py", line 109, in <module>
    train(x)
  File "/Users/user/PycharmProjects/ProveTF/main.py", line 84, in train
    _, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y})
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 905, in run
    run_metadata_ptr)
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1106, in _run
    np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
  File "/Users/user/venv/lib/python2.7/site-packages/numpy/core/numeric.py", line 492, in asarray
    return array(a, dtype, copy=False, order=order)
TypeError: float() argument must be a string or a number

edit 2

Traceback (most recent call last):
  File "/Users/user/PycharmProjects/ProveTF/main.py", line 111, in <module>
    train(x)
  File "/Users/user/PycharmProjects/ProveTF/main.py", line 84, in train
    _, c = sess.run([optimizer, cost])
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 905, in run
    run_metadata_ptr)
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1137, in _run
    feed_dict_tensor, options, run_metadata)
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1355, in _do_run
    options, run_metadata)
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1374, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
     [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?,100,100,1], [?,17]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]

Caused by op u'IteratorGetNext', defined at:
  File "/Users/user/PycharmProjects/ProveTF/main.py", line 109, in <module>
    x, y = train_iterator.get_next()
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 330, in get_next
    name=name)), self._output_types,
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 866, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
    op_def=op_def)
  File "/Users/user/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1650, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

OutOfRangeError (see above for traceback): End of sequence
     [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?,100,100,1], [?,17]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]

edit 3

def train(input):
    prediction = cnn(input)
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=prediction, labels=y))
    optimizer = tf.train.AdadeltaOptimizer().minimize(cost)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())

        for epoch in xrange(num_epochs):
            epoch_loss = 0
            for _ in xrange(int(1020/batch_size)):
                try:
                    _, c = sess.run([optimizer, cost])
                    epoch_loss += c
                except tf.errors.OutOfRangeError:
                    train_set.repeat()

            print('Epoch {} completed out of {} - loss {}'.format(epoch + 1, num_epochs, epoch_loss))

Answer:

dict / array cast error

There's too much code and dependencies to reproduce your problem. It seems however to me that your error may come from your parser(record) function, which returns your images wrapped in a dict (c.f. {'image': image}, while it is not the case for your label). As epoch_x would then contain dict elements, Tensorflow (and numpy) would fail trying to convert them into the expected data type (a tf.float32 tensor, c.f. definition of placeholder x), which may explain your cast-related error.

Long story short, try replacing return {'image': image}, label by return image, label in your parser.


Tensorflow Dataset API vs feed_dict

Somehow this problem escaped me at first. Given your Tensorflow-Dataset-based input pipeline, you are not supposed to use placeholder / feed_dict. The latter are meant to pass your data on the CPU to Tensorflow (supposedly running on GPUs). This duplication/conversion of the inputs done through feed_dict is much overhead, hence the development of the Tensorflow Dataset API, which short-circuits all this by reading and converting your data in parallel of the actual graph runs. In other words, your epoch_x, epoch_y don't need to be fed to Tensorflow ; they are already part of its graph.

Basically, your pipeline should look like something similar to the following:

train_dataset = read_image_dataset_tfrecordfile(my_filenames)
train_dataset = train_dataset.repeat() # if you want to loop indefinitely
train_iterator = train_dataset.make_one_shot_iterator()

x, y = train_iterator.get_next()
# x, y will represent your data batches, fed with the next ones every time 
# they are called.
# So you just use them directly instead of placeholders:
prediction = cnn(x) 
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(
    logits=prediction, labels=y))
optimizer = tf.train.AdadeltaOptimizer().minimize(cost)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for epoch in xrange(num_epochs):
        epoch_loss = 0
        for _ in xrange(int(1020/batch_size)):
            _, c = sess.run([optimizer, cost])
            # ...