Can we use tf.spectral fourier functions in keras?

tensorflow fourier transform
keras fourier transform
tf fourier
keras backend fft
tensorflow signal rfft
tensorflow fft2d example
tf signal rfft
fourier series python

Let us start with an input that is a simple time series and try to build an autoencoder that simply fourier transforms then untransforms our data in keras.

If we try to do this:

inputs = Input(shape=(MAXLEN,1), name='main_input')
x = tf.spectral.rfft(inputs)
decoded = Lambda(tf.spectral.irfft)(x)

Then the third line throws an error when entered:

>> ValueError: Tensor conversion requested dtype complex64 for Tensor with dtype float32

You see, the output of tf.spectral.irfft is float32 but it looks like Lambda thinks it is complex64?? (Complex64 is the input x from the previous step)

We can fix that error at model entry time with:

inputs = Input(shape=(MAXLEN,1), name='main_input')
x = tf.spectral.rfft(inputs)
decoded = Lambda(tf.cast(tf.spectral.irfft(x),dtype=tf.float32)))

This is accepted at input time but then when we try to build the model:

autoencoder = Model(inputs, decoded)

It generates the error:

TypeError: Output tensors to a Model must be Keras tensors. Found: <keras.layers.core.Lambda object at 0x7f24f0f7bbe0>

Which I guess is reasonable and was the reason I didn't want to cast it in the first place.

Main question: how do I successfully wrap the tf.spectral.irfft function which outputs float32 ?

More general question for learning: Let's assume I actually want to do something between the rfft and the irfft, how can I cast those imaginary numbers into absolute values without breaking keras so I can apply various convolutions and the like?

I think you just need more Lambda wrapping (using tf.keras since that's what I have installed):

import numpy
import tensorflow as tf
K = tf.keras

inputs = K.Input(shape=(10, 8), name='main_input')
x = K.layers.Lambda(tf.spectral.rfft)(inputs)
decoded = K.layers.Lambda(tf.spectral.irfft)(x)
model = K.Model(inputs, decoded)
output = model(tf.ones([10, 8]))
with tf.Session():
  print(output.eval())

The output of irfft should be real, so probably no need to cast it. But if you do need to cast it (or in general combine operations in a Lambda layer), I'd wrap that in a Python lambda: K.layers.Lambda(lambda v: tf.cast(tf.spectral.whatever(v), tf.float32))

For example if you know your intermediate values (between rfft and irfft) will have an imaginary component of zero, you can truncate that off:

import numpy
import tensorflow as tf
K = tf.keras

inputs = K.Input(shape=(10, 8), name='main_input')
x = K.layers.Lambda(lambda v: tf.real(tf.spectral.rfft(v)))(inputs)
decoded = K.layers.Lambda(
    lambda v: tf.spectral.irfft(tf.complex(real=v, imag=tf.zeros_like(v))))(x)
model = K.Model(inputs, decoded)
output = model(tf.reshape(tf.range(80, dtype=tf.float32), [10, 8]))
with tf.Session():
  print(output.eval())

Note that this isn't true for general sequences, since even real-valued inputs can have imaginary components once transformed. It works for the tf.ones input above, but the tf.range input gets mangled:

[[ 0.  4.  4.  4.  4.  4.  4.  4.]
 [ 8. 12. 12. 12. 12. 12. 12. 12.]
 [16. 20. 20. 20. 20. 20. 20. 20.]
 [24. 28. 28. 28. 28. 28. 28. 28.]
 [32. 36. 36. 36. 36. 36. 36. 36.]
 [40. 44. 44. 44. 44. 44. 44. 44.]
 [48. 52. 52. 52. 52. 52. 52. 52.]
 [56. 60. 60. 60. 60. 60. 60. 60.]
 [64. 68. 68. 68. 68. 68. 68. 68.]
 [72. 76. 76. 76. 76. 76. 76. 76.]]

(Without the casting we get 0. through 79. reconstructed perfectly)

How to use tf.spectral fourier functions in keras, Main question: how do I successfully wrap the tf.spectral.irfft function which outputs float32 ? More general question for learning: Let's assume I  So, for a Fourier Convolution Layer you need to: Take the input layer and transform it to the Fourier domain: input_fft = tf.spectral.rfft2d(input) Take each kernel and transform it to the Fourier domain: weights_fft = tf.spectral.rfft2d(layer.get_weights()) Note: The Fourier domain "images" for the input and the kernels need to be of the same size.

Just to add more to what's going on above for anyone who gets here from search engines. The following, contributed in this google group discussion, will run rfft then ifft with convolutions and other layers in between:

inputs = Input(shape=(10, 8), name='main_input')
x = Lambda(lambda v: tf.to_float(tf.spectral.rfft(v)))(inputs)
x = Conv1D(filters=5, kernel_size=3, activation='relu', padding='same')(x)
x = Lambda(lambda v: tf.to_float(tf.spectral.irfft(tf.cast(v, dtype=tf.complex64))))(x)
x = Flatten()(x)
output = Dense(1)(x)
model = Model(inputs, output)
model.summary()

It uses the same concepts as Allen's answer but the slight differences allow compatibility with intermediate convolutions.

Re: How to use tf.spectral fourier functions in keras, Before trying to debug that can you actually put layers (e.g. a Conv1D layer) afterwards and have it work? I have been shown another path  Reducing Deep Network Complexity with Fourier Transform Methods - xy0806/Reducing-Deep-Network-Complexity-with-Fourier-Transform-Methods

I stumbled upon this as I was trying to solve the same problem. You can make the transition lossless by wrapping tf.real and tf.imag into Lambda layers (I'm using stft because there's no real valued equivalent):

x = tf.keras.layers.Lambda(
    lambda v: tf.signal.stft(
        v,
        frame_length=1024,
        frame_step=256,
        fft_length=1024,
    ), name='gen/FFTLayer')(inputs)
real = tf.keras.layers.Lambda(tf.real)(x)
imag = tf.keras.layers.Lambda(tf.imag)(x)
...
# transform real and imag either separately or by concatenating them in the feature space.
...
x = tf.keras.layers.Lambda(lambda x: tf.complex(x[0], x[1]))([real, imag])
x = tf.keras.layers.Lambda(
    lambda v: tf.signal.inverse_stft(
        v,
        frame_length=1024,
        frame_step=256,
        fft_length=1024,
    ))(x)

How to implement a Fourier Convolution layer in keras?, So, for a Fourier Convolution Layer you need to: Take the input layer and transform it to the Fourier domain: input_fft = tf. spectral. Perform element-wise multiplication between the input's Fourier transform and Fourier transform of each of the kernels: conv_fft = keras. The FCNN was implemented using the deep learning frameworks Keras [16] and Theano [17]. Theano is the machine learning backend of Keras. This backend was used to code the Fourier layers. The Theano FFT func-tion Theano was used to convert our training and test data. The Theano FFT function is a tensor representation of the multi-dimensional Cooley-

The fft2d function in tensorflow 1.13.1 is broke.

tf.signal.fft, Computes the 1-dimensional discrete Fourier transform over the inner-most dimension of input . Args: input : A Tensor . Must be one of the following types:  Teams. Q&A for Work. Stack Overflow for Teams is a private, secure spot for you and your coworkers to find and share information.

tf.signal.rfft, Computes the 1-dimensional discrete Fourier transform of a real-valued signal over the inner-most dimension of input . Since the DFT of a real signal is Hermitian-  Pre-trained models and datasets built by Google and the community

Audio processing in TensorFlow, TensorFlow comes with an implementation of the Fast Fourier Transform, but it the Short Time Fourier Transform can be used anywhere in the computation graph. I haven't found an implementation of the Short Time Fourier Transform in TF, So for the spectral case you get 2 DFTs, one for the positive frequencies and  Hey TF, Trying to integrate the new Unitary RNN's into tensorflow. Unfortunately, they require Fast Fourier Transform and Inverse Fast Fourier Transforms. For efficiency this needs to be done on the GPU. On the author's code (theano), th

Keras fft example, You should set `image_dim_ordering="tf"` in your Keras config located at ~/. 2 So far we have used one method for evaluating probability distributions – based on a simple example of even just showing a real-time frequency spectrum. when you are first learning how to use TensorFlow. fft - Discrete Fourier Transform  When both the function and its Fourier transform are replaced with discretized counterparts, it is called the discrete Fourier transform (DFT). Given a vector x of n input amplitudes such as: {x[0], x[1], x[2], x[3], …, x[N-1]} the Discrete Fourier Transform yields a set of n frequency magnitudes. The DFT is defined by this equation:

Comments
  • I'm marking this as right because I can't deny it runs as shown :) Even if I use tf.keras it won't seem to accept other layers types between the rfft and irfft, though. If I try to put K.layers.Conv1D between those two, for example, even wrapped in a Lambda, I get: "Value passed to parameter 'input' has DataType complex64 not in list of allowed values:" despite rfft supposedly having real outputs. Do you know how to put convolutions between the rfft and irfft?
  • rfft returns complex64, so I don't think the dtype is a bug (although maybe convolutions should work on complex numbers?). You can cast/truncate off the complex bit if you know it will be real. I'll add an example of that.
  • One option would be to split real/imaginary into channels (tf.real() in one channel, tf.imag() in the other) for the convolution.
  • It's an fft so I would need the magnitude, I'll try that with numerical operations. Thanks!
  • tf.to_float throws away the imaginary part though?
  • Also the rfft is only acting along one axis?