Hot questions for Using Neural networks in julia

Question:

I would like to recognize if a cell is infected by malaria or not (https://www.kaggle.com/iarunava/cell-images-for-detecting-malaria). But the pictures have different size, so I want to resize every image to the same size to be able to use my first layer (that has a static size)

How can I resize an image in Julia? Or can I just use flux to be able to compute image of different dimension?


Answer:

You can do this with the packages Images and ImageMagick (you need to install both), and then:

using Images
download("https://juliaimages.org/latest/assets/logo.png","test.png");
img = Images.load("test.png");
size(img) # outputs (128, 128)
img2 = imresize(img,(50,50));
Images.save("test2.png",img2);

The file test2.png has the size 50x50. More info about imsize is available at:

https://juliaimages.org/latest/function_reference.html#Spatial-transformations-and-resizing-1

You would do this operation before hand (i.e. outside of Flux.jl) because otherwise the interpolation would need to be done at every time you compute the gradient.

The package Interpolations would also allow you to resize the images. You would then interpolate the red, green, blue channels individually.

For the particular images of the kaggle challenge, padding the data with black pixels to a common size would also be an option.

Question:

In the Julia library Flux, we have the ability to take a neural network, let's call it network m and extract the weights of network m with the following code:

params(m)

This returns a Zygote.Params type of object, of the form:

Params([Float32[0.20391908 -0.101616435 0.09610984 -0.1013181 -0.13325627 -0.034813307 -0.13811183 0.27022845 ...]...)

If I wanted to alter each of the weights slightly, how would I be able to access them?

Edit: As requested, here is the structure for m:

Chain(LSTM(8,10),Dense(10,1))

Answer:

You can iterate on a Params object to access each set of parameters as an array, which you can manipulate in place.

Supposing you want to change every parameter by 1‰, you could do something like the following:

julia> using Flux

julia> m = Dense(10, 5, σ)
Dense(10, 5, σ)

julia> params(m)
Params([Float32[-0.026854342 -0.57200056 … 0.36827534 -0.39761665; -0.47952518 0.594778 … 0.32624483 0.29363066; … ; -0.22681071 -0.0059174187 … -0.59344876 -0.02679312; 
-0.4910349 0.60780525 … 0.114975974 0.036513895], Float32[0.0, 0.0, 0.0, 0.0, 0.0]])                                                                                     

julia> for p in params(m)
           p .*= 1.001
       end

julia> params(m)
Params([Float32[-0.026881196 -0.5725726 … 0.3686436 -0.39801428; -0.4800047 0.5953728 … 0.32657108 0.2939243; … ; -0.22703752 -0.0059233364 … -0.5940422 -0.026819913; -0.
49152592 0.60841304 … 0.11509095 0.03655041], Float32[0.0, 0.0, 0.0, 0.0, 0.0]])                                                                                         

Question:

I am try to rewrite code from this tutorial on python to julia and getting unexpected result - [0.5; 0.5; 0.5; 0.5] I look to the line again and again but not see difference.

Python code:

from numpy import exp, array, random, dot
training_set_inputs = array([[0, 0, 1], [1, 1, 1], [1, 0, 1], [0, 1, 1]])
training_set_outputs = array([[0, 1, 1, 0]]).T
random.seed(1)
synaptic_weights = 2 * random.random((3, 1)) - 1
for iteration in xrange(10000):
    output = 1 / (1 + exp(-(dot(training_set_inputs, synaptic_weights))))
    synaptic_weights += dot(training_set_inputs.T, (training_set_outputs - output) * output * (1 - output))
print 1 / (1 + exp(-(dot(array([1, 0, 0]), synaptic_weights))))

My julia code:

function activate(x)
    return 1./(1+exp(-x))
end

function g_activate(x)
    return x.*(1-x)
end

function test(iter)

Input = [0 0 1;0 1 1;1 0 1;1 1 1]
TInput = transpose(Input)
Test = [0, 1, 1, 0]
Weights = 2 * rand(3, 1) - 1

for i in 1:iter

output = activate(Input*Weights)
error = Test - output
delta = error.*g_activate(output)
Weights +=  TInput*delta

end

println(activate(Input*Weights))
end

What I am doing wrong and how do it more idiomatic way in Julia


Answer:

You are using wrong input data in Julia code. To match the Python example

Input = [0 0 1;0 1 1;1 0 1;1 1 1]

should be

Input = [0 0 1;1 1 1;1 0 1;0 1 1]

That's what I'm getting with corrected input:

julia> test(10000)
[0.00966854; 0.992117; 0.993589; 0.00786553]

And if I'm running Python code with training_set_inputs = array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]]) I'm getting [ 0.5].

Question:

I'm trying to build a neural network that can answer to the xor problem. My code is the following:

using MXNet
using Distributions
using PyPlot

xor_data = zeros(4,2)
xor_data[1:0] = 1
xor_data[1:1] = 1
xor_data[2:0] = 1
xor_data[2:1] = 0
xor_data[3:0] = 0
xor_data[3:1] = 1
xor_data[4:0] = 0
xor_data[4:1] = 0

xor_labels = zeros(4)
xor_labels[1] = 0
xor_labels[2] = 1
xor_labels[3] = 1
xor_labels[4] = 0

batchsize = 4
trainprovider = mx.ArrayDataProvider(:data => xor_data, batch_size=batchsize, shuffle=true, :label => xor_labels)
evalprovider = mx.ArrayDataProvider(:data => xor_data, batch_size=batchsize, shuffle=true, :label => xor_labels)

data = mx.Variable(:data)
label = mx.Variable(:label)
net = @mx.chain     mx.Variable(:data) =>
                    mx.FullyConnected(num_hidden=2) =>
                    mx.Activation(act_type=:relu) =>
                    mx.FullyConnected(num_hidden=2) =>
                    mx.Activation(act_type=:relu) =>
                    mx.FullyConnected(num_hidden=1) =>
                    mx.Activation(act_type=:relu) =>

model = mx.FeedForward(net, context=mx.cpu())
optimizer = mx.SGD(lr=0.01, momentum=0.9, weight_decay=0.00001)
initializer = mx.NormalInitializer(0.0,0.1)
eval_metric = mx.MSE()

mx.fit(model, optimizer, initializer, eval_metric, trainprovider, eval_data = evalprovider, n_epoch = 100)
mx.fit(model, optimizer, eval_metric, trainprovider, eval_data = evalprovider, n_epoch = 100)

But I'm getting the following error:

LoadError: AssertionError: Number of samples in label is mismatch with data in expression starting on line 22 in #ArrayDataProvider#6428(::Int64, ::Bool, ::Int64, ::Int64, ::Type{T}, ::Pair{Symbol,Array{Float64,2}}, ::Pair{Symbol,Array{Float64,1}}) at io.jl:324 in (::Core.#kw#Type)(::Array{Any,1}, ::Type{MXNet.mx.ArrayDataProvider}, ::Pair{Symbol,Array{Float64,2}}, ::Pair{Symbol,Array{Float64,1}}) at :0 in include_string(::String, ::String) at loading.jl:441 in include_string(::String, ::String) at sys.dylib:? in include_string(::Module, ::String, ::String) at eval.jl:32 in (::Atom.##59#62{String,String})() at eval.jl:81 in withpath(::Atom.##59#62{String,String}, ::String) at utils.jl:30 in withpath(::Function, ::String) at eval.jl:46 in macro expansion at eval.jl:79 [inlined] in (::Atom.##58#61{Dict{String,Any}})() at task.jl:60

I want to feed to the network to values (0 or 1) and get a single value. Were is my error?


Answer:

Dimensions of xor_data are wrong, it should have 4 columns, not 4 rows (and, by the way, you're not initialising it the way you think you do, since arrays in Julia are indexed from 1, not from 0).

Look:

julia> xor_data = [ [1. 1]; [0 1]; [1 0]; [0 0] ]
4×2 Array{Float64,2}:
 1.0  1.0
 0.0  1.0
 1.0  0.0
 0.0  0.0

julia> xor_labels
4-element Array{Float64,1}:
 0.0
 1.0
 1.0
 0.0

julia> mx.ArrayDataProvider(:data => xor_data, :labels => xor_labels)
ERROR: AssertionError: Number of samples in  labels is mismatch with data
 in #ArrayDataProvider#6428(::Int64, ::Bool, ::Int64, ::Int64, ::Type{T}, ::Pair{Symbol,Array{Float64,2}}, ::Pair{Symbol,Array{Float64,1}}) at /Users/alexey/.julia/v0.5/MXNet/src/io.jl:324
 in MXNet.mx.ArrayDataProvider(::Pair{Symbol,Array{Float64,2}}, ::Pair{Symbol,Array{Float64,1}}) at /Users/alexey/.julia/v0.5/MXNet/src/io.jl:280

julia> xor_data = [ [1. 0 1 0]; [1 1 0 0] ]
2×4 Array{Float64,2}:
 1.0  0.0  1.0  0.0
 1.0  1.0  0.0  0.0

julia> mx.ArrayDataProvider(:data => xor_data, :labels => xor_labels)
MXNet.mx.ArrayDataProvider(Array{Float32,N}[
Float32[1.0 0.0 1.0 0.0; 1.0 1.0 0.0 0.0]],Symbol[:data],Array{Float32,N}[
Float32[0.0 1.0 1.0 0.0]],Symbol[:labels],4,4,false,0.0f0,0.0f0,MXNet.mx.NDArray[mx.NDArray{Float32}(2,4)],MXNet.mx.NDArray[mx.NDArray{Float32}(4,)])

Question:

I'm using julia to create a neural network with TensorFlow.

My networks runs but the error does not converge, here is the TensorBoard result:

To check my error function, I used the tutorial of Malmaud and I replaced "accuracy" by my function.

It works:

Following that, I think my network has a problem.

Can you help me ?

Here is my code:

ENV["CUDA_VISIBLE_DEVICES"] = "0" # It is to use the gpu
using TensorFlow
using Distributions

sess = Session(Graph())

batch_size = 30  
num_pixels = 64

###########

# Data base: 1000 arrays, the first array is fill with 1, the second with 2 etc...

arrays_data = zeros(Float32,1000,num_pixels,num_pixels)

arrays_labels = zeros(Float32,1000)

for k in 1:num_pixels, j in 1:num_pixels, i in 1:1000
        arrays_data[i,j,k] = i
end

for i in 1:1000
    arrays_labels[i] = i
end

###########

# inputs

x = placeholder(Float32, shape= [batch_size, 1, 1, 1])

y = placeholder(Float32)

###########

 # Function to create a batch

function create_batch(batch_size)
    x = zeros(Float32, batch_size,num_pixels, num_pixels)
    y = zeros(Float32, batch_size)

index = shuffle(1:1000) # To choose a random batch

    for i in 1:batch_size
        x[i, : ,:] = arrays_data[index[i],:,:]

        y[i] = arrays_labels[index[i]]
    end
    y, x
end


###########


# Summary to use TensorBoard

 summary = TensorFlow.summary

# Create the different layers ; poids = weight

variable_scope("mymodel" * randstring(), initializer=Normal(0, .001)) do
    global poids_1 = get_variable("p1", [2,2,2,1], Float32)
    global poids_2 = get_variable("p2",[4,4,3,2],Float32)
    global poids_3 = get_variable("p3",[2,2,4,3],Float32)
    global poids_4 = get_variable("p4",[1,4,4,4],Float32)
    global poids_5 = get_variable("p5",[1,4,4,4],Float32)
    global poids_6 = get_variable("p6",[1,4,4,4],Float32)
    global biases_1 = get_variable("b1",[2],Float32)
    global biases_2 = get_variable("b2",[3],Float32)
    global biases_3 = get_variable("b3",[4],Float32)
    global biases_4 = get_variable("b4",[4],Float32)
    global biases_5 = get_variable("b5",[4],Float32)
    global biases_6 = get_variable("b6",[4],Float32)
end

logits_1 = nn.relu(nn.conv2d_transpose(x, poids_1, [batch_size,2,2,2], [1,2,2,1],padding = "SAME") + biases_1)

logits_2 = nn.relu(nn.conv2d_transpose(logits_1,poids_2, [batch_size,4,4,3], [1,2,2,1],padding = "SAME") + biases_2)

logits_3 = nn.relu(nn.conv2d_transpose(logits_2,poids_3, [batch_size,8,8,4], [1,2,2,1],padding = "SAME") + biases_3)

logits_4 = nn.relu(nn.conv2d_transpose(logits_3,poids_4, [batch_size,16,16,4], [1,2,2,1],padding = "SAME") + biases_4)

logits_5 = nn.relu(nn.conv2d_transpose(logits_4,poids_5, [batch_size,32,32,4], [1,2,2,1],padding = "SAME") + biases_5)

 logits_6 = nn.relu(nn.conv2d_transpose(logits_5,poids_6, [batch_size,64,64,4], [1,2,2,1],padding = "SAME") + biases_6)

logits_6 = reduce_sum(logits_6, axis=[4])



logits = reshape(logits_6, [batch_size,num_pixels*num_pixels])  # Output of network




smax = nn.softmax(logits)



cross_entropy = reduce_mean(-reduce_sum(y.*log(smax))) # loss function

optimizer = train.AdamOptimizer(0.0001)

train_op = train.minimize(optimizer,cross_entropy)

error = (1/(num_pixels*num_pixels*batch_size)).*sqrt(sum((smax - y)^2))

 summary.histogram("Error",error)

 merged = summary.merge_all()

run(sess, global_variables_initializer())

# summary_writer = summary.FileWriter("Folder Path") # If you want use TensorBoard

# Train loop

for i in 1:500

batch = create_batch(batch_size)


x_ = run(sess, train_op, Dict(x => reshape(batch[1], (batch_size,1,1,1)), y => reshape(batch[2], (batch_size,64*64))))


  if i%100 == 1

    err = run(sess, error, Dict(x => reshape(batch[1], (batch_size,1,1,1)), y => reshape(batch[2], (batch_size,64*64))))
       info("train $i , error = $err")
       end

# If you use TensorBoard, please use the following commands

      # new = run(sess,merged, Dict(x => reshape(batch[1], (batch_size,1,1,1)), y => reshape(batch[2], (batch_size,64*64))))

      # write(summary_writer, new, i)

end

close(sess)

EDIT

The following code is working:

using TensorFlow
using Distributions

sess = Session(Graph())

batch_size = 30
num_pixels = 256

###########

# Data base: 10000 arrays, the first array is fill with 1, the second with 2 etc...

arrays_data = zeros(Float32,10000,num_pixels,num_pixels)

arrays_labels = zeros(Float32,10000)

for k in 1:num_pixels, j in 1:num_pixels, i in 1:10000
            arrays_data[i,j,k] = i
end


for i in 1:10000
    arrays_labels[i] = i
end

###########

# inputs

x = placeholder(Float32, shape= [batch_size, 1, 1, 1])

y = placeholder(Float32)

###########

 # Function to create a batch

function create_batch(batch_size)
    x = zeros(Float32, batch_size,num_pixels, num_pixels)
    y = zeros(Float32, batch_size)

index = shuffle(1:10000) # To choose a random batch

    for i in 1:batch_size
        x[i, : ,:] = arrays_data[index[i],:,:]

        y[i] = arrays_labels[index[i]]
    end
    y, x
end


###########


# Summary to use TensorBoard

 summary = TensorFlow.summary

# Create the different layers ; poids = weight

variable_scope("mymodel" * randstring(), initializer=Normal(0, .001)) do
    global poids_1 = get_variable("p1", [3,3,2,1], Float32)
    global poids_2 = get_variable("p2",[3,3,3,2],Float32)
    global poids_3 = get_variable("p3",[3,3,4,3],Float32)
    global poids_4 = get_variable("p4",[3,3,4,4],Float32)
    global poids_5 = get_variable("p5",[3,3,4,4],Float32)
    global poids_6 = get_variable("p6",[3,3,4,4],Float32)
    global poids_7 = get_variable("p7",[3,3,8,4],Float32)
    global poids_8 = get_variable("p8",[3,3,8,8],Float32)
    global biases_1 = get_variable("b1",[2],Float32)
    global biases_2 = get_variable("b2",[3],Float32)
    global biases_3 = get_variable("b3",[4],Float32)
    global biases_4 = get_variable("b4",[4],Float32)
    global biases_5 = get_variable("b5",[4],Float32)
    global biases_6 = get_variable("b6",[4],Float32)
    global biases_7 = get_variable("b7",[8],Float32)
    global biases_8 = get_variable("b8",[8],Float32)
end

logits_1 = nn.relu(nn.conv2d_transpose(x, poids_1, [batch_size,2,2,2], [1,2,2,1],padding = "SAME") + biases_1)

logits_2 = nn.relu(nn.conv2d_transpose(logits_1,poids_2, [batch_size,4,4,3], [1,2,2,1],padding = "SAME") + biases_2)

logits_3 = nn.relu(nn.conv2d_transpose(logits_2,poids_3, [batch_size,8,8,4], [1,2,2,1],padding = "SAME") + biases_3)

logits_4 = nn.relu(nn.conv2d_transpose(logits_3,poids_4, [batch_size,16,16,4], [1,2,2,1],padding = "SAME") + biases_4)

logits_5 = nn.relu(nn.conv2d_transpose(logits_4,poids_5, [batch_size,32,32,4], [1,2,2,1],padding = "SAME") + biases_5)

 logits_6 = nn.relu(nn.conv2d_transpose(logits_5,poids_6, [batch_size,64,64,4], [1,2,2,1],padding = "SAME") + biases_6)

logits_7 = nn.relu(nn.conv2d_transpose(logits_6,poids_7, [batch_size,128,128,8], [1,2,2,1],padding = "SAME") + biases_7)

logits_8 = nn.relu(nn.conv2d_transpose(logits_7,poids_8, [batch_size,256,256,8], [1,2,2,1],padding = "SAME") + biases_8)

logits_8 = reduce_sum(logits_8, axis=[4])


logits = reshape(logits_8, [batch_size,num_pixels*num_pixels])  # Output of network


# Don't use a softmax here...


least_square = reduce_mean(sqrt(sum((y - logits).^2))) # Loss function

optimizer = train.AdamOptimizer(0.0001)

train_op = train.minimize(optimizer,least_square)


error = sqrt(sum((y - logits).^2)./(num_pixels.*num_pixels.*batch_size))

 summary.histogram("Error",error)

 merged = summary.merge_all()

run(sess, global_variables_initializer())

# summary_writer = summary.FileWriter("Folder Path") # If you want use TensorBoard

# Train loop

for i in 1:1500

batch = create_batch(batch_size)


x_ = run(sess, train_op, Dict(x => reshape(batch[1], (batch_size,1,1,1)), y => reshape(batch[2], (batch_size,num_pixels*num_pixels))))


  if i%100 == 1

    err = run(sess, error, Dict(x => reshape(batch[1], (batch_size,1,1,1)), y => reshape(batch[2], (batch_size,num_pixels*num_pixels))))
       info("train $i , error = $err")
       end

# If you use TensorBoard, please use the following commands

       # newer = run(sess,merged, Dict(x => reshape(batch[1], (batch_size,1,1,1)), y => reshape(batch[2], (batch_size,num_pixels*num_pixels))))

       # write(summary_writer, newer, i)

end

close(sess)

Answer:

Does error need to be defined as a function?

Something like: error(smax, y) = (1/(num_pixels*num_pixels*batch_size)).*sqrt(sum((smax - y)^2))