Hot questions for Using Neural networks in nnet

Question:

Hello and thanks in advance. I'm using caret to cross validate a neural-network from the nnet package. In the method parameter for the trainControl function I can specify my cross-validation type, but all of these choose the observations at random to cross-validate against. Is there anyway I can use caret to cross-validate on specific observations in my data by either an ID or a hard-coded parameter? For example here's my current code:

library(nnet) 
library(caret) 
library(datasets) 

data(iris) 

train.control <- trainControl( 
    method = "repeatedcv" 
    , number = 4 
    , repeats = 10 
    , verboseIter = T 
    , returnData = T 
    , savePredictions = T 
    ) 

tune.grid <- expand.grid( 
    size = c(2,4,6,8)
    ,decay = 2^(-3:1) 
    ) 

nnet.train <- train( 
    x = iris[,1:4] 
    , y = iris[,5] 
    , method = "nnet" 
    , preProcess = c("center","scale")  
    , metric = "Accuracy" 
    , trControl = train.control 
    , tuneGrid = tune.grid 
    ) 
nnet.train 
plot(nnet.train)

Suppose I wanted to add another column CV_GROUP to the iris data frame and I wanted caret to cross-validate the neural-network on observations with a value of 1 for that column:

iris$CV_GROUP <- c(rep.int(0,times=nrow(iris)-20), rep.int(1,times=20))

Is this possible with caret?


Answer:

Use index and indexOut control options. I coded a way to implement this that let's you select the number of repeats and folds that you want:

library(nnet)
library(caret)
library(datasets)
library(data.table)
library(e1071)

r <- 2 # number of repeats
k <- 5 # number of folds
data(iris)
iris <- data.table(iris)

# Create folds and repeats here - you could create your own if you want #
set.seed(343)
for (i in 1:r) {
    newcol <- paste('fold.num',i,sep='')
    iris <- iris[,eval(newcol):=sample(1:k, size=dim(iris)[1], replace=TRUE)]
}

folds.list.out <- list()
folds.list <- list()
list.counter <- 1
for (y in 1:r) {
    newcol <- paste('fold.num', y, sep='')
    for (z in 1:k) {
        folds.list.out[[list.counter]] <- which(iris[,newcol,with=FALSE]==z)
        folds.list[[list.counter]] <- which(iris[,newcol,with=FALSE]!=z)
        list.counter <- list.counter + 1
    }
    iris <- iris[,!newcol,with=FALSE]
}

tune.grid <- expand.grid( 
    size = c(2,4,6,8)
    ,decay = 2^(-3:1) 
    ) 

train.control <- trainControl( 
    index=folds.list
    , indexOut=folds.list.out
    , verboseIter = T 
    , returnData = T 
    , savePredictions = T 
    ) 

iris <- data.frame(iris)

nnet.train <- train( 
    x = iris[,1:4] 
    , y = iris[,5] 
    , method = "nnet" 
    , preProcess = c("center","scale")  
    , metric = "Accuracy" 
    , trControl = train.control 
    , tuneGrid = tune.grid 
    ) 

nnet.train
plot(nnet.train)

Question:

I am trying to use nnet in R, and encounter a problem for using softmax.

I am trying to builda three layer network, with input layer have 25 neurons, hidden layer have 25 neurons, output layer only have one neuron. Here is how to reproduce the problem.

library('nnet')
X <- replicate(25, rnorm(40))
y <- sample(0:1,40,replace=T)

mynnet <- nnet(X, y, size = 25, 
                  softmax = T,
                  rang = 0.8, 
                  maxit = 2000, 
                  model=TRUE)

When I run this piece of code, I got a error:

Error in nnet.default(X, y, size = 25, 
softmax = T, rang = 0.8, maxit = 2000,  :
'softmax = TRUE' requires at least two response categories

What 'requires at least two response categories' means? And how to fix it? Thanks.


Answer:

softmax is meant for fitting classification networks with a factor response variable. If you have a real-valued response, you probably want to fit a regression neural network, which can be obtained with linout=TRUE.

Question:

I am trying to create a neural network using the python module nnet-ts. It has a built-in method named TimeSeriesNnet(), which takes two arguments; hidden_layers and activation_functions.

See documentation for this module, as well as example in the README.md:

https://github.com/hawk31/nnet-ts

I am running python version 2.7.13

The nnet-ts module has dependencies to 5 particular packages, which I am listing below together with the current versions I am using:

numpy-1.13.0, pandas-0.20.2, scipy-0.19.0, theano-0.9.0 and keras-2.0.5

Following the example in the README (link above), my code reads as follows:

from nnet_ts import *
neural_net = TimeSeriesNnet(hidden_layers = [7, 3], activation_functions = ['tanh', 'tanh'])

Execution of this code results in a NameError being thrown. Output:

NameError: name 'TimeSeriesNnet' is not defined

My guess is that the reason for this error could have something to do with different module versions, as the code is nearly identical to the example given in the README. Any help is much appreciated.


Answer:

Definitely, there is something wrong with your deployment of the code. For your Python developments, I suggest always use a local virtual environment. For this Neural Network Timeseries, you can build it locally with the python setup.py build command, it will build necessary files in the build/lib.linux-x86_64-2.7 directory. Assuming you have all dependencies, there shouldn't be any problem. In the deployment, you just have to set your PYTHONPATH to this directory, or you just include this path in the runtime with sys.path.insert() method. For example, assuming I am in nnet-ts directory, I have built and run nnet-ts as follow:

    $ python setup.py build
    $ python
    > import sys
    > sys.path.insert(0, 'build/lib.linux-x86_64-2.7')
    > from nnet_ts import *
    > time_series = np.array(pd.read_csv("nnet_ts/AirPassengers.csv")["x"]) 
    > neural_net = TimeSeriesNnet(hidden_layers = [20, 15, 5], activation_functions = ['sigmoid', 'sigmoid', 'sigmoid'])

Question:

I have fitted an Averaged neural network in R with Caret. See code below. Does the term Averaged mean that the average is based on the outcomes of 1000 neural networks? (since there are 1000 iterations in this case)

Thanks.

library(AppliedPredictiveModeling)
data(solubility)

### Create a control funciton that will be used across models. We
### create the fold assignments explictily instead of relying on the
### random number seed being set to identical values.

library(caret)
set.seed(100)
indx <- createFolds(solTrainY, returnTrain = TRUE)
ctrl <- trainControl(method = "cv", index = indx)

################################################################################
### Section 7.1 Neural Networks

### Optional: parallel processing can be used via the 'do' packages,
### such as doMC, doMPI etc. We used doMC (not on Windows) to speed
### up the computations.

### WARNING: Be aware of how much memory is needed to parallel
### process. It can very quickly overwhelm the availible hardware. We
### estimate the memory usuage (VSIZE = total memory size) to be 
### 2677M/core.

library(doMC)
registerDoMC(10)


library(caret)

nnetGrid <- expand.grid(decay = c(0, 0.01, .1), 
                        size = c(1, 3, 5, 7, 9, 11, 13), 
                        bag = FALSE)

set.seed(100)
nnetTune <- train(x = solTrainXtrans, y = solTrainY,
                  method = "avNNet",
                  tuneGrid = nnetGrid,
                  trControl = ctrl,
                  preProc = c("center", "scale"),
                  linout = TRUE,
                  trace = FALSE,
                  MaxNWts = 13 * (ncol(solTrainXtrans) + 1) + 13 + 1,
                  maxit = 1000,
                  allowParallel = FALSE)
nnetTune

plot(nnetTune)

testResults <- data.frame(obs = solTestY,
                          NNet = predict(nnetTune, solTestXtrans))

################################################################################

See also:

https://scientistcafe.com/post/nnet.html


Answer:

avNNet is a model where the same neural network model is fit using different random number seeds. All the resulting models are used for prediction. For regression, the output from each network are averaged. For classification, the model scores are first averaged, then translated to predicted classes. Source.

The number of models fit is controlled by the argument repeats which is passed down to the model in caret via ...

repeats - the number of neural networks with different random number seeds. At default this is set to 5. So five models will be averaged. In caret's definition of the model I do not see this changing.

If the bag argument is set to TRUE model fitting and aggregation is performed by bootstrap aggregation, which in my opinion is almost guaranteed to provide better predictive performance if the number of models is high enough.

Question:

I want to fit a neural network model using caret package.There are 208 predictors all of which are important and cannot be discarded. The maximum value that I can give to the size parameter is 4 beyond which I get an error saying that there are too many weights.

> ctrl<-trainControl(method = 'cv',number = 5)
> my.grid <- expand.grid(.decay = 0.1, .size =5)
> nn.fit <- train(train_predictors,train_responses[["r2c1"]],method = "nnet",algorithm = 'backprop', tuneGrid = my.grid,trace=F, linout = TRUE,trControl = ctrl)
Something is wrong; all the RMSE metric values are missing:
      RMSE        Rsquared        MAE     
 Min.   : NA   Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA   Max.   : NA  
 NA's   :1     NA's   :1     NA's   :1    
Error: Stopping
In addition: Warning messages:
1: model fit failed for Fold1: decay=0.1, size=5 Error in nnet.default(x, y, w, ...) : too many (1051) weights

2: model fit failed for Fold2: decay=0.1, size=5 Error in nnet.default(x, y, w, ...) : too many (1051) weights

3: model fit failed for Fold3: decay=0.1, size=5 Error in nnet.default(x, y, w, ...) : too many (1051) weights

4: model fit failed for Fold4: decay=0.1, size=5 Error in nnet.default(x, y, w, ...) : too many (1051) weights

5: model fit failed for Fold5: decay=0.1, size=5 Error in nnet.default(x, y, w, ...) : too many (1051) weights

6: In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.

The model performs very badly with 4 neurons(size=4).What can I do to make the model work if I want to have more than 5 neurons?


Answer:

You can always use the ... optional argument from caret train method to pass additional arguments to the underlying training method (in this case nnet). The CRAN documentation of the nnet package describes the MaxNwts parameter that allow to take control over the maximum number of hidden units.

Question:

In the description of nnet in CRAN projects(https://cran.r-project.org/web/packages/nnet/nnet.pdf) it says that nnet fits a single hidden layer:

 Description: Fit single-hidden-layer neural network, possibly with skip-layer connections

Is it possible for me to specify the number of hidden layers using nnet? My understanding was that my selection of hidden layers and number of neurons in the hidden layer were the parameters that can be changed to improve a model. Is it true that it could help a model to add/remove hidden layers? Or, are there separate areas of application of single-layered and multi-layered neural networks?

I am new to ANN. I am working on a classification model with training sample size of 55000 x 54. Thanks in advance!


Answer:

Simple answer NO, nnet always has a single layer where you specify the number of nodes. You can find more information in a similar question here. You will need to use other packages such as neuralnet or something more sophisticated like h20 or MXNet.

Regarding parameters to improve the model, there are many different parts of neural networks beyond the raw architecture (i.e. layers, nodes). These include optimization functions, activation functions, batchsizes, among many others. You likely want to consult some further resources on using neural networks.