How to convert a huge list-of-vector to a matrix more efficiently?

list.as.matrix package
convert list to matrix python
r convert list to dataframe
large list to dataframe r
convert list to matrix mathematica
convert to list in r
convert list to numeric matrix r
create matrix from list python

I have a list of length 130,000 where each element is a character vector of length 110. I would like to convert this list to a matrix with dimension 1,430,000*10. How can I do it more efficiently?\ My code is :

output=NULL
for(i in 1:length(z)) {
 output=rbind(output,
              matrix(z[[i]],ncol=10,byrow=TRUE))
}

This should be equivalent to your current code, only a lot faster:

output <- matrix(unlist(z), ncol = 10, byrow = TRUE)

How to Create a Data Frame from Scratch in R, To convert a huge list of a vector to a matrix, you can use the unlist() function before creating a matrix as follows: output <- matrix(unlist(z), ncol  Stack Overflow for Teams is a private, secure spot for you and your coworkers to find and share information. Learn more How to convert a huge list-of-vector to a matrix more efficiently?

I think you want

output <- do.call(rbind,lapply(z,matrix,ncol=10,byrow=TRUE))

i.e. combining @BlueMagister's use of do.call(rbind,...) with an lapply statement to convert the individual list elements into 11*10 matrices ...

Benchmarks (showing @flodel's unlist solution is 5x faster than mine, and 230x faster than the original approach ...)

n <- 1000
z <- replicate(n,matrix(1:110,ncol=10,byrow=TRUE),simplify=FALSE)
library(rbenchmark)
origfn <- function(z) {
    output <- NULL 
    for(i in 1:length(z))
        output<- rbind(output,matrix(z[[i]],ncol=10,byrow=TRUE))
}
rbindfn <- function(z) do.call(rbind,lapply(z,matrix,ncol=10,byrow=TRUE))
unlistfn <- function(z) matrix(unlist(z), ncol = 10, byrow = TRUE)

##          test replications elapsed relative user.self sys.self 
## 1   origfn(z)          100  36.467  230.804    34.834    1.540  
## 2  rbindfn(z)          100   0.713    4.513     0.708    0.012 
## 3 unlistfn(z)          100   0.158    1.000     0.144    0.008 

If this scales appropriately (i.e. you don't run into memory problems), the full problem would take about 130*0.2 seconds = 26 seconds on a comparable machine (I did this on a 2-year-old MacBook Pro).

Slow Matrices? Try Lists., How do I convert a list to a vector in R? I have a list of length 130,000 where each element is a character vector of length 110. I would like to convert this list to a matrix with dimension 1,430,000*10.

It would help to have sample information about your output. Recursively using rbind on bigger and bigger things is not recommended. My first guess at something that would help you:

z <- list(1:3,4:6,7:9)
do.call(rbind,z)

See a related question for more efficiency, if needed.

LongVectors: Long Vectors, How do I convert a vector to a Dataframe in R? Sometimes in R, I have a list z of length with each element being a vector of the same length .I want to convert this list into an matrix, where each element of the list corresponds to a column in the matrix.

You can also use,

output <- as.matrix(as.data.frame(z))

The memory usage is very similar to

output <- matrix(unlist(z), ncol = 10, byrow = TRUE)

Which can be verified, with mem_changed() from library(pryr).

list.as.matrix: Convert list to matrix in sprof: Profiling, timing and , How do I convert a list into a Dataframe in R? Hi all, I'm looking for a way to automatically convert a list of values into a matrix. For example, A1 A1 1000 A1 A2 998 A1 A3 468 A1 A4 491 A2 A1 998 A2 A2 1000 A2 A3 464 A2 A4 488 A3 A1 468 A3 A2 464 A3 A3 1000 A3 A4 992 A4 A1 491 A4 A2 488 A4 A3 992 A4 A4 1000 should be converted to A1

you can use as.matrix as below:

output <- as.matrix(z)

Using vectors and matrices in R, List data structures can be more efficient than their matrix and array counterparts. But data frames have big performance problems of their own. LIST) will transform any list containing equal length vectors into a matrix,  Goal: from a list of vectors of equal length create a matrix where each vector becomes a row. Example: > a <- list() 10 1 2 3 4 5

1. Vectors, Matrices, and Arrays, Vectors of 2^31 or more elements were added in R 3.0.0. Currently all atomic (​raw, logical, integer, numeric, complex, character) vectors, lists and expressions can Arrays (including matrices) can be based on long vectors provided each of and even then factors will be more efficient (4 bytes per element rather than 8). I have a data.frame that looks like this. x a 1 . x b 2 . x c 3 . y a 3 . y b 3 . y c 2 . I want this in matrix form so I can feed it to heatmap to make a plot.

Subsetting · Advanced R., a list of numeric vectors. byrow. boolean. Arrange list entries as rows. Default is to use columns. filler. Say you want to convert a matrix to a list, where each element of the list contains one column. list() or as.list() obviously won't work, and until now I use a hack using the behavior of tapply : x <- matrix(1:10,ncol=2)

Chapter 2 Working with vectors, matrices, and arrays, It was mentioned earlier that all the elements of a vector must be of the same mode. as a list that can store different types of objects without having to change their with data of a single mode, a matrix may be more efficient than a data frame. data frame, and is very handy to check to make sure a large data.frame really  The built-in matrix function has the nice option to enter data byrow.Combine that with an unlist on your source list will give you a matrix. We also need to specify the number of rows so it can break up the unlisted data.

Comments
  • If you want the dimensions to be 1430000*11 why do you set ncol to be 10?
  • Wait- when you say that each entry has 11 characters, you mean that it is a vector with 11 items? I originally thought that each was a string with 11 characters in it. Can you show z[1:2] as an example?
  • Thank Dason and David! That's a typo. I have corrected it.
  • @user1787675: I still don't understand. What is an "entry"? Is it a vector? Can you show z[1:2]?
  • Hi David, I looked up an dictionary and found that I mean the components in the list. I am sorry for the confusion I caused. I am not good at English :)
  • Bingo. This should be much faster than my solution too, but I couldn't think of it fast enough.
  • +1, but I'd recommend setting USE.NAMES=FALSE in unlist in order to save time and memory.
  • It should be use.names (i.e. in lowercase).
  • Just to clarify, it should be output <- matrix(unlist(z), ncol = 10, byrow = TRUE, use.names=FALSE) to be the most efficient.
  • That's magical! It takes about 20 seconds to do this on my one-year-old toshiba machine, which saves me a lot of time. And your function to show the run time is very interesting too.