Calling np.sum(np.fromiter(generator))

I have a generator that returns numpy arrays. For example sake, let it be:

import numpy as np
a = np.arange(9).reshape(3,3)
gen = (x for x in a)

Calling:

np.sum(gen)

On numpy 1.17.4:

DeprecationWarning: Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.fromiter(generator)) or the python sum builtin instead.

Trying to refactor the above:

np.sum(np.fromiter(gen, dtype=np.ndarray))

I get:

ValueError: cannot create object arrays from iterator

What is wrong in the above statement?

Since you're summing instances of arrays you can just use the built-in sum:

result = sum(gen)

DeprecationWarning: numpy sum :: np.sum(generator) is , Warning: score += np.sum(cnt * logsumexp(Elogthetad + Elogbeta[:, int(id)]) DeprecationWarning: Calling np.sum(generator) is deprecated, and in th. Use np.sum(np.fromiter(generator)) or the python sum builtin instead. Use np.sum (np.fromiter (generator)) or the python sum builtin instead. score += np.sum (cnt * logsumexp (Elogthetad + Elogbeta [:, int (id)]) for id, cnt in doc) Any solution for this ?

The problem is the second argument, np.ndarray in the fromiter(). Numpy fromiter expected a 1D and returns a 1D array:

Create a new 1-dimensional array from an iterable object.

Therefore, you cannot create object arrays from iterator. Furthermore the .reshape() will also raise an error, because of what I stated in the first line. All in all, this works:

import numpy as np
a = np.arange(9)
gen = (x for x in a)
print(np.sum(np.fromiter(gen,float)))

Output:

36

DeprecationWarning: Calling np.sum(generator) is deprecated , DeprecationWarning: Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.fromiter(generator))� numpy.fromiter¶ numpy.fromiter (iterable, dtype, count=-1) ¶ Create a new 1-dimensional array from an iterable object. Parameters iterable iterable object. An iterable object providing data for the array.

What about simply converting your generator to a list and then passing it to the np.sum?

a = np.arange(9).reshape(3,3)
gen = (x for x in a)

Summing all the elements:

>>> np.sum(list(gen))
36

Summing column-wise:

>>> np.sum(list(gen), axis=0)
array([ 9, 12, 15])

Summing row-wise:

>>> np.sum(list(gen), axis=1)
array([ 3, 12, 21])

Calling np.sum(np.fromiter(generator)), DeprecationWarning: Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.fromiter(generator))� Fix #2270. Since np.sum(generator) is deprecated, it is replaced by python inbuilt sum.

numpy.fromiter — NumPy v1.19 Manual, It allows fromiter to pre-allocate the output array, instead of resizing it on iterable = (x*x for x in range(5)) >>> np.fromiter(iterable, float) array([� return - np.sum(p * np.math.log(p) for p in probs if p != 0) DeprecationWarning: Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.fromiter(generator)) or the python sum builtin instead.

Python deprecation warning about sum function, gensim calling np sum generator is deprecated numpy fromiter 2d Use np. sum(np.from_iter(generator)) or the python sum builtin instead. obje_1=detmas. Use np. sum ( np. fromiter ( generator )) or the python sum builtin instead . >>> np. all ( gen) # Returns the generator <generator object <genexpr> at 0x7f6aa64d1e40> >>> np. mean ( gen) # Raises TypeError TypeError: unsupported operand type ( s) for /: 'generator' and 'int'. Copy link.

Use np.sum (np.from_iter (generator)) or the python sum builtin instead. score += np.sum (cnt * logsumexp (Elogthetad + Elogbeta [:, int (id)]) for id, cnt in doc) This comment has been minimized. Sign in to view. Copy link.

Comments
  • For the given example, what's the expected result? Is it 36 (summing all the elements) or is it [ 3, 12, 21] (summing row-wise)?
  • Note that this gives the column-wise sum of the array a, i.e. the result is [ 9, 12, 15]. P.s. I didn't downvote your answer.
  • @AndreasK. This gives exactly the same result as the OP's code which raised a warning: np.sum(gen). And I cannot imagine what other result you would want to have (except of course summing everything together which is trivial by doing sum(gen).sum()). Since the generator yields rows of a I cannot imagine that the OP expected a row-wise sum, since that's just another generator: (x.sum() for x in gen).
  • The error in the OP has nothing to do with dimensionality of the input. You just can't create object arrays from iterators (because it's not implemented), e.g. np.fromiter(iter(range(3)), dtype=object). Reasons are probably because object arrays need to deal with reference counting rather than convert the items to an internal data type. If you specified dtype=float only then you would run into the dimension problem. However I don't see how your answer addresses the OP's question. The OP states that they have a generator that returns arrays while you return numbers (this is just a.sum()).
  • I think we're addressing different issues. The title of the questions refers to np.sum(np.fromiter()) and that's why I'm answering with the same tools OP uses. It seems you understood perfectly the intent of OPs answer because of the accepted answer, however there is no the relation to the title, nor making the functions OP asks about work.