Yield in a recursive function

yield from
generator function can have multiple yield expressions
yield from recursive
generator function recursion
yield from iterable
python yield from if
python yield multiple values
python recursive function in class

I am trying to do something to all the files under a given path. I don't want to collect all the file names beforehand then do something with them, so I tried this:

import os
import stat

def explore(p):
  s = ''
  list = os.listdir(p)
  for a in list:
    path = p + '/' + a
    stat_info = os.lstat(path )
    if stat.S_ISDIR(stat_info.st_mode):
     explore(path)
    else:
      yield path

if __name__ == "__main__":
  for x in explore('.'):
    print '-->', x

But this code skips over directories when it hits them, instead of yielding their contents. What am I doing wrong?

Use os.walk instead of reinventing the wheel.

In particular, following the examples in the library documentation, here is an untested attempt:

import os
from os.path import join

def hellothere(somepath):
    for root, dirs, files in os.walk(somepath):
        for curfile in files:
            yield join(root, curfile)


# call and get full list of results:
allfiles = [ x for x in hellothere("...") ]

# iterate over results lazily:
for x in hellothere("..."):
    print x

How to Yield in a recursive function in Python, Use yield from with your recursive call, otherwise you are just ignoring the result of the recursive call: def boil_down_array(key, data): if  You need to yield each of the items yielded by the recursive call: public static IEnumerable<Control> GetDeepControlsByType<T> (this Control control) { foreach(Control c in control.Controls) { if (c is T) { yield return c; } if(c.Controls.Count > 0) { foreach (Control control in c.GetDeepControlsByType<T> ()) { yield return control; } } } }

Iterators do not work recursively like that. You have to re-yield each result, by replacing

explore(path)

with something like

for value in explore(path):
    yield value

Python 3.3 added the syntax yield from X, as proposed in PEP 380, to serve this purpose. With it you can do this instead:

yield from explore(path)

If you're using generators as coroutines, this syntax also supports the use of generator.send() to pass values back into the recursively-invoked generators. The simple for loop above would not.

The power of Python's yield, P.S. Notice that where the original permute function called itself (the recursive call) a slight modification had to be done: by invoking yield, the permute function is  Now use these functions to recursively get all files within a directory and all its subdirectories (using generators): def get_files_recursive(directory): for file in get_files(directory): yield file for subdirectory in get_directories(directory): for file in get_files_recursive(subdirectory): # here the recursive call yield file

Can generators be recursive?, calls itself in a for loop, which sees an iterator and automatically uses the iteration protocol on it, so it actually gets values from it. And the reason for that is that you can only "yield return" a single element at a time, so you must iterate each time on the entire result of the recursive call. (You can't "yield return TopDown();"). I used a "dirty" trick to do it, which works like a charm 2.

Change this:

explore(path)

To this:

for subpath in explore(path):
    yield subpath

Or use os.walk, as phooji suggested (which is the better option).

Python Tutorial: Generators, function is a function defined in terms of itself via self-referential expressions. This means that the function will continue to call itself and repeat its behavior until some condition is met to return a result. Yieldis a keyword that ‘returns’ single value to the sequence, but does not stop the generator function. Interesting thing with generators is that they generate values on demand- meaning, values

That calls explore like a function. What you should do is iterate it like a generator:

if stat.S_ISDIR(stat_info.st_mode):
  for p in explore(path):
    yield p
else:
  yield path

EDIT: Instead of the stat module, you could use os.path.isdir(path).

Thinking Recursively in Python – Real Python, A helper function to read only files from a directory: def get_files(path): for file in listdir(path): full_path = join(path, file) if isfile(full_path): if exists(full_path): yield  Description. The yield keyword pauses generator function execution and the value of the expression following the yield keyword is returned to the generator's caller. It can be thought of as a generator-based version of the return keyword. yield can only be called directly from the generator function that contains it.

Python yield, Generators and Generator Expressions, I built a tree structure with a recursive function. Then I tried to use a recursive function to sum up some values across the tree, yield c.value() The yield statement suspends function’s execution and sends a value back to the caller, but retains enough state to enable function to resume where it is left off. When resumed, the function continues execution immediately after the last yield run.

Python Language, Generators vs Python Functions. How to create Generators in Python? Generators and For Loop. Recursive Generators in Python. Python Generator Expression  When a yield return statement is reached in the iterator method, expression is returned, and the current location in code is retained. Execution is restarted from that location the next time that the iterator function is called. You can use a yield break statement to end the iteration. For more information about iterators, see Iterators.

A thing I learned about Python recursion, The yield operator is called within the generator function to return single values back to the caller. The generator remembers the state of the previous call, so subsequent yields will return the next logical value. The caller uses the next() method to get each subsequent value from the generator function. For instance, the recursive Fibonacci number function is a very inefficient way to compute the number. Think about fibo(5) : It calls fibo(3) and fibo(4) . The former calls fibo(1) and fibo(2

Comments
  • Some languages can yield an entire sequence, not just individual elements. I do not think Python is one of them. mindscapehq.com/blog/index.php/2011/02/28/…
  • Since the title suggests a more general problem than can be solved by os.walk, consider this: def explore(p): if isinstance(p, (list, tuple)): for x in p: explore(p) else: yield p This has the same problem. Why doesn't it work?
  • Giving working code is good, but explaining what the OP did wrong, especially when they ask for that, is even better.
  • the question is about yield and recursion and not about best way to implement the os.walk
  • also : in Python 2 walk is slower then listdir, see python.org/dev/peps/pep-0471
  • This should be the accepted answer IMHO, as the question is about yield and recursion and not about best way to implement the os.walk ;-) !!! I was breaking my head on this very simple loop... And actually all of the other answers are on the same line...
  • Thanks man! Mentioning of 3.3 and iterators specific was really useful.
  • As mentioned in this talk at PyCon2014 generators can be used to bypass the recursion limit! youtube.com/watch?v=D1twn9kLmYg