Recursive version of 'reload'

recursion java
recursion python
recursive function
recursion c++
recursive function example
recursion linguistics
recursion in c
recursion trick

When I'm developing Python code, I usually test it in an ad-hoc way in the interpreter. I'll import some_module, test it, find a bug, fix the bug and save, and then use the built-in reload function to reload(some_module) and test again.

However, suppose that in some_module I have import some_other_module, and while testing some_module I discover a bug in some_other_module and fix it. Now calling reload(some_module) won't recursively re-import some_other_module. I have to either manually reimport the dependency (by doing something like reload(some_module.some_other_module), or import some_other_module; reload(some_other_module), or, if I've changed a whole bunch of dependencies and lost track of what I need to reload, I need to restart the entire interpreter.

What'd be more convenient is if there were some recursive_reload function, and I could just do recursive_reload(some_module) and have Python not only reload some_module, but also recursively reload every module that some_module imports (and every module that each of those modules imports, and so on) so that I could be sure that I wasn't using an old version of any of the other modules upon which some_module depends.

I don't think there's anything built in to Python that behaves like the recursive_reload function I describe here, but is there an easy way to hack such a thing together?

I've run up against the same issue, and you inspired me to actually solve the problem.

from types import ModuleType

try:
    from importlib import reload  # Python 3.4+
except ImportError:
    # Needed for Python 3.0-3.3; harmless in Python 2.7 where imp.reload is just an
    # alias for the builtin reload.
    from imp import reload

def rreload(module):
    """Recursively reload modules."""
    reload(module)
    for attribute_name in dir(module):
        attribute = getattr(module, attribute_name)
        if type(attribute) is ModuleType:
            rreload(attribute)

Or, if you are using IPython, just use dreload or pass --deep-reload on startup.

What Is Recursion in Java Programming?, What is the difference between recursive and non recursive function? adjective. pertaining to or using a rule or procedure that can be applied repeatedly. Mathematics, Computers. pertaining to or using the mathematical process of recursion: a recursive function; a recursive procedure.

Wouldn't it be simpler to actually write some test cases and run them every time you are done with modifying your module?

What you are doing is cool (you are in essence using TDD (test driven development) but you are doing it wrong.

Consider that with written unit tests(using the default python unittest module, or better yet nose) you get to have tests that are reusable, stable and help you detect inconsitencies in your code much much faster and better than with testing your module in the interactive environment.

Recursive Algorithm, Recursion (adjective: recursive) occurs when a thing is defined in terms of itself or of its type. Recursion is used in a variety of disciplines ranging from linguistics​  The function calls itself recursively on a smaller version of the input (n - 1) and multiplies the result of the recursive call by n, until reaching the base case, analogously to the mathematical definition of factorial. Recursion in computer programming is exemplified when a function is defined in terms of simpler, often smaller versions of

I've run against the same issue and I've built up on @Mattew and @osa answer.

from types import ModuleType
import os, sys
def rreload(module, paths=None, mdict=None):
    """Recursively reload modules."""
    if paths is None:
        paths = ['']
    if mdict is None:
        mdict = {}
    if module not in mdict:
        # modules reloaded from this module
        mdict[module] = [] 
    reload(module)
    for attribute_name in dir(module):
        attribute = getattr(module, attribute_name)
        if type(attribute) is ModuleType:
            if attribute not in mdict[module]:
                if attribute.__name__ not in sys.builtin_module_names:
                    if os.path.dirname(attribute.__file__) in paths:
                        mdict[module].append(attribute)
                        rreload(attribute, paths, mdict)
    reload(module)
    #return mdict

There are three differences:

  1. In the general case, reload(module) has to be called at the end of the function as well, as @osa pointed out.
  2. With circular import dependencies the code posted earlier would loop forever so I've added a dictionary of lists to keep track of the set of modules loaded by other modules. While circular dependencies are not cool, Python allows them, so this reload function deals with them as well.
  3. I've added a list of paths (default is ['']) from which the reloading is allowed. Some modules don't like been reloaded the normal way, (as shown here).

What is the difference between recursive function and non recursive , Recursive functions will add additionally elements on the stack. This can potentially cause stackoverflow errors depending on the state of the  Can somebody give me advice on how to create a recursive version of GetEnumerator()? The well-known Towers of Hanoi problem may serve as an example that is comparable to the actual problem I have.

The code worked great for dependency modules imported just as import another_module, but it failed when the module imported functions with from another_module import some_func.

I expanded on @redsk's answer to try and be smart about these functions. I've also added a blacklist because unfortunately typing and importlib don't appear in sys.builtin_module_names (maybe there are more). Also I wanted to prevent reloading of some dependencies I knew about.

I also track the reloaded module names and return them.

Tested on Python 3.7.4 Windows:

def rreload(module, paths=None, mdict=None, base_module=None, blacklist=None, reloaded_modules=None):
    """Recursively reload modules."""
    if paths is None:
        paths = [""]
    if mdict is None:
        mdict = {}
    if module not in mdict:
        # modules reloaded from this module
        mdict[module] = []
    if base_module is None:
        base_module = module
    if blacklist is None:
        blacklist = ["importlib", "typing"]
    if reloaded_modules is None:
        reloaded_modules = []
    reload(module)
    reloaded_modules.append(module.__name__)
    for attribute_name in dir(module):
        attribute = getattr(module, attribute_name)
        if type(attribute) is ModuleType and attribute.__name__ not in blacklist:
            if attribute not in mdict[module]:
                if attribute.__name__ not in sys.builtin_module_names:
                    if os.path.dirname(attribute.__file__) in paths:
                        mdict[module].append(attribute)
                        reloaded_modules = rreload(attribute, paths, mdict, base_module, blacklist, reloaded_modules)
        elif callable(attribute) and attribute.__module__ not in blacklist:
            if attribute.__module__ not in sys.builtin_module_names and f"_{attribute.__module__}" not in sys.builtin_module_names:
                if sys.modules[attribute.__module__] != base_module:
                    if sys.modules[attribute.__module__] not in mdict:
                        mdict[sys.modules[attribute.__module__]] = [attribute]
                        reloaded_modules = rreload(sys.modules[attribute.__module__], paths, mdict, base_module, blacklist, reloaded_modules)
    reload(module)
    return reloaded_modules

Some notes:

  1. I don't know why some builtin_module_names are prefixed with an underscore (for example collections is listed as _collections, so I have to do the double string check.
  2. callable() returns True for classes, I guess that's expected but that was one of the reasons I had to blacklist extra modules.

At least now I am able to deep reload a module at runtime and from my tests I was able to go multiple levels deep with from foo import bar and see the result at each call to rreload()

(Apologies for the long and ugly depth, but the black formatted version doesn't look so readable on SO)

Recursion, Let's have a look at an iterative version of the factorial function. def iterative_factorial(n): result = 1 for i in range(2,n+1): result *= i return result. It is common practice  The adjective "recursive" originates from the Latin verb "recurrere", which means "to run back". And this is what a recursive definition or a recursive function does: It is "running back" or returning to itself.

Technically, in each file you could put a reload command, to ensure that it reloads each time it imports

a.py:

def testa():
    print 'hi!'

b.py:

import a
reload(a)
def testb():
    a.testa()

Now, interactively:

import b
b.testb()
#hi!

#<modify a.py>

reload(b)
b.testb()
#hello again!

what is the difference between a recursive version of "find" and a not , Let's have a look at an iterative version of the factorial function. def iterative_factorial(n): result = 1 for i in range(2,n+1): result *= i return result. The Pitfalls  A good developer will construct his recursive solution, if possible, in such a manner that it is tail recursive. A good compiler will recognize a tail-recursive construct and optimize it into iteration. Tail Recursion is a special case of recursion where the last operation of the recursive function is the recursive call. Such a construct may be

Python Tutorial: Recursive Functions, Recursion is the process of defining a problem (or the solution to a problem) in terms of (a simpler version of) itself. For example, we can define the operation "​find  Tree created using the Logo programming language and relying heavily on recursion. Each branch can be seen as a smaller version of a tree. Each branch can be seen as a smaller version of a tree. Programming paradigms

Python2 Tutorial: Recursive Functions, Recursion is a common technique used in divide and conquer algorithms. The most common example of this is the Merge Sort, which recursively divides an array  The Selection Sort algorithm sorts maintains two parts.. First part that is already sorted; Second part that is yet to be sorted. The algorithm works by repeatedly finding the minimum element (considering ascending order) from unsorted part and putting it at the end of sorted part.

Programming - Recursion, Learn how to work with recursion in your Python programs by mastering A data structure is recursive if it can be defined in terms of a smaller version of itself. Recursive Insertion Sort Insertion sort is a simple sorting algorithm that works the way we sort playing cards in our hands. Below is an iterative algorithm for insertion sort

Comments
  • possible duplicate of how to find list of modules which depend upon a specific module in python
  • Actually, now that I am experimenting with it, we may want to call reload(module) twice; before and after reloading submodules. We need to call it before in case it refers any new modules. We need to call it after to handle statements such as from X import Y. If X has just been reloaded, we still need to reimport Y. I suspect it can be much more tricky, so we'd need to keep reloading until the dust settles.
  • The above implementation of rreload can lead to infinite recursion (in my case, within the threading module). Consider a graph DFS modification: gist.github.com/shwang/09bd0ffef8ef65af817cf977ce4698b8
  • importlib.reload would work after this merge. github.com/streamlit/streamlit/pull/537/files/…
  • you should put reload(module) after rreload all sub-modules
  • +1, It sounds like this testing methodology has been outgrown. It doesn't need to even be real cases, just have a small script that keep as a scratch pad for testing code (if the project is small enough that full testing is overkill).
  • Irony: I was provoked to ask this question after I hit upon this issue while testing some unit tests I was writing for another application. If I were to apply your advice here, I'd end up infinitely recursing and creating endless layers of tests. ;)
  • More seriously: proper tests of course have their place, but sometimes a more casual approach is called for (even if it's in addition to, rather than instead of, proper tests) and part of what's nice about Python is that it makes that casual approach easy. Maybe I've written half a function and want to check that so far it outputs what I expect, or maybe I've just chucked in a whole bunch of print statements into some tricky code and want to run the function on some perverse parameters that I specify and see what gets printed. Unit tests don't apply in those scenarios.
  • To put that another way: unit tests are good tools for confirming that code works - or for detecting errors in places where you don't expect them to exist - with minimal human effort. They're not good tools for fiddling with code that you are currently building from scratch and already know to be unfinished or broken, and it's in that latter situation that I use the interactive interpreter.
  • Sure, I could do this, but then my actual code files would be ugly and inefficient. I don't mind using silly hacks when I'm playing in the interactive interpreter, but I'd prefer not to introduce a silly hack into every file in my code just to let me be a bit lazier when I'm in the interpreter. :)
  • A cautious -1 because you haven't explained any advantage of this over redsk's version. You say it's "simplified for the user" but your version and redsk's both allow the caller to just call rreload(some_module); where's the simplification? Maybe there's some value here I'm not appreciating, but if so, you've hidden it well.
  • @MarkAmery, long time has passed since then, but I think the two "advantages for the user" can be expressed as: 1. All needed imports are embedded in the call, and; 2. there is an exception thrown if the __init__.py file is missing. For the second, it means that the developer is informed and can fix the problem, rather than the sub-module being ignored. Of course, it maybe makes sense to merge the two alternatives in a single solution.