Save workspace in IPython

how to save python session
jupyter notebook session
python save and load
how to save dataframe in jupyter notebook
pycharm save all variables
pickle save var
how to save variable value in python
save python variables to pickle

Is it possible to save an IPython workspace (defined functions, different kinds of variables, etc) so that it can be loaded later?

This would be a similar function to save.image() in MATLAB or R. Similar questions has been asked before, such as:

Save session in IPython like in MATLAB?

However, since a few years passed, I am wondering if there is a good solution now.

EDIT: this answer (and gist) has been modified to work for IPython 6

I added a somewhat ad-hoc solution that automates the process of storing/restoring user space variables using the underlying code from IPython's %store magic which is from what I understand what you wanted. See the gist here. Note that this only works for objects that can be pickled.

I can't guarantee its robustness, especially if any of the autorestore mechanisms in IPython change in the future, but it has been working for me with IPython 2.1.0. Hopefully this will at least point you in the right direction.

To reiterate the solution here:
  1. Add the save_user_variables.py script below to your ipython folder (by default $HOME/.ipython). This script takes care of saving user variables on exit.
  2. Add this line to your profile's ipython startup script (e.g., $HOME/.ipython/profile_default/startup/startup.py):

    get_ipython().ex("import save_user_variables;del save_user_variables")

  3. In your ipython profile config file (by default $HOME/.ipython/profile_default/ipython_config.py) find the following line:

    # c.StoreMagics.autorestore = False

    Uncomment it and set it to true. This automatically reloads stored variables on startup. Alternatively you can reload the last session manually using %store -r.

save_user_variables.py
def get_response(quest,default=None,opts=('y','n'),please=None,fmt=None):
    try:
        raw_input = input
    except NameError:
        pass
    quest += " ("
    quest += "/".join(['['+o+']' if o==default else o for o in opts])
    quest += "): "

    if default is not None: opts = list(opts)+['']
    if please is None: please = quest
    if fmt is None: fmt = lambda x: x

    rin = input(quest)
    while fmt(rin) not in opts: rin = input(please)

    return default if default is not None and rin == '' else fmt(rin)

def get_user_vars():
    """
    Get variables in user namespace (ripped directly from ipython namespace
    magic code)
    """
    import IPython
    ip = IPython.get_ipython()    
    user_ns = ip.user_ns
    user_ns_hidden = ip.user_ns_hidden
    nonmatching = object()
    var_hist = [ i for i in user_ns
                 if not i.startswith('_') \
                 and (user_ns[i] is not user_ns_hidden.get(i, nonmatching)) ]
    return var_hist

def shutdown_logger():
    """
    Prompts for saving the current session during shutdown
    """
    import IPython, pickle
    var_hist = get_user_vars()
    ip = IPython.get_ipython()
    db = ip.db

    # collect any variables that need to be deleted from db
    keys = map(lambda x: x.split('/')[1], db.keys('autorestore/*'))
    todel = set(keys).difference(ip.user_ns)
    changed = [db[k] != ip.user_ns[k.split('/')[1]]
               for k in db.keys('autorestore/*') if k.split('/')[1] in ip.user_ns]

    try:
        if len(var_hist) == 0 and len(todel) == 0 and not any(changed): return
        if get_response("Save session?", 'n', fmt=str.lower) == 'n': return
    except KeyboardInterrupt:
        return

    # Save interactive variables (ignore unsaveable ones)
    for name in var_hist:
        obj = ip.user_ns[name]
        try:
            db[ 'autorestore/' + name ] = obj
        except pickle.PicklingError:
            print("Could not store variable '%s'. Skipping..." % name)
            del db[ 'autorestore/' + name ]

    # Remove any previously stored variables that were deleted in this session
    for k in todel:
        del db['autorestore/'+k]

import atexit
atexit.register(shutdown_logger)
del atexit

I think you need to use dill for this purpose. "dill extends python's pickle module for serializing and de-serializing python objects to the majority  A workspace in Matlab is a fundamentally different concept than the interpreter state of a Python process. Python is a general purpose programming language and you might have any number of artifacts (variables, modules) in your Python session which are depending on external factors (sockets, open file descriptors, database connections, and so on). You can use Python through iPython and the Jupyter “notebook” interface.

You can try

%save name lines

Like if you have input 67 commands and you want to save all of them:

%save myhistory 1-67

Stores variables, aliases and macros in IPython's database. To automatically restore stored variables at startup, add this to your ipython_config.py file: c. Recommend: python - Save a function definition in ipython. ion, e.g.: In [1]: def func1 ():: pass: In [2]: %save func1.py func1func1 is neither a string nor a macro. Instead I have to pick out the function definition line number from enumerate (_ih), or manually copy and paste from.

You can use the dill python package:

import dill                            
filepath = 'session.pkl'
dill.dump_session(filepath) # Save the session
dill.load_session(filepath) # Load the session

To install it:

pip install dill

You can use Python through iPython and the Jupyter “notebook” interface. In general that environment allows you to save your work/session in a way that will​  save-ipython-variables lets you save your global IPython variables to disk easily, and load them back into the global namespace when you need them again, even in a whole new IPython session. I haven't had much chance to use it yet, but looks promising.

Although not so convenient as save.image(), you can use one of the checkpoint/restore applications. If you're using Linux, you might try http://criu.org. I'm using it from time to time to dump my ipython state and restore it later.

In order to dump a shell app with CRIU, you need to find its PID (e.g. pstree -p) and then use something like this (you'll need a second terminal for this; CRIU can't dump stopped jobs):

sudo criu dump -t PID --images-dir ~/tmp/imgs --log-file dump.log -v4 --shell-job

this will write all necessary images to ~/tmp/imgs (remember the --shell-job option). In order to restore the state later to the current terminal (don't forget to hit enter to get the next ipython prompt):

sudo criu restore --images-dir ~/tmp/imgs/ --log-file restore.log -v4 --shell-job

Check out the logs in case of any problems.

Obviously CRIU will work with any app (with some limits, of course). It's just an idea so you can use it for ipython.

r/IPython: If you have a question about IPython, (now Jupyter) the programming language written by scientists for scientists with an eye towards … The definitive guide to setup my Python workspace. I want one iPython Console for Python3 and one iPython Console for Python2, so no need to install iPython on my projects’ virtualenvs.

you can certainly do this in the ipython notebook.

when the notebook is saved--either manually or by default config--the notebook is persisted as an .ipynb file, which is just a json file (an example in a github gist).

Next time you start the ipython server in the directory where that file resides, the server will detect it.

when you open that notebook in the browser, all of the code & config is there but unexecuted; you can execute the code in every cell by selecting execute all cells from the cells menu.

in addition, you can manually persist snapshots of your notebook, as ipynb_checkpoints, which are stored in a directory of that name preceded by a dot.

and finally, from the file menu option, you can persist your notebook as a pure python source file (.py)

I had been holding out against using Jupyter or its predecessor Ipython becuase I was under the impression that it was only for Python users, but  I don't really know, but IPython launches python kernels, which are killed when you shut it down and therefore, all vcariables vanish. I don't know if IPython has a mechanism to store variables (kinda doubt it), but you could pickle the variables you need manually (like in the last cell of the notebook) and load them (also with pickle) in tbe first cell.

Is there a way to save “session” in Jupyter? I'm using my personal desktop, and I'​m shutting it down at night. So I would like to resume next day  Dill has functions to save and restore all the variables in a session. It's pretty easy to use. To Save a session: import dill dill.dump_session('notebook_env.db') To restore a session. import dill dill.load_session('notebook_env.db') Dill is based on Pickel, but you can't use Pickel, because it doesn't serialize modules.

Jupyter Notebook is an open-source application that allows you to embed live code and explanatory text, create plots, and save results and images that  The cache magic cannot save the state of a whole notebook yet. To my knowledge there is no other system yet to resume a "notebook". This would require to save all the history of the python kernel. After loading the notebook, and connecting to a kernel, this information should be loaded.

Data generated by running an analysis in a Jupyter notebook is saved to the lets the notebook grab variables such as the workspace name and Google bucket  storemagic¶ %store magic for lightweight persistence. Stores variables, aliases and macros in IPython’s database. To automatically restore stored variables at startup, add this to your ipython_config.py file:

Comments
  • No, but you can save sets of variables. Perhaps better would be to do your ipythoning in notebook form, where results, code, visualisations and comments all live side-by-side.
  • look at the %store magic
  • %store magic seems to be an acceptable solution. At least it works for numpy array and pandas DataFrame, which are most of the data structure I use. Is there a way to automate this in ipython notebook, similar to what Rstudio does?
  • The dill package has a dump_session() function which will save the contents of an interactive session to a file, including some things that pickle can't handle normally. I'm fairly sure this works with IPython. You could write an extension to automate that when you shut down a kernel.
  • Has anybody else tried this in Ipython 6 and Windows?
  • it didnt work for me.. perhaps something has changed since ipython 2.0
  • does this actually save the state (i.e., the variables / objects)? My understanding is that it does not. If it does, can you provide an example? Thanks!
  • %save will save the history command into a .py file. Then, I can run this .py file anytime to recover the workspace. ` In [1]: s = 'how are you?' In [2]: a = 123 In [3]: %save x 1-2 The following commands were written to file x.py: s = 'how are you?' a = 123`
  • @maoyang the bad point is that it may take much time to rerun the commands
  • I think OP is asking about saving the variables and workspace, not the history. The history is automatically saved in ipython, and can be retreived easily. Not so with the workspace.
  • That would be super interesting to implement in tmux-resurrect !
  • Thanks, but this is not exactly what I want. What I need is to preserve the state of a session (all the variable values). That would be something equivalent to R's save.image() function.
  • @qkhhly that's in my answer: penultimate sentence, 'ipynb_checkpoints'; load a file in this directory, then execute the code in the cells manually or via selecting 'execute all' from 'Cells' in menubar).
  • @doug, executing the code (again) might be quite impractical, e.g. it could take hours, so it is not always a feasible solution to the question.
  • Downvoted because this is not what the OP asked -- not the ability to re-run everything, but get their state back "This would be a similar function to save.image() in MATLAB or R."