How to check if an object is pickleable

python make object picklable
pickle dump object
dill python
python serialize object
clients have non-trivial state that is local and unpickleable.
pickling and unpickling in python
unpicklable objects
python pickle class

I have a list of objects of various types that I want to pickle. I would like to pickle only those which are pickleable. Is there a standard way to check if an object is of pickleable type, other than trying to pickle it?

The documentation says that if a pickling exception occurs it may be already after some of the bytes have been written to the file, so trying to pickle the objects as a test doesn't seem like a good solution.

I saw this post but it doesn't answer my question.

I would propose duck testing in this case. Try to pickle into a temporary file or a memory file, as you find suitable, then if it fails discard the result, if it succeeds rename.

Why?

In python you can check if the object has some properties in two ways.

Check if object is an instance of some Abstract Base Class. E.g. Number "The root of the numeric hierarchy. If you just want to check if an argument x is a number, without caring what kind, use isinstance(x, Number)."

Or try it and then handle exceptions. This occurs during many occasions. The pythonic philosopy is based around the duck. Duck typing, duck test, and EAFP are the keywords.

I even believe the 1st one has been properly introduced with python3 under the pressure from the part of the community, while many still strongly believe duck is the way to go with python.

AFAIK there is no special preconditions that can be checked, nor any ABC that object can be checked against in case of pickling. So all that is left is duck.

Maybe something else could be attempted but probably it is not worth of it. It would be very hard to do manual introspection of the object to find out preliminarily if it's suitable for pickling.

3.14.4 What can be pickled and unpickled?, tuples, lists, sets, and dictionaries containing only picklable objects pickle unpicklable objects will raise the PicklingError exception; when this  In python you can check if the object has some properties in two ways. Check if object is an instance of some Abstract Base Class. E.g. Number "The root of the numeric hierarchy. If you just want to check if an argument x is a number, without caring what kind, use isinstance(x, Number)." Or try it and then handle exceptions.

There's the dill.pickles method in dill package that does just that.

>>> class Foo(object):
...   x = iter([1,2,3])
... 
>>> f = Foo()     
>>> 
>>> dill.pickles(f)
False

We can use methods in dill to look for what causes the failure.

>>> dill.detect.badtypes(f)
<class '__main__.Foo'>
>>> dill.detect.badtypes(f, depth=1)
{'__setattr__': <type 'method-wrapper'>, '__reduce_ex__': <type 'builtin_function_or_method'>, '__reduce__': <type 'builtin_function_or_method'>, '__str__': <type 'method-wrapper'>, '__format__': <type 'builtin_function_or_method'>, '__getattribute__': <type 'method-wrapper'>, '__class__': <type 'type'>, '__delattr__': <type 'method-wrapper'>, '__subclasshook__': <type 'builtin_function_or_method'>, '__repr__': <type 'method-wrapper'>, '__hash__': <type 'method-wrapper'>, 'x': <type 'listiterator'>, '__sizeof__': <type 'builtin_function_or_method'>, '__init__': <type 'method-wrapper'>}
>>> dill.detect.badtypes(f, depth=1).keys()
['__setattr__', '__reduce_ex__', '__reduce__', '__str__', '__format__', '__getattribute__', '__class__', '__delattr__', '__subclasshook__', '__repr__', '__hash__', 'x', '__sizeof__', '__init__']

So, the only thing that's failing that's not a "built-in" method of the class is x… so that's a good place to start. Let's check 'x', then replace it with something else if it's the problem.

>>> dill.pickles(Foo.x)
False
>>> Foo.x = xrange(1,4)
>>> dill.pickles(Foo.x)
True

Yep, x was causing a failure, and replacing it with an xrange works because dill can pickle an xrange. What's left to do?

>>> dill.detect.badtypes(f, depth=1).keys()
[]
>>> dill.detect.badtypes(f, depth=1)       
{}
>>> dill.pickles(f)                 
True
>>> 

Apparently (likely because references to x in the class __dict__ now pickle), f now pickles… so we are done.

dill also provides trace to show the exact path in pickling the object.

>>> dill.detect.trace(True)
>>> dill.pickles(f)
T2: <class '__main__.Foo'>
F2: <function _create_type at 0x10e79b668>
T1: <type 'type'>
F2: <function _load_type at 0x10e79b5f0>
T1: <type 'object'>
D2: <dict object at 0x10e7c6168>
Si: xrange(1, 4)
F2: <function _eval_repr at 0x10e79bcf8>
D2: <dict object at 0x10e7c6280>
True

Checking if an object is "atomically pickleable", I have a list of objects of various types that I want to pickle. I would like to pickle only those which are pickleable. Is there a standard way to check if an object is of​  So the check must be a bit more then an equal value. This is specified as a single statement in the custom config file. There is a specific config file for each program, which must be as small as possible to be used. One resource is checked with a single statement in the config file.

dill allows for pickling more things that the builtin pickle.

This should do what you what, I think:

def is_picklable(obj):
  try:
    pickle.dumps(obj)

  except pickle.PicklingError:
    return False
  return True

pickle, Hello folks. What's an accurate way of checking whether an object can be atomically pickled? When I say "atomically pickled", I mean without considering other to avoid making unnecessary copy of big object (a la actual deepcopy), we recursively check whether an object is picklable, and only make unpicklable part. For instance, for a tuple of a picklable list and and an unpickable object, we will make a copy of the tuple - just the container - but not its member list.

The Python pickle Module: How to Persist Objects in Python – Real , Pickling: It is a process where a Python object hierarchy is converted into a If fix_imports is true and protocol is less than 3, the pickle will try to map the and dictionaries containing only picklable objects; functions defined at the top See your article appearing on the GeeksforGeeks main page and help other Geeks. Thanks for contributing an answer to Stack Overflow! Please be sure to answer the question. Provide details and share your research! But avoid … Asking for help, clarification, or responding to other answers. Making statements based on opinion; back them up with references or personal experience. To learn more, see our tips on writing great

Serialization of un-picklable objects, It differs from the json module in that it serializes objects in a binary format, But even if you're working with Python, it's important to know that the pickle module has evolved over time. However, not everything is picklable. This means that only the function name is pickled, along with the name of module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. 3.5

Programming Python: Powerful Object-Oriented Programming, If you are on a UNIX system, it is possible to fallback to the old multiprocessing backend, which can pickle interactively defined functions with the default pickle  class ComplicatedThing(object): def __reduce__(self): return (ComplicatedThing, (self.progress_int,)) There are a few optional extra things you can put into the tuple, mostly useful for when your object graph has cyclic dependencies, but you shouldn't need them here. share. Share a link to this answer. Copy link.

Comments
  • Trying to write it to a file can be a solution. Just don't write it to your real output file but somewhere else. To /dev/null or somewhere.
  • Here are the rules for what can be pickled: docs.python.org/3/library/…
  • Thanks, I am familiar with python duck testing. It just surprises me that there isn't a better way of checking pickleability. Does every pickleable object need to implement certain methods? Can't we just duck test for one of these methods?
  • Well when I first stumbled upon it I was baffled. I needed to check if object is iterable. The simplest method I found is try: mock = iter(data[0]) except TypeError:. And it is quite against some python-ways because ideally I should treat it as iterable and pass further. However, this had serious drawback of errors popping up too low to find them easily. From what I read in the docs, python uses its internal knowledge to pickle objects. Its not like __str__. You can provide some helpers in strange cases, but they are not necessary everywhere. I didn't find any other reliable method.
  • That explains it, thanks. Still seems to me like a weird design choice - but I am probably not seeing all the angles on this.
  • @Bitwise Pickling is a complex and, perhaps more importantly, recursive process that depends heavily on what's in scope where. Remember that Python is a dynamic language, so "what is where" is not an easy question to ask at all, and indeed asking can change the answer. So in fact the only possible way to determine if an object is pickleable... is to try pickling it. Actually, a perfect isPickleable() would probably need to solve the halting problem.