Filter list of dictionaries with higher key value remove duplicate dictionaries

python remove duplicate values from dictionary
python list of dictionaries find duplicates based on value
python remove duplicates from list by key
python find duplicates in list of dictionaries
duplicate values in dictionary python
python duplicate keys in dictionary
find duplicate keys in dictionary python
python remove duplicates from list of dictionaries by key

I have list of dictionaries like:

sel_list = [{'a': 8}, {'a': 4}, {'a': 4}, {'b': 8}, {'b': 9}]

I want to remove duplicate dictionaries and if more than one dictionaries have same key but different values then choose that dictionary with higher value.

Like :

sel_list = [{'a': 8}, {'b': 9}]

I have tried:

[i for n, i in enumerate(sel_list) if i not in sel_list[n + 1:]]

its results in:

[{'a': 8}, {'a': 4}, {'b': 8}, {'b': 9}]

What I can do to achieve my results?

We can do this by constructing a dictionary that "folds" the values by picking the maximum each time. Like:

dummy = object()
maximums = {}
for subd in sel_list:
    for k, v in subd.items():
        cur = maximums.get(k, dummy)
        if cur is dummy or v > cur:
            maximums[k] = v
result = [{k: v} for k, v in maximums.items()]

We thus iterate over the key-value pairs of the dictionaries in the list, and each time update the maximums dictionary in case the key does not yet exists, or the current value is less.

After this iteration step, we produce a list of dictionaries with the maximum key-value pairs.

This approach works on all types that can be ordered (numbers, strings, etc.), and the keys should be hashable, but this assumption holds since in the list of dictionaries, the keys are already hashed.

Furthermore it works rather robust in the sense that it will ignore empty dictionaries, and will process a dictionary with multiple key-value pairs as well, by seeing these as independent key-value pairs.

You can also decide to work with maximums directly: a dictionary that contains all the keys in your original list, and associates these with the maximum value seen in the list.

How to Remove Duplicate Dictionaries in a List - Better , I want to remove duplicate dictionaries and if more than one dictionaries have same key but different values then choose that dictionary with  The filtered dictionary i.e. newDict now contains filtered elements from the original dictionary i.e. elements whose key is divisible by 2. Similarly we can have conditional filtering based on value field instead of key. But instead of writing code for iteration and condition checking again and again, we move the code to a generic function and

Disclaimer : I'm not sure How much Pythonic is my solutionAssuming the dict contain only one key,value pair. (ie, {"a":2,"b":3} is not a valid dict here)

sel_list = [{'a': 0}, {'a': 4}, {'a': 4}, {'b': 8}, {'b': 9}, {'d': 9}]
result_dict = {}
for item in sel_list:
    key = list(item.keys())[0]
    if key in result_dict:
        if item[key] > result_dict[key][key]:
            result_dict.update({key: item})
    else:
        result_dict.update({key: item})
result_list = [v for k, v in result_dict.items()]
print(result_list)

Python, Create a function to remove duplicate dictionaries from a list. A dictionary is considered a duplicate if it has the exact same keys and values. The function should  The common utility to remove the dictionary corresponding to particular key in a list of dictionaries is also a problem whose concise version is always helpful. This has application in web development due to the introduction of No-SQL databases, which work mostly on Key-Value pairs. Let’s discuss certain ways in which this task can be performed.

Code:

from collections import defaultdict

sel_list = [{'a': 8}, {'a': 4}, {'a': 4}, {'b': 8}, {'b': 9}]

results = defaultdict(list)

for element in sel_list:
    for key, value in element.items():
        results[key].append(value)

for item_key, item_value in results.items():
    results[item_key] = max(item_value)

print(results)

Output:

defaultdict(<class 'list'>, {'b': 9, 'a': 8})

How can I remove duplicate dict in list in Python?, List of dictionaries are quite common and sometimes we require to duplicate the duplicated. frozenset is used to assign a value to key in dictionary as a set. Given a dictionary, the task is to find keys with duplicate values. Let’s discuss a few methods for the same. Suppose you need to find keys having duplicate values. Check out this Author's contributed articles. If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your

You can do it via a reduce operation where a dict is your state:

from functools import reduce
from itertools import chain

sel_list = [{'a': 8}, {'a': 4}, {'a': 4}, {'b': 8}, {'b': 9}]

# function to aggregate max value item into a `dict`
def agg(d, kv):
    k, v = kv
    d[k] = max(d.get(k, v), v)
    return d

# concatenate all items from all `dict`s
sel_items = chain.from_iterable(map(dict.items, sel_list))

# aggregate using a single `dict` which implicitly holds required semantics
result = reduce(agg, sel_items, {}) # {'a': 8, 'b': 9}  <-- better?

To get your output format (which is not favorable over dict IMO):

formatted = [dict((item, )) for item in result.items()]

dict has semantics that work in your favor. I think it is a better data structure for you.

Python Remove Duplicates from List, I have a list of dicts, and I'd like to remove the dicts with identical key and value pairs. For this print {frozenset(item.items()):item for item in data}.values() # [{'a': 3222, How can I convert a list of dictionaries from a CSV into a JSON object in Python? len() >>> mylist=[] >>> print len(mylist) 0 READ MORE. We thus iterate over the key-value pairs of the dictionaries in the list, and each time update the maximums dictionary in case the key does not yet exists, or the current value is less. After this iteration step, we produce a list of dictionaries with the maximum key-value pairs.

Python: Find duplicates in a list with frequency count & index , Dictionaries in Python cannot include duplicate keys, and so if we convert our list to a dictionary, it will automatically remove any duplicate values. Takes dictionaries as input and returns a list with each item in the list being a dictionary with ‘key’ and ‘value’ as keys to the previous dictionary’s structure. Common return values are documented here, the following are the fields unique to this lookup: If you notice any issues in this documentation, you can edit this document to

Python : How to get all keys with maximum value in a Dictionary , Step 1: Get duplicate elements in a list with a frequency count Filter key-value pairs in dictionary. pairs whose value is greater than 1 i.e. only duplicate elements from list. If the element exists in dictionary keys, then increments the value of Remove elements from dictionary whose value is 1, i.e. non  Python | Filter dictionary key based on the values in selective list In Python, sometimes we require to get only some of the dictionary keys and not all. This problem is quite common in web development that we require to get only the selective dictionary keys from some given list.

Python Dictionaries, It return the item (key/value tuple) with maximum value in dictionary. Subscribe with us to join 1500+ Python & C++ developers, to get more Tips & Tutorials like this. Python : How to Remove multiple keys from Dictionary while Iterating ? Python: How to create an empty list and append items to it? Till now, we have seen the ways to creating dictionary in multiple ways and different operations on the key and values in dictionary. Now, let’s see different ways of creating a dictionary of list. Note that the restriction with keys in Python dictionary is only immutable data types can be used as keys, which means we cannot use a dictionary

Comments
  • So every dictionary contains exactly one key-value pair?
  • Is the order of the result important?
  • yes, every dict contains one key value pair and order is not important.
  • @UsmanMaqbool If order is not important and you want one value per key, why are you not using a dict as your data structure? Why the list? See my answer for more details.
  • This looks very interesting. I think I'm going to learn about some hidden gems in python. Would you please explain why do you use object() and not for example None? Wouldn't it be simpler to just do: cur = maximums.get(k, None) and then compare: if cur is None or v > cur:?
  • @cezar: because technically speaking, one can make objects with an order relation with None. If for example None occurs once, we might want to include it into the result list as well. By adding a dummy, we make a distinction between a None in the dictionary, and not found.
  • Thank you for the explanation. I didn't think about the possibility of having None as value in the dict. I'll remember this, it may evenutally save my ass at some point.
  • @WillemVanOnsem I recommend maximums.get(k, v) and lose the dummy. I still think OP has the wrong data layout (see my answer).