Remove duplicates key from list of dictionaries python

Related searches

I am trying to remove the duplicates from following list

 distinct_cur = [{'rtc': 0, 'vf': 0, 'mtc': 0, 'doc': 'good job', 'foc': 195, 'st': 0.0, 'htc': 2, '_id': ObjectId('58e86a550a0aeff4e14ca6bb'), 'ftc': 0}, 
 {'rtc': 0, 'vf': 0, 'mtc': 0, 'doc': 'good job', 'foc': 454, 'st': 0.8, 'htc': 1, '_id': ObjectId('58e8d03958ae6d179c2b4413'), 'ftc': 1},
 {'rtc': 0, 'vf': 2, 'mtc': 1, 'doc': 'test', 'foc': 45, 'st': 0.8, 'htc': 12, '_id': ObjectId('58e8d03958ae6d180c2b4446'), 'ftc': 0}] 

of dictionaries based on condition that if 'doc' key value text is same then one of the dictionary should be removed. I have tried the following solution

distinct_cur = [dict(y) for y in set(tuple(x.items()) for x in cur)] 

but duplicates are still present in the final list.

Below is the desired output as in 1st and 2nd distinct_cur text of key 'doc' value is same (good job):

[{'rtc': 0, 'vf': 0, 'mtc': 0, 'doc': 'good job', 'foc': 195, 'st': 0.0, 'htc': 2, '_id': ObjectId('58e86a550a0aeff4e14ca6bb'), 'ftc': 0}, 
 {'rtc': 0, 'vf': 2, 'mtc': 1, 'doc': 'test', 'foc': 45, 'st': 0.8, 'htc': 12, '_id': ObjectId('58e8d03958ae6d180c2b4446'), 'ftc': 0}] 

Thanks in advance!

You're creating a set out of different elements and expect that it will remove the duplicates based on a criterion that only you know.

You have to iterate through your list, and add to the result list only if doc has a different value than the previous ones: for instance like this:

done = set()
result = []
for d in distinct_cur:
    if d['doc'] not in done:
        done.add(d['doc'])  # note it down for further iterations
        result.append(d)

that will keep only the first occurrence(s) of the dictionaries which have the same doc key by registering the known keys in an aux set.

Another possibility is to use a dictionary with the key as the "doc" key of the dictionary, iterating backwards in the list so the first items overwrite the last ones in the list:

result = {i['doc']:i for i in reversed(distinct_cur)}.values()

Remove duplicates key from list of dictionaries python, Python | Removing duplicate dicts in list. Last Updated: 02-12- frozenset is used to assign a value to key in dictionary as a set. The repeated� Sometimes, while working with Python dictionaries, we can have problem in which we need to perform the removal of all the duplicate values of dictionary, and we are not concerned if any key get removed in the process.

I see 2 similar solutions that depend on your domain problem: do you want to keep the first instance of a key or the last instance?

Using the last (so as to overwrite the previous matches) is simpler:

d = {r['doc']: r for r in distinct_cur}.values()

Python, Create a function to remove duplicate dictionaries from a list. A dictionary is considered a duplicate if it has the exact same keys and values. For this list: return this: [{'a': 123}, {'b': 123}] I have a list of dicts, and I'd like to remove the dicts with identical key and value pairs. 30829/how-can-i-remove-duplicate-dict-in-list-in-python

Try this:

distinct_cur  =[dict(t) for t in set([tuple(d.items()) for d in distinct_cur])]

Worked for me...

How to Remove Duplicate Dictionaries in a List, I have a list of dicts, and I'd like to remove the dicts with identical key and value pairs. For this list: return this: [{'a': 123}, {'b': 123}] Create a dictionary, using the List items as keys. This will automatically remove any duplicates because dictionaries cannot have duplicate keys.

One liner to deduplicate the list of dictionaries distinct_cur on the primary_key of doc

[i for n, i in enumerate(distinct_cur) if i.get('doc') not in [y.get('doc') for y in distinct_cur[n + 1:]]]

How can I remove duplicate dict in list in Python?, Write a Python program to remove duplicates from Dictionary. in student_data. items(): if value not in result.values(): result[key] = value� The common utility to remove the dictionary corresponding to particular key in a list of dictionaries is also a problem whose concise version is always helpful. This has application in web development due to the introduction of No-SQL databases, which work mostly on Key-Value pairs.

Python: Remove duplicates from Dictionary, The basic method that comes to mind while performing this operation is the naive method of iterating the list of dictionaries and manually removing the duplicate dictionary and append in new list. filter_none

One simple approach would be to create a reverse dictionary using the concatenation of the string data in each inner dictionary as a key. So say you have the above data in a dictionary, d: >>> import collections >>> reverse_d = collections.defaultdict(list) >>> for key, inner_d in d.iteritems():

Recommended Posts: Python - Convert Flat dictionaries to Nested dictionary; Python - Remove Duplicate Dictionaries characterized by Key; Python - Convert Dictionaries List to Order Key Nested dictionaries

Comments
  • It would be better if you could provide a sample output.
  • You're not allowed to use duplicate keys in dictionary. What you mean by removing duplicate key? what should be removed?
  • So after you find dictionaries with same doc key, how you decide which one should be removed?
  • Here are some similar questions stackoverflow.com/questions/15511903/… and stackoverflow.com/questions/9427163/…