Python Remove Duplicates from a List

The job is simple. We need to take a list, with duplicate elements in it and generate another list that only contains the element without the duplicates in them.

Examples: 

Input : [2, 4, 10, 20, 5, 2, 20, 4] Output : [2, 4, 10, 20, 5] Input : [28, 42, 28, 16, 90, 42, 42, 28] Output : [28, 42, 16, 90]

We can use not in on list to find out the duplicate items. We create a result list and insert only those that are not already not in.

Python3

# Python code to remove duplicate elements def Remove(duplicate): final_list = [] for num in duplicate: if num not in final_list: final_list.append(num) return final_list # Driver Code duplicate = [ 2 , 4 , 10 , 20 , 5 , 2 , 20 , 4 ] print (Remove(duplicate))

Output: 

[2, 4, 10, 20, 5]

Easy Implementation:

A quick way to do the above using set data structure from the python standard library (Python 3.x implementation is given below)

Python3

duplicate = [ 2 , 4 , 10 , 20 , 5 , 2 , 20 , 4 ] print ( list ( set (duplicate)))

Output: 

[2, 4, 10, 20, 5]

Method 2: Using Dictionary/hashmap

Approach:

  1. Create a dictionary and by default keep the count of every element to zero, using the default dict.
  2. If the count of elements is ZERO, increment the value in the dictionary and continue.
  3. If the count of element is greater than zero, Then remove the element from the given list using the remove() method.

You can read more about default dict here.

Python3

from collections import defaultdict def default_val(): # dict : maintain count of each element. with default value of key is 0 mydict = defaultdict(default_val) l = [ 1 , 2 , 3 , 2 , 6 , 3 , 5 , 3 , 7 , 8 ] for i in l: # if the element already present in the array, remove the element. if mydict[i] = = 1 : # If the elements appears first time keep it count as 1 mydict[i] = 1 # printing the final array
Output 
[1, 2, 6, 5, 3, 7, 8]

Method: Using dict.fromkeys()

Python3

input_list = [ 1 , 2 , 3 , 2 , 6 , 3 , 5 , 3 , 7 , 8 ] mylist = list ( dict .fromkeys(input_list)) print (mylist)
Output 
[1, 2, 3, 6, 5, 7, 8]

Time complexity: Average-case time complexity: O(n); n is the size of the array

Worst-case time complexity: O( ); n is the size of the array

Auxiliary Space: O(n); n is the size of the array

Method: Using list comprehension:

Python

list1 = [ 1 , 2 , 3 , 2 , 6 , 3 , 5 , 3 , 7 , 8 ] mylist = [ list1[i] for i in range ( len (list1)) if list1.index(list1[i]) = = i] print (mylist)

Output:

[1, 2, 3, 6, 5, 7, 8]

Method: Using Counter() function

Python3

# Python code to remove duplicate elements from collections import Counter # Driver Code duplicate = [ 2 , 4 , 10 , 20 , 5 , 2 , 20 , 4 ] unique = Counter(duplicate) print ( list (unique.keys()))
Output 
[2, 4, 10, 20, 5]

Time Complexity: O(N)

Auxiliary Space : O(N)

Method: Using operator.countOf() method

Python3

import operator as op # Python code to remove duplicate elements def Remove(duplicate): final_list = [] for num in duplicate: if op.countOf(final_list, num) = = 0 : final_list.append(num) return final_list # Driver Code duplicate = [ 2 , 4 , 10 , 20 , 5 , 2 , 20 , 4 ] print (Remove(duplicate))
Output 
[2, 4, 10, 20, 5]

Time Complexity: O(N)

Auxiliary Space : O(N)

Like Article -->

Please Login to comment.

Similar Reads

Python | Remove consecutive duplicates from list

In Python, we generally wish to remove the duplicate elements, but sometimes for several specific usecases, we require to have remove just the elements repeated in succession. This is a quite easy task and having a shorthand for it can be useful. Let's discuss certain ways in which this task can be performed. Method #1 : Using groupby() + list comp

4 min read Python - Ways to remove duplicates from list

This article focuses on one of the operations of getting a unique list from a list that contains a possible duplicate. Removing duplicates from list operation has a large number of applications and hence, its knowledge is good to have in Python. Ways to Remove duplicates from the list:Below are the methods that we will cover in this article: Using

10 min read Python groupby method to remove all consecutive duplicates

Given a string S, remove all the consecutive duplicates. Examples: Input : aaaaabbbbbb Output : ab Input : geeksforgeeks Output : geksforgeks Input : aabccba Output : abcba We have existing solution for this problem please refer Remove all consecutive duplicates from the string link. We can solve this problem in python quickly using itertools.group

2 min read Python | Remove all duplicates words from a given sentence

Given a sentence containing n words/strings. Remove all duplicates words/strings which are similar to each others. Examples: Input : Geeks for Geeks Output : Geeks for Input : Python is great and Java is also great Output : is also Java Python and great We can solve this problem quickly using python Counter() method. Approach is very simple. 1) Spl

7 min read Remove All Duplicates from a Given String in Python

We are given a string and we need to remove all duplicates from it. What will be the output if the order of character matters? Examples: Input : geeksforgeeks Output : geksfor This problem has an existing solution please refer to Remove all duplicates from a given string. Method 1: [GFGTABS] Python from collections import OrderedDict # Function to

3 min read Remove duplicates from a dataframe in PySpark

In this article, we are going to drop the duplicate data from dataframe using pyspark in Python Before starting we are going to create Dataframe for demonstration: C/C++ Code # importing module import pyspark # importing sparksession from pyspark.sql module from pyspark.sql import SparkSession # creating sparksession and giving an app name spark =

2 min read Python | Program to print duplicates from a list of integers

Given a list of integers with duplicate elements in it. The task is to generate another list, which contains only the duplicate elements. In simple words, the new list should contain elements that appear as more than one. Examples: Input : list = [10, 20, 30, 20, 20, 30, 40, 50, -20, 60, 60, -20, -20] Output : output_list = [20, 30, -20, 60]Input :

5 min read Python | Consecutive duplicates all elements deletion in list

Sometimes, while working with Python list, a problem can occur to filter list to remove duplicates. The solution to this has been discussed before. But sometimes, we may have a problem in which we need to delete the duplicate and element itself if it occurs more than 1 in consecution. This type of problem can occur in day-day programming and other

4 min read Python - Ways to print longest consecutive list without considering duplicates element

Given, a list of Numbers, the task is to print the longest consecutive (Strictly) list, without considering duplicate elements. If there is more than one answer, print anyone. These type of problem are quite common while working on some web development projects in Django or flask. Below are some ways to solve the above task. Note : If there are mul

4 min read What is the fastest way to drop consecutive duplicates a List[int] column?

Handling data efficiently is a critical task in many programming and data analysis scenarios. One common problem is dealing with consecutive duplicates in lists, especially when working with columns of data in formats like CSV files or databases. This article explores the fastest ways to drop consecutive duplicates from a List[int] column, ensuring

3 min read Python | Remove all values from a list present in other list

Sometimes we need to perform the operation of removing all the items from the lists that are present in another list, i.e we are given some of the invalid numbers in one list which need to be get ridden from the original list. Let's discuss various ways How to remove the elements of a list from another list in Python. Illustration: Input: List one

10 min read PyQt5 - How make duplicates insertions in ComboBox

In this article we will see how we can allow duplicates insertions in the combo box. By default no duplicate insertion is allowed although programmatically inserting duplicate items into the combo box is possible but user can't insert duplicate values. In order to allow duplicates insertion we will use setDuplicatesEnabled method Syntax : combo_box

2 min read How to count duplicates in Pandas Dataframe?

Let us see how to count duplicates in a Pandas DataFrame. Our task is to count the number of duplicate entries in a single column and multiple columns. Under a single column : We will be using the pivot_table() function to count the duplicates in a single column. The column in which the duplicates are to be found will be passed as the value of the

2 min read Concatenate Pandas DataFrames Without Duplicates

In this article, we are going to concatenate two dataframes using pandas module. In order to perform concatenation of two dataframes, we are going to use the pandas.concat().drop_duplicates() method in pandas module. Step-by-step Approach: Import module.Load two sample dataframes as variables.Concatenate the dataframes using pandas.concat().drop_du

2 min read Minimum distance between duplicates in a String

Given a string S and its length N (provided N > 0). The task is to find the minimum distance between same repeating characters, if no repeating characters present in string S return -1. Examples: Input: S = "geeksforgeeks", N = 13Output: 0 Explanation: The repeating characters in string S = "geeksforgeeks" with minimum distance is 'e'.The minimu

15+ min read How to drop duplicates and keep one in PySpark dataframe

In this article, we will discuss how to handle duplicate values in a pyspark dataframe. A dataset may contain repeated rows or repeated data points that are not useful for our task. These repeated values in our dataframe are called duplicate values. To handle duplicate values, we may use a strategy in which we keep the first occurrence of the value

3 min read Delete duplicates in a Pandas Dataframe based on two columns

A dataframe is a two-dimensional, size-mutable tabular data structure with labeled axes (rows and columns). It can contain duplicate entries and to delete them there are several ways. The dataframe contains duplicate values in column order_id and customer_id. Below are the methods to remove duplicate values from a dataframe based on two columns. Me

2 min read Python | Remove empty tuples from a list

In this article, we will see how can we remove an empty tuple from a given list of tuples. We will find various ways, in which we can perform this task of removing tuples using various methods and ways in Python. Examples: Input : tuples = [(), ('ram','15','8'), (), ('laxman', 'sita'), ('krishna', 'akbar', '45'), ('',''),()]Output : [('ram', '15',

8 min read Python | Remove and print every third from list until it becomes empty

Given a list of numbers, Your task is to remove and print every third number from a list of numbers until the list becomes empty. Examples: Input : [10, 20, 30, 40, 50, 60, 70, 80, 90] Output : 30 60 90 40 80 50 20 70 10 Explanation: The first third element encountered is 30, after 30 we start the count from 40 for the next third element which is 6

4 min read Ways to remove particular List element in Python

List is an important container and is used almost in every code of day-day programming as well as web development. The more it is used, more is the required to master it and hence knowledge of its operations is necessary. Let's see the different ways of removing particular list elements. Method #1 : Using remove() remove() can perform the task of r

6 min read Python | Remove empty strings from list of strings

In many scenarios, we encounter the issue of getting an empty string in a huge amount of data and handling that sometimes becomes a tedious task. Let's discuss certain way-outs to remove empty strings from list of strings. Method #1: Using remove() This particular method is quite naive and not recommended use, but is indeed a method to perform this

7 min read Python | Remove square brackets from list

Sometimes, while working with displaying the contents of list, the square brackets, both opening and closing are undesired. For this when we need to print the whole list without accessing the elements for loops, we require a method to perform this. Let's discuss a shorthand by which this task can be performed. Method 1: Using str() + list slicing T

5 min read Python - Remove Key from Dictionary List

Sometimes, while working with Python dictionaries, we can have a problem in which we need to remove a specific key from a dictionary list. This kind of problem is very common and has application in almost all domains including day-day programming and web development domain. Let's discuss certain ways in which this task can be performed. Method #1 :

7 min read Python - Remove empty value types in dictionaries list

Sometimes, while working with Python dictionaries, we require to remove all the values that are virtually Null, i.e does not hold any meaningful value and are to be removed before processing data, this can be an empty string, empty list, dictionary, or even 0. This has applications in data preprocessing. Let us discuss certain ways in which this ta

7 min read Python | Remove Redundant Substrings from Strings List

Given list of Strings, task is to remove all the strings, which are substrings of other Strings. Input : test_list = ["Gfg", "Gfg is best", "Geeks", "for", "Gfg is for Geeks"] Output : ['Gfg is best', 'Gfg is for Geeks'] Explanation : "Gfg", "for" and "Geeks" are present as substrings in other strings.Input : test_list = ["Gfg", "Geeks", "for", "Gf

5 min read Remove falsy values from a list in Python

Prerequisite: Truthy vs Falsy Values in Python In Python, the value that evaluates to False is considered as a Falsy value. The value such as empty list, empty dictionary, empty tuple, empty set, empty string, None, False, 0 are considered as Falsy values. So our task is to remove all the Falsy values from the List. Examples: Input: [10,20,30,0,Fal

3 min read Remove all Alphanumeric Elements from the List - Python

Alphanumeric elements consist of only alphabetical and numerical characters. Any of the special characters are not included in alphanumeric elements. This article is going to show you multiple ways to remove all the alphanumeric elements from a List of strings using Python. Example 1: Input: ['a','b', '!', '1', '2', '@', 'abc@xyz.com'] Output: ['!'

2 min read Remove Last Element from List in Python

Given a list, the task is to write a Python program to remove the last element present in the list and update the original list in Python. Example: Input: ["geeks", "for", "geeks"]Output:["geeks", "for"] Input: [1, 2, 3, 4, 5]Output:[1, 2, 3, 4] Explanation: Here simply we have to remove the last element present in the list and print the resultant

4 min read Remove all the occurrences of an element from a list in Python

The task is to perform the operation of removing all the occurrences of a given item/element present in a list. Example Input1: 1 1 2 3 4 5 1 2 1 Output1: 2 3 4 5 2 Explanation : The input list is [1, 1, 2, 3, 4, 5, 1, 2] and the item to be removed is 1. After removing the item, the output list is [2, 3, 4, 5, 2]Remove all Occurrences of an Item fr

4 min read Python List remove() Method

Python list remove() method removes a given element from the list. Example: C/C++ Code lis.remove("b") print(lis) Output['a', 'c'] List remove() Syntaxlist_name.remove(obj) Parameterobj: object to be removed from the list ReturnsThe method does not return any value but removes the given object from the list. ExceptionIf the element doesn'