JIYIK CN >

This tutorial explored the concept of removing duplicate columns from a Pandas DataFrame.

Dropping Duplicate Columns in Pandas

In this tutorial, let us understand how and why to remove identical or similar columns in a Pandas DataFrame. Most businesses and organizations need to eliminate these duplicate columns as they may not be important to glean insights from them.

Also, when we need to add some additional data to the database, they clutter the database and create issues in storage space. Lastly, duplicate columns may also affect certain statistical or machine learning models as the data may become skewed and will result in very low model accuracy.

Let's see how this can be done in action.

However, before we start, we will create a dummy DataFrame to work with. Here, we create two DataFrames, namely dat1and dat2, with some entries.

import pandas as pd

dat1 = pd.DataFrame({"dat1": [9, 5]})

The above code creates a DataFrame with some entries namely 9and 5. To see the entries in the data, we use the following code.

print(dat1)

The above code gives the following output.

   dat1
0     9
1     5

As shown in the figure, we have 2 columns and 2 rows, where one column represents the index and the second column represents the values ​​in the DataFrame. Now, let us create another dat2DataFrame named using the following code.

dat2 = pd.DataFrame({"dat2": [9, 5]})

Just as we dat1did with , we can visualize this dat2DataFrame using the following code.

print(dat2)

The code gives the following DataFrame.

   dat2
0     9
1     5

As we dat1did with , we have 2 rows and 2 columns, one representing the index and the second representing the values ​​in the DataFrame.

Now, let’s dat2merge the columns of DataFrame into dat1DataFrame. We can do this using the following code.

val = pd.concat([dat1, dat2], axis=1)

As shown in the figure, we have used the function in Pandas concat. This function merges or concatenates multiple DataFrames into one using a single argument passed as an array and merges all the DataFrames.

We also need to specify the axis to which we are adding the DataFrame to alter the DataFrame based on columns or rows.

As it is evident from the code, we use the parameter 1with value axis. It helps in adding columns to the DataFrame of the array assigned in the first parameter.

The output of the code is as follows.

   dat1  dat2
0     9     9
1     5     5

As shown, the DataFrame dat1has been altered so that an additional column has been added on the first axis.

Again, this output print(val)is visualized using the code. We have a DataFrame that contains two columns named dat1and dat2with the same values.

In particular, we added a new row to the DataFrame using jointhe add_rows function in Pandas .dat1

drop_duplicates()Remove duplicate columns in Pandas using the function

Now let us eliminate the duplicate columns from the DataFrame. We can do this using the following code.

print(val.reset_index().T.drop_duplicates().T)

This helps us to easily reset the index and remove duplicate columns from the DataFrame. The output of the code is given below.

	index	dat1
0	0		9
1	1		5

As shown in the figure, we have successfully eliminated the duplicate column named from our DataFrame dat2. It is also important to note that we have valreset the index for the DataFrame, which may help analysts to reconfigure their data points and gather better insights.

Therefore, we eliminate any duplicate columns that may exist in the DataFrame using concatthe and functions.drop_duplicates()

To understand this concept better, you can learn about the following topics.

1. ConcatFunctions in Pandas .
2. Drop DuplicatesFunctions in Pandas .