A Tip A Day – Python Tip #5 – Pandas Concat & Append

In this article we are going to learn about Concat & Append and the comparision between these two functions.

import pandas as pd

Lets take 2 dataframes of fruits.

fruit = {
    'orange' : [3,2,0,1],
    'apple' : [0,3,7,2],
    'grapes' : [7,14,6,15]
}
df1 = pd.DataFrame(fruit)
df1

Output:

fruit = {
    'grapes' : [13,12,10,2,55,98],
    'mango' : [10,13,17,2,9,76],
    'banana' : [20,23,27,4,0,9],
    'pear' : [21,24,28,51,22,25],
    'pineapple' : [30,33,38,30,36,31]
}
df2 = pd.DataFrame(fruit)
df2

Output:

df2 = df2.drop(df2.index[2]) 
df2

Output:

Concat

If both the dataframes contains rows with same index then both the rows will be retained without any change or reset in indexes.

pd.concat((df1, df2), axis = 0)

Output:

If axis = 1 then concatenates in column wise. If no data available with same index, then NaN will be filled.

For example, the second dataframe df2 does not contain record with index 2. So, After concatenation, the records of df2 will have NaN for index 2 record as df1 has record with index 2.

If both the dataframes contains same columns then both the columns will be retained without any change in column name.

pd.concat((df1, df2), axis = 1)

Output:

Append

Append is the specific case of concat which concats the second dataframe’s records at the end of first dataframe.

Append has no axis argument.

Syntax of Append is different from Concat. Append considers the calling dataframe as main object and adds rows to that dataframe from the dataframes that are passed to the function as argument.

If any of the dataframe contains new columns that is not existing in calling dataframe, then it will be added as new column

Syntax: DataFrame.append(other, ignore_index=False, verify_integrity=False, sort=False)

df1

Output:

df2

Output:

df1.append(df2)

Output:

Performance: Which is faster pandas concat or append?

Well, both are almost equally faster.

However there will be a slight change depending on the data.

1. Append function will add rows of second data frame to first dataframe iteratively one by one. Concat function will do a single operation to finish the job, which makes it faster than append().

2. As append will add rows one by one, if the dataframe is significantly very small, then append operation is fine as only a few appends will be done for the number of rows in second dataframe.

%%time
df = pd.DataFrame(columns=['A'])
for i in range(30):
    df = df.append({'A': i*2}, ignore_index=True)

Wall time: 51.4 ms

%%time
df = pd.concat([pd.DataFrame([i*2], columns=['A']) for i in range(30)], ignore_index=True)

Wall time: 9.93 ms

3. Append function will create a new resultant dataframe instead of modifying the existing one. Due to this buffering and creating process, Append operation’s performance is less than concat() function. However Append() is fine if the number of append operation is a very few. If there are a multiple append operations needed, it is better to use concat().

We will meet with a new tip in Python. Thank you! 👍

Like to support? Just click the heart icon ❤️.

Happy Programming!🎈.