Concatenating DataFrames through function calls

pandas concat
pandas merge on different column names
pandas merge columns into one
python pandas concatenate multiple columns
pandas join vs merge
pandas merge multiple dataframes
merging two csv files with a common column python
concatenate list of dataframes

I am trying to concatenate a single row dataframe (df) and add it to the end of another dataframe (df_all) using the following code:

import pandas as pd
import numpy as np
from IPython.display import display, HTML

global df_all    
df_all = pd.DataFrame()

def save_data(df):
     df_all = pd.concat([df, df_all], axis=0)
     display(df_all)
     return df_all

def opt():
    df = pd.DataFrame(np.random.randn(1, 4), columns=list('ABCD'))  # one row data
    display(df)
    save_data(df)

Using this loop I suppose to get 3 rows are saved to df_all. But I am getting an error message (local variable 'df_all' referenced before assignment)

for i in range (3):
    opt()
    display(df_all)

I don't believe in functions that rely on global variables—it just isn't good hygiene.

Functions should be pure. First, define your opt function. This just generates df and nothing more.

def opt():
    df = ...  # df is generated here
    return df

Next, define save_data. Well, I've renamed it to augment to be more in line with what you're doing. This concatenates two DataFrames together.

def augment(df, df_new):
    return pd.concat([df, df_new], axis=1)

Finally, your mainloop. All state is maintained here, not in the functions—

df_all = pd.DataFrame()
for i in range(3):
    df_all = augment(df_all, opt())
    display(df_all)

Merge, join, and concatenate, The concat() function (in the main pandas namespace) does all of the heavy lifting of indicator : Add a column to the output DataFrame called _merge with  pandas.concat() function concatenates the two DataFrames and returns a new dataframe with the new columns as well. The dataframe row that has no value for the column will be filled with NaN short for Not a Number.

move global df_all to save_data(df) block. like below

def save_data(df):
    global df_all    
    df_all = pd.concat([df, df_all], axis=0)
    display(df_all)
    return df_all

Combining DataFrames with Pandas – Data Analysis and , We can use the concat function in pandas to append either columns or rows from one Combining DataFrames using a common field is called “joining”. Concatenating objects¶ The concat() function (in the main pandas namespace) does all of the heavy lifting of performing concatenation operations along an axis while performing optional set logic (union or intersection) of the indexes (if any) on the other axes. Note that I say “if any” because there is only a single possible axis of concatenation for Series.

This approach avoids use of global variable. Pass df_all as an argument to opt().

def save_data(df, df_all):
    df_all = pd.concat([df, df_all], axis=0)
    return df_all

def opt(df_all):
    df = pd.DataFrame(np.random.randn(1, 4), columns=list('ABCD'))  # one row data
    df_all = save_data(df, df_all)
    return df_all

df_all = pd.DataFrame()
for i in range(3):
    df_all = opt(df_all)
    display(df_all)

Concatenation (Combining Data Tables) in Python and Pandas: A , Throughout the tutorial, I will refer to DataFrames and tables interchangeably. when you only want to add new rows called the DataFrame.append() method. In python pandas concat tutorial, we learn how to concatenate different pandas series and DataFrame using multiple parameters of pd.concat() function. According to the structure of data use appropriate parameters which we have learned. If pd.concat() the function is not able to concatenate your data perfectly then you can refer below methods.

Python for Librarians: Combining DataFrames with pandas, We can use the concat function in Pandas to append either columns or rows from one Combining DataFrames using a common field is called “joining”. To join these DataFrames, pandas provides multiple functions like concat(), merge() , join(), etc. In this section, you will practice using merge() function of pandas. You can join DataFrames df_row (which you created by concatenating df1 and df2 along the row) and df3 on the common column (or key) id.

Joining DataFrames in Pandas, Joining and merging DataFrames is the core process to start with data the argument ignore_index as True while calling the concat() function: Concatenate pandas objects along a particular axis with optional set logic along the other axes. Can also add a layer of hierarchical indexing on the concatenation axis, which may be useful if the labels are the same (or overlapping) on the passed axis number. Parameters objs a sequence or mapping of Series or DataFrame objects

paste: Concatenate Strings, Concatenate vectors after converting to character. paste converts its arguments (via as.character ) to character strings, and concatenates them (separating them by the string given by sep ). toString typically calls paste(*, collapse=", ") . by: Apply a Function to a Data Frame Split by Factors c: Combine Values into a  Code #1 : DataFrames Concatenation concat() function does all of the heavy lifting of performing concatenation operations along an axis while performing optional set logic (union or intersection) of the indexes (if any) on the other axes.

Comments
  • Recommending globals, even if the OP is asking for it, is something I don't condone.
  • Agree with you. I just fix the bug, but didn't consider whether this is a good way.
  • Thanks, Lambda. It works. I don't know how I did this silly mistake :)
  • This is effectively my answer, except with a argument here instead of there.