Multiple for loops to create a pandas dataframe

create pandas dataframe from loop
for loop pandas dataframe
advanced for loop in python
how to put the results of loop into a dataframe python
nested for loop python dataframe
for loop multiple dataframes python
pandas for loop
pandas iterate over rows and create new column

I am trying to create a pandas dataframe that looks like this:

          -5      0      5
index                     
-5       NaN  slope  slope
 0     slope    NaN  slope
 5     slope  slope    NaN

but the closest I can get is the code below which returns a dataframe with only one column (which is the list from the last iteration through the ctr1 loop)

weather = np.linspace(-5, 5, 3)

for ctr1 in weather:
    slope_list = []
    X1 = round(ctr1,1)
    for ctr2 in weather:
        X2 = round(ctr2,1)

        Y1 = regressor[0] * X1**3 + \
        regressor[1] * X1**2 + \
        regressor[2] * X1 + \
        regressor[3] 

        Y2 = regressor[0] * X2**3 + \
        regressor[1] * X2**2 + \
        regressor[2] * X2 + \
        regressor[3]

        slope = (Y2-Y1)/(X2-X1)
        slope_list.append(slope)

    df_final = pd.DataFrame({X1:slope_list})

Can anyone help?

You can try directly assign values in DataFrame. Just create empty DataFrame with index=weather:

import numpy as np
weather = np.linspace(-5, 5, 3)
df_final = pd.DataFrame([], index=weather)
for ctr1 in weather:
    X1 = round(ctr1,1)
    for ctr2 in weather:
        X2 = round(ctr2,1)

        Y1 = regressor[0] * X1**3 + \
        regressor[1] * X1**2 + \
        regressor[2] * X1 + \
        regressor[3] 

        Y2 = regressor[0] * X2**3 + \
        regressor[1] * X2**2 + \
        regressor[2] * X2 + \
        regressor[3]

       slope = (Y2-Y1)/(X2-X1)

       df_final.loc[X1, X2] = np.NaN if X1 == X2 else slope

Iterating over rows and columns in Pandas DataFrame , ' iterrows() returns an iterator containing index of each row and the data in each row as a Series. Since iterrows() returns iterator, we can use next function to see the content of the iterator. Create a pandas column using for loop Let’s see how to create a column in pandas dataframe using for loop. Such operation is needed sometimes when we need to process the data of dataframe created earlier for that purpose, we need this type of computation so we can process the existing data and make a separate column to store the data.

df_final is getting only 3 elements because it's at the same indentation level as for ctr2 in weather, so it's getting reassigned every outer loop. Although, if you fix that, you'll get a dataframe that's only a single long column: you only have a single slope_list getting appended to that turns into a dataframe at the end.

This is how I would solve that without changing your assignment method:

weather = np.linspace(-5, 5, 3)
slope_list = []
for ctr1 in weather:
X1 = round(ctr1,1)
for ctr2 in weather:
    X2 = round(ctr2,1)

    Y1 = regressor[0] * X1**3 + \
    regressor[1] * X1**2 + \
    regressor[2] * X1 + \
    regressor[3] 

    Y2 = regressor[0] * X2**3 + \
    regressor[1] * X2**2 + \
    regressor[2] * X2 + \
    regressor[3]

    slope = (Y2-Y1)/(X2-X1)
    slope_list.append(slope)


#make it 3 columns and 3 rows as intended
slope_list = np.array(slope_list).reshape(3, 3)
#make the dataframe
df_final = pd.DataFrame({X1:slope_list})
#manually add the desired row and column indexes
df_final = df.set_index(weather)
df_final.columns = weather

Although you should keep in mind that unless you know exactly what you're doing, making loops and nested loops when working with pandas usually means you're missing a much easier and better way to go about things.

How do you append rows in a Pandas DataFrame in a for loop? Create a column using for loop in Pandas Dataframe Let’s see how to create a column in pandas dataframe using for loop. Such operation is needed sometimes when we need to process the data of dataframe created earlier for that purpose, we need this type of computation so we can process the existing data and make a separate column to store the

slope_list = [] resets resulting list at each iteration, so just the last one remains. You need to define result list outside the outer loop, and append subresults to it.

How do I iterate over rows and columns in pandas? 567 µs ± 81.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) %prun -l 4 using_pandas_builtin(df) 645 function calls (642 primitive calls) in 0.002 seconds Ordered by: internal time

As mentioned, for completeness, I have posted an answer to my question that works with a larger weather array. The only difference is that I did the rounding earlier in the code:

weather = np.round(np.linspace(-5, 35, 401), decimals = 1)
df_final = pd.DataFrame([], index=weather)
for ctr1 in weather:
    X1 = ctr1
    for ctr2 in weather:
        X2 = ctr2

        Y1 = regressor[0] * X1**3 + \
        regressor[1] * X1**2 + \
        regressor[2] * X1 + \
        regressor[3] 

        Y2 = regressor[0] * X2**3 + \
        regressor[1] * X2**2 + \
        regressor[2] * X2 + \
        regressor[3]

        slope = (Y2-Y1)/(X2-X1)

        df_final.loc[X1, X2] = np.NaN if X1 == X2 else slope

Create multiple dataframes in loop · python pandas dataframe. I have a list, with each entry being a company name companies =  Create A pandas Column With A For Loop. 20 Dec 2017. Preliminaries. import pandas as pd import numpy as np. Create an example dataframe. raw_data = {'student_name':

We learned how to iterate over different types of data structures, and how loops can be used with pandas DataFrames and matplotlib to create multiple traces or​  Here, you are overwriting the year index with each loop and therefore only the last continent dataframe is remaining for years 2010-2014. df_highest_countries[year] = pd.DataFrame(highest_countries) Here, you can add continent and then concatenate to one final dataframe. Adding continent results in having a more unique dictionary key.

Split a text column into two columns in Pandas DataFrame · Python | Change column names and row indexes in Pandas DataFrame · Jitender_1998. Check out  Once dis created the DataFramefor company xcan be retrieved as d[x], so you can look up a specific company quite easily. To operate on all companies you would typically use a loop like: for name, df in d.items(): # operate on DataFrame 'df' for company 'name'. In Python 2 you are better writing.

Here, you are overwriting the year index with each loop and therefore only the last continent dataframe is remaining for years 2010-2014. Use a for loop to create another list called dataframes containing the three DataFrames loaded from filenames: Iterate over filenames. Read each CSV file in filenames into a DataFrame and append it to dataframes by using pd.read_csv() inside a call to .append(). Print the first 5 rows of the first DataFrame of the list dataframes. This has been done for you, so hit 'Submit Answer' to see the results.

Comments
  • Please provide expected output.
  • Hi andrew_reece, the expected output is as I have shown right at the top where each 'slope' is a number (based on the calculation using X1, X2, Y1, Y2).
  • For integer positional setting, I recommend you use the more efficient .iat instead of .loc.
  • Hi Alexey, thank you for your solution. For completeness, I thought I would provide the only update I needed to add for it to work perfectly for a larger weather array. I have added my own solution based mostly on your code.
  • Thanks Kosist, the reason I reset slope_list inside the outer loop is because I only want each column to have 3 rows. If I set it outside the outer loop, I end up with a list with 9 values.
  • Could you post regressor data also (better then full code), so we could try to run the code? B/c now you reset data by this, and that's why there is just one column with the last ctr1 iteration.