How to leave NaN behind after shifting over

pandas series shift
pandas shift index by 1
pandas groupby shift
pandas shift one column
pandas lag column by group
pandas apply shift
pandas previous row
pandas shift column up

I have a function that shifts the values of one column (Col_5) into another column (Col_6) if that column (Col_6) is blank, like this:

def shift(row):
    return row['Col_6'] if not pd.isnull(row['Col_6']) else row['Col_5']

I then apply this function to my columns like this:

df[['Col_6', 'Col_5']].apply(shift, axis=1)

This works fine, but instead of leaving the original value in Col_5, I need it to shift to Col_6 and in its place, leave a np.nan (so I can apply the same function to the preceeding column.) Thoughts?

fillna + mask: vectorise, not row-wise

With Pandas, you should try to avoid row-wise operations via apply, as these are processed via Python-level loops. In this case, you can use:

null_mask = df['Col_6'].isnull()
df['Col_6'] = df['Col_6'].fillna(df['Col_5'])
df['Col_5'] = df['Col_5'].mask(null_mask)

Notice we calculate and store a Boolean series representing where Col_6 is null first, then use it later to make those values null where values have been moved across via fillna.

The Adventures of a Soldier's Wife, Dick I will leave the keys with my sister Nan, who lives in number 10 Gibbs Avenue,just up behind us, so that when the time comes. If you're on one of your shifts. Tips on Recovering from Night Shifts. In many ways, today’s world is a 24-hour society. Among the professionals who may be required to work night shifts are health care workers, firefighters, police officers, commercial drivers, convenience store clerks and security officers.

import pandas as pd
import numpy as np
df = pd.DataFrame({'Col_5':[1, np.nan, 3, 4, np.nan],
                   'Col_6':[np.nan, 8, np.nan, 6, np.nan]})
col_5 = df['Col_5'].copy()
df.loc[pd.isnull(df['Col_6']), 'Col_5'] = np.nan
df.loc[pd.isnull(df['Col_6']), 'Col_6'] = col_5

Output:

# Original Dataframe:
   Col_5  Col_6
0    1.0    NaN
1    NaN    8.0
2    3.0    NaN
3    4.0    6.0
4    NaN    NaN
# Fill Col_5 with NaN where Col_6 is NaN:
   Col_5  Col_6
0    NaN    NaN
1    NaN    8.0
2    NaN    NaN
3    4.0    6.0
4    NaN    NaN
# Assign the original col_5 values to Col_6:
   Col_5  Col_6
0    NaN    1.0
1    NaN    8.0
2    NaN    3.0
3    4.0    6.0
4    NaN    NaN

Traddlepin Fold; and Other Tales, Nan frowned , struggled , hesitated , and was within “ two tumbles ” of — anything , of Come , " exclaimed the other , following up his suit with a persistence that The soot alighted on the fire , causing it to blaze up , and in that momentary flash in the way — and made for the door , leaving Nan behind to shift for herself . There's no reason why you can't leave things for the next shift, some times it happens. Either you can't do it like in this case, or you simply ran out of time. If it's urgent and needs to be done ASAP, or there's no reason why you didn't do it, or your constantly passing on work that's when there's an issue.

Setup (using the setup from @cosmic_inquiry)

df = pd.DataFrame({'Col_5':[1, np.nan, 3, 4, np.nan],
                   'Col_6':[np.nan, 8, np.nan, 6, np.nan]})

You can look at this problem like a basic swap operation with a mask

numpy.flip + numpy.isnan
a = df[['Col_5', 'Col_6']].values
m = np.isnan(a[:, 1])
a[m] = np.flip(a[m], axis=1)
df[['Col_5', 'Col_6']] = a
np.isnan + loc:
m = np.isnan(df['Col_6'])
df.loc[m, ['Col_5', 'Col_6']] = df.loc[m, ['Col_6', 'Col_5']].values

   Col_5  Col_6
0    NaN    1.0
1    NaN    8.0
2    NaN    3.0
3    4.0    6.0
4    NaN    NaN

Performance

test_df = \
    pd.DataFrame(np.random.choice([1, np.nan], (1_000_000, 2)), columns=['Col_5', 'Col_6'])

In [167]: %timeit chris(test_df)
68.3 ms ± 291 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [191]: %timeit chris2(test_df)
43.9 ms ± 296 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [168]: %timeit jpp(test_df)
86.7 ms ± 394 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [169]: %timeit cosmic(test_df)
130 ms ± 1.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Tales and sketches of Lancashire life, Nan frowned , struggled , hesitated , and was within " two tumbles ” of anything , of Come , ” exclaimed the other , following up his suit with a persistence that The soot alighted on the fire , causing it to blaze up , and in that momentary flash in the way — and made for the door , leaving Nan behind to shift for · herself . Over time, however, a healthy relationship provides both members with a sense of commitment and support from the other. Friends don’t let friends do all the work!

Changing Contexts, Shifting Meanings: Transformations of Cultural , the entire island, killing all its inhabitants and leaving behind a couple to by turning himself into a rock on the Kiti coast that comes to be called Neparak. forces of Nan Madol and their local allies appear poised to carry the day when one of  The unwritten policy at our place is that if admissions come in at the last half-hour of the shift, the next shift has the honor of processing them. Offgoing may get the bulk of the paperwork done and leave the assessment/safety searches/etc. for the oncoming. Whatever isn't done by offgoing is handed over in report to oncoming.

pandas.DataFrame.shift, DataFrame.drop · pandas. That is, use freq if you would like to extend the index when shifting and preserve the original data. The scalar value to use for newly introduced missing values. the default depends on the dtype of self . df.shift(​periods=1, axis='columns') Col1 Col2 Col3 0 NaN 10.0 13.0 1 NaN 20.0 23.0 2  After all, meaningful child care assistance is extremely limited, access to long-term care is piecemeal at best, and too few workers have access to paid family and medical leave, which means that

Python, Pandas dataframe.shift() function Shift index by desired number of periods This function takes a scalar parameter called period, which represents the number of shifts to be made over the desired axis. This function is very helpful when dealing with time-series data. Python exit commands: quit(), exit(), sys.exit() and os. It’s another if tenants leave pieces that just get carried over each time there’s a new occupant. Regardless, when you complete your walk through and find property has been left behind (e.g. clothing, furniture, electronics, etc.), there is a certain process you must follow to either return or get rid of the abandoned property.

Comments
  • I think there are quite a few issues with the code. You have columns hard-coded in your function so it isn't reusable, and this couldn't ever do any operation on the previous row because it is only returning one column.
  • This worked - will look into .mask()