Dataframe Warning : SettingWithCopyWarning in python

settingwithcopywarning new column
ignore settingwithcopywarning
self.obj[key] = _infer_fill_value(value)
pandas suppress warnings
pandas loc
python mute settingwithcopywarning
self._setitem_with_indexer(indexer, value)
suppress copy warning python

Processing file from http://portal.amfiindia.com/spages/NAV0.txt to get output as follows: 31012017,1,1,135765,12,10.8536000, 31012017,1,1,135762,12,10.8543000, 31012017,1,1,135760,12,10.6599000, 31012017,1,1,135759,12,10.6554000, 31012017,1,1,135763,12,10.8536000, .. .. ..

I have tried using below code but getting below warning.

CODE:

import pandas
import numpy as np

#Sample file for NAV0.txt can be downloaded from url: http://portal.amfiindia.com/spages/NAV0.txt
#creating pandas with selected columns
df=pandas.read_table('NAV0.txt',sep=';',usecols=['Date','Scheme Code','Net Asset Value'])

#converting column with name 'Scheme Code' to digit to remove string part
fil_df=df[df['Scheme Code'].apply(lambda x : str(x).isdigit())]

#converting column with name 'Net Asset value' to numberic and set each value with 7 decimal places 
fil_df['Net Asset Value']=pandas.to_numeric(fil_df['Net Asset Value'],errors='coerce')
fil_df['Net Asset Value']=fil_df['Net Asset Value'].map(lambda x: '%2.7f' % x)


#Formating Date column as YYYMMDD
fil_df['Date']=pandas.to_datetime(fil_df['Date']).dt.strftime('%d%m%Y')

#adding extra column in dataframe
fil_df['ser1']=1
fil_df['ser2']=1
fil_df['period']=12
fil_df['lcol']=''
fil_df=fil_df[['Date','ser1','ser2','Scheme Code','period','Net Asset Value','lcol']]

#Converting datafile to csv
fil_df.to_csv('NAV_1.csv',index=False,header=None)
fil_df.dtypes

ERROR:

c:\users\administrator\appdata\local\programs\python\python35-32\lib\site-packages\ipykernel__main__.py:12: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

c:\users\administrator\appdata\local\programs\python\python35-32\lib\site-packages\ipykernel__main__.py:13: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

c:\users\administrator\appdata\local\programs\python\python35-32\lib\site-packages\ipykernel__main__.py:17: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

c:\users\administrator\appdata\local\programs\python\python35-32\lib\site-packages\ipykernel__main__.py:20: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

c:\users\administrator\appdata\local\programs\python\python35-32\lib\site-packages\ipykernel__main__.py:21: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

c:\users\administrator\appdata\local\programs\python\python35-32\lib\site-packages\ipykernel__main__.py:22: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

c:\users\administrator\appdata\local\programs\python\python35-32\lib\site-packages\ipykernel__main__.py:23: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

Csv file is getting generated as expected but how can I overcome this warning? I have tried using fil_df.loc[ pandas.to_numeric(fil_df['Net Asset Value'],errors='coerce').map(lambda x: '%2.7f' % x] but it didnt help. Help would be appreciated.

I think you need add copy:

fil_df=df[df['Scheme Code'].apply(lambda x : str(x).isdigit())].copy()

If you modify values in fil_df later you will find that the modifications do not propagate back to the original data (df), and that Pandas does warning.

How to deal with SettingWithCopyWarning in Pandas?, You can safely disable this new warning with the following assignment. df2 = df​[['A']] df2['A'] /= 2 /Library/Frameworks/Python.framework/Versions/3.6/lib/  SettingWithCopyWarning is one of the most common hurdles people run into when learning pandas. A quick web search will reveal scores of Stack Overflow questions, GitHub issues and forum posts from programmers trying to wrap their heads around what this warning means in their particular situation.

If you know what your code is doing, you can use

pd.options.mode.chained_assignment = None  # default='warn'

in your code to disable this warning.

Indexing and Selecting Data, Warning. You can use this access only if the index element is a valid python SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a  c:\users\administrator\appdata\local\programs\python\python35-32\lib\site-packages\ipykernel__main__.py:23: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead. Csv file is getting generated as expected but how can I overcome this warning?

You'll get to the heart of the matter in adding new columns to a DataFrame from this guy's 2017 edit to this answer. Basically the route is to use the .assign('newCol' = enumerableValues )

Setting with copy warning pandas: short story - lu edward, SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. The origin of the warning is called Chained Assignment. What it  Pandas dataframe copy warning. When you go and do something like this: quote_df = quote_df.ix[:,[0,3,2,1,4,5,8,9,30,31]] pandas.ix in this case returns a new, stand alone dataframe. Any values you decide to change in this dataframe, will not change the original dataframe. This is what pandas tries to warn you about. Why .ix is a bad idea

.loc[] = value returns SettingWithCopyWarning · Issue #17476 , This code in Pandas 20.3 throws SettingWithCopyWarning and of df so, even though we're using .loc[] df_q pandas is warning us that it won't  Estou querendo copiar um elemento de um dataframe e inserir num outro dataframe. Em essência há um dataframe com nome x área e outro que preciso carregar com os dados da área, a partir da comparação do nome no primero dataframe. O código está funcionando, mas dá este warning do título. O que estou fazendo:

Selectings Subsets of Data from a Pandas DataFrame, So, why did our assignment with chained indexing fail? Whenever you take a slice of a list, Python creates a brand new copy (a shallow-copy to  If you've been using pandas for a while, you've likely encountered a SettingWithCopyWarning. The proper response is to modify your code appropriately, not to turn off the warning! In this video, I

How do I avoid a SettingWithCopyWarning in pandas?, If you've been using pandas for a while, you've likely encountered a SettingWithCopyWarning Duration: 13:30 Posted: Aug 2, 2016 This is what the warning means by “a value is trying to be set on a copy of a slice from a DataFrame”. As there are no references to this copy, it will ultimately be garbage collected . The SettingWithCopyWarning is letting us know that pandas cannot determine whether a view or a copy was returned by the first __getitem__ call, and so it's unclear whether the assignment changed the original object or not.

Comments
  • I think you need copy - fil_df=df[df['Scheme Code'].apply(lambda x : str(x).isdigit())].copy()
  • I am not getting warning at above line. warning coming from this line : fil_df['Net Asset Value']=pandas.to_numeric(fil_df['Net Asset Value'],errors='coerce')
  • Yes, but problem is in comment. If add copy, still problem?
  • Yes, I tried copy() still same problem
  • @jezrael Thanks! it works.. last time i placed it at another line.