python SettingWithCopyWarning 原理及解决办法

Regardless of how long you worked with pandas, be it a day or a year, sooner or later you are likely to run into the infamous SettingWithCopyWarning. In this article, I explain what causes the problem and how to properly address the issue.

Warning not an error

Before I dive into the technicalities, I want to highlight that SettingWithCopyWarning is — as the name suggests — a warning, not an error. So the code we are executing will most likely not break and produce the end result. However, the end result might not be the one we actually intended to obtain.

The reason why I wanted to highlight the distinction is that we might be tempted to ignore the warning when we see that the code actually succeeds in returning a result. And as a matter of fact, the result might be correct! The best practice is to be extra careful and actually understand the underlying principles. This way, we can often save a lot of time trying to identify an obscure bug, which we could have avoided in the first place.

Views versus copies

The key concepts that are connected to theSettingWithCopyWarning are views and copies. Some operations in pandas (and numpy as well) will return views of the original data, while other copies.

To put it very simply, a view is a subset of the original object (DataFrame or Series) linked to the original source, while a copy is an entirely new object. A consequence of the distinction is that when we modify the view, we modify the original object as well. That does not happen with copies, as they are not connected to the original objects.

Having described the difference, SettingWithCopyWarning is actually letting us know that the code we have written might have done one thing, when in fact we wanted to do another. I will illustrate this with a real-life example. Imagine having a large DataFrame. For some analysis, you filter (slice) the DataFrame to only contain a subset of the full data, for example, users from a certain country. Then, you might want to modify some values in the extracted DataFrame, let’s say cap the maximum value of a feature at 100. This is the typical case in which you could encounter the infamous warning — you wanted to only modify the extracted frame, while you ended up modifying the source data. You can easily imagine that this is not something you wanted to do and can lead to potential problems later on.

Note: To understand whether a frame is a view of a copy, you can use the internal _is_view and _is_copy methods of a pandas.DataFrame. The first one returns a boolean, while the second either a weakref to the original DataFrame, or None.

Common occurrences of the warning

In this section, I go over the most common cases when the SettingWithCopyWarning