I have a DataFrame df1 :
df1.head() =
wght num_links
id_y id_x
3 133 0.000203 2
186 0.000203 2
5 6 0.000203 2
98 0.000203 2
184 0.000203 2
I need to calculate a variable called thr,
thr = N*(N-1)*2,
where Nis the number of rows of df1.
The problem is that when I calculate thr,Python throws a negative value(although all of the inputs are positive):
ipdb> df1['wght'].count()*(df1['wght'].count()-1)*2
-712569744
Possible hint
The number of rows N is
ipdb> df1['wght'].count()
137736
therefore,
ipdb> 137736*137735*2
37942135920.
Taking into account that the max value that can be assigned to a int32 is 2147483647, I suspect that NumPy considers type(thr) = , when it should be . Does this make sense?
Please note that I have not written the code that generates df1 because
ipdb> df1['wght'].count()
137736
However, if it is needed to reproduce the error, let me know.
Thanks in advance.
解决方案
You are experiencing np.int32 overflow, so just use len(df) instead of df.column.count().
Here is a small demo:
In [149]: x = pd.DataFrame(np.random.randint(0,100,size=(137736, 3)), columns=list('ABC'))
In [150]: x.A.count() * (x.A.count() - 1) * 2
Out[150]: -712569744
In [151]: len(x) * (len(x) - 1) * 2
Out[151]: 37942135920
In [153]: type(x.A.count())
Out[153]: numpy.int32
In [154]: type(len(x))
Out[154]: int