python性能优化

优化

代码实现

• 使用迭代器
• 使用生成器
• 使用join 拼接字符串, 不用 +

cython

• 利用cython 将计算逻辑转为cython，编译成so，有1~2倍的提速

方法可参见https://blog.csdn.net/kevin_darkelf/article/details/98624120

• 函数参数和返回值都使用 静态类型

pypy

• Embedding PyPy
  已不被推荐， 替换方案是 CFFI’s native embedding support

不适用于使用第三方库的情况

• 例子：uWSGI Pypy插件

numba

• 官网手册
• 用户手册值得一看：https://numba.readthedocs.io/en/stable/user/index.html

安装

pip install numba

使用

以下示例 说明 numba的适用场景
Numba works well on code that looks like this:

from numba import jit
import numpy as np

x = np.arange(100).reshape(10, 10)

@jit(nopython=True) # Set "nopython" mode for best performance, equivalent to @njit
def go_fast(a): # Function is compiled to machine code when called the first time
    trace = 0.0
    for i in range(a.shape[0]):   # Numba likes loops
        trace += np.tanh(a[i, i]) # Numba likes NumPy functions
    return a + trace              # Numba likes NumPy broadcasting

print(go_fast(x))

It won’t work very well, if at all, on code that looks like this:

from numba import jit
import pandas as pd

x = {'a': [1, 2, 3], 'b': [20, 30, 40]}

@jit
def use_pandas(a): # Function will not benefit from Numba jit
    df = pd.DataFrame.from_dict(a) # Numba doesn't know about pd.DataFrame
    df += 1                        # Numba doesn't understand what this is
    return df.cov()                # or this!

print(use_pandas(x))

其它

other thins of instrest 也有一些值得注意的地方

@cfunc - declare a function for use as a native call back (to be called from C/C++ etc). Docs are here.

CFFI

• Using CFFI for embedding

Pythran

Pythran is an ahead of time compiler for a subset of the Python language, with a focus on scientific computing. It takes a Python module annotated with a few interface descriptions and turns it into a native Python module with the same interface, but (hopefully) faster.

It is meant to efficiently compile scientific programs, and takes advantage of multi-cores and SIMD instruction units.

Until 0.9.5 (included), Pythran was supporting Python 3 and Python 2.7. It now only supports Python 3

doc: https://pythran.readthedocs.io/en/latest/index.html

---

Profile

Profile

python 自带的profile 模块：https://docs.python.org/2/library/profile.html

---

ref

• python性能优化 写得很好
• python 优化策略
• oneDAL: Intel oneAPI Data Analytics Library
  • Improving the Performance of XGBoost and LightGBM Inference