[machine learning] MACS、MACs、FLOPS、FLOPs

本文介绍机器学习中衡量一个模型计算复杂度的四个指标：MACS、MACs、FLOPS、FLOPs。

首先从含义上讲，可以分类两类：MACS/FLOPS和MACs/FLOPs。MACs/FLOPs表示总的操作数(后缀s可以看成是表示复数)，MACS/FLOPS表示每秒可以执行的操作数(即：MACs per Second/FLOPs per Second)。

从名称上讲，MAC (Multiply-Accumulate Operation)表示乘加操作，FLOP (Floating Point Operation)表示浮点操作，很容易可以得到一次MAC是两次FLOP，即：FLOPs = 2 × MACs。

下面以一个简单的例子，计算模型的MACs:

假设模型是一个三层的FFN模型，每一层的Neuron数都是1024个，输入维数是4，输出维数是2，求这个模型的MACs。


第一层：MACs = 1024×4 = 4096
第二层：MACs = 1024×1024 = 1,048,576
第三层：MACs = 2×1024 = 2048
Total MACs = 4096 + 1,048,576 + 2048 = 1,054,720

我们也可以简单说这个模型的计算复杂度是2 MFLOPs(2×MACs)

在PyTorch中，我们可以使用fvcore第三方库直接得到模型的预估计算复杂度：

import torch
from torchvision.models import resnet50
from fvcore.nn import FlopCountAnalysis

# Example model, replace with your model
model = resnet50()

# Example input, replace with the appropriate input size for your model
inputs = torch.randn(1, 3, 224, 224)

# Calculate FLOPs
flops = FlopCountAnalysis(model, inputs)
print(f"FLOPs: {flops.total()}")

根据复杂度的级数，对应不同FLOPs的称谓：

• KFLOPs: 10^3 (thousand)
• MFLOPs: 10^6 (million)
• GFLOPs: 10^9 (billion)
• TFLOPs: 10^12 (trillion)

了解以上的概念之后，我们拿到一个模型和一台机器，就能大概评估出这个模型在这台机器上运行的性能。比如说一个模型的计算复杂度是1 GFLOPs，机器CPU的性能是2 GFLOPS，不考虑其他性能损耗，理想状况下这台机器一秒钟可以推理这个模型两次。

最后贴一下NVIDIA一些常见GPU的性能：

    # https://www.techpowerup.com/gpu-specs/h100-pcie-80-gb.c3899
    "H100": {
        torch.float32: 51.22e12,  # 51.22 TFLOPS for FP32 on NVIDIA H100
        torch.float16: 204.9e12,  # 204.9 TFLOPS for FP16 on NVIDIA H100
        torch.bfloat16: 204.9e12
    },
    # https://www.techpowerup.com/gpu-specs/l4.c4091
    "L4": {
        torch.float32: 30.29e12,  # 30.29 TFLOPS for FP32 on NVIDIA L4
        torch.float16: 30.29e12,  # 30.29 TFLOPS for FP16 on NVIDIA L4
        torch.bfloat16: 30.29e12
    },
    # https://www.techpowerup.com/gpu-specs/tesla-t4.c3316
    "T4": {
        torch.float32: 8.1e12,  # 8.1 TFLOPS for FP32 on NVIDIA T4
        torch.float16: 65.13e12,  # 65.13 TFLOPS for FP16 on NVIDIA T4
        torch.bfloat16: 65.13e12
    },
    # https://www.techpowerup.com/gpu-specs/a10g.c3798
    "A10G": {
        torch.float32: 31.52e12,  # 31.52 TFLOPS for FP32 on NVIDIA A10G
        torch.float16: 31.52e12,  # 31.52 TFLOPS for FP16 on NVIDIA A10G
        torch.bfloat16: 31.52e12
    },
    # https://www.techpowerup.com/gpu-specs/a100-pcie-40-gb.c3623
    "A100": {
        torch.float32: 19.49e12,  # 19.49 TFLOPS for FP32 on NVIDIA A100
        torch.float16: 77.97e12,  # 77.97 TFLOPS for FP16 on NVIDIA A100
        torch.bfloat16: 77.97e12
    },
    # https://www.techpowerup.com/gpu-specs/geforce-rtx-3080.c3621
    "RTX_3080": {
        torch.float32: 29.77e12,  # 29.77 TFLOPS for FP32 on NVIDIA RTX 3080
        torch.float16: 29.77e12,  # 29.77 TFLOPS for FP16 on NVIDIA RTX 3080
        torch.bfloat16: 29.77e12
    },
    # https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622
    "RTX_3090": {
        torch.float32: 35.58e12,  # 35.58 TFLOPS for FP32 on NVIDIA RTX 3090
        torch.float16: 35.58e12,  # 35.58 TFLOPS for FP16 on NVIDIA RTX 3090
        torch.bfloat16: 35.58e12
    }