20240605日志:OmniQuant

1 basic programming knowledge about this article’s code

1.1 Abstractclass, abstractmethod, classmethod and metaclass

Abstractclass, abstractmethod is mainly used to define a set of specifications. Ensure that the implementation of the class follows a specific interface specification. For example, abstractmethods are created according to Google’s abc.py:

def abstractmethod(funcobj):
    """A decorator indicating abstract methods. Requires that the metaclass is ABCMeta or derived from it.  A class that has a metaclass derived from ABCMeta cannot be instantiated unless all of its abstract methods are overridden. The abstract methods can be called using any of the normal 'super' call mechanisms.  abstractmethod() may be used to declare abstract methods for properties and descriptors.
    Usage:
        class C(metaclass=ABCMeta):
            @abstractmethod
            def my_abstract_method(self, ...):
                ...
    """
    funcobj.__isabstractmethod__ = True
    return funcobj

if we want to instantiate a metaclass(Father), all methods of this metaclass(Father) must be overridden.
usage:

from abc import ABCMeta, abstractmethod

class Father(metaclass=ABCMeta):
    def __init__(self, name, sex):
        self.name = name
        self.sex = sex
    
    @abstractmethod
    def my_instance_method(self):
        """这是一个抽象方法，子类必须实现它"""
        pass

	@classmethod
	@abstractmethod
	def my_class_method(cls):
		"""这是一个抽象类,需要子类实现它，cls 是类方法的第一个参数，它是一个对类的引用，允许类方法访问和修改类的状态"""
		pass
		
	class Son(Father):
	    def my_instance_method(self):
	        print("实例方法的实现")
	    @classmethod
	    def my_class_method(cls):
	        print("类方法的实现", cls)
# 调用类方法
Son.my_class_method()  # 输出: 类方法的实现 <class '__main__'.Son>

So, if u want to instantiate ‘Father’, all methods of this ‘Father’ must be overridden.

1.2 property

add a read-only property which can be deleted to an instantiated class.

class Circle:
    def __init__(self, radius):
        self.radius = radius
    @property
    def diameter(self):
        return self.radius * 2
    @diameter.setter
    def diameter(self, value):
        self.radius = value / 2
    @diameter.deleter
    def diameter(self):
        del self.radius

# 使用示例
circle = Circle(5)
print(circle.radius)    # 输出: 5
print(circle.diameter)  # 输出: 10

circle.diameter = 8     # 如果没有@diameter.setter这一步将不能操作
print(circle.radius)    # 输出: 4
print(circle.diameter)  # 输出: 8

del circle.diameter
print(circle.radius)    # 抛出 AttributeError: 'Circle' object has no attribute 'radius'

2 Some quantization methods

Fig. 2.1 mixed-precision quantization

SpQR,OWQ,AWQ¹

3 OmniQuant

the goal of a quantization
$$\arg \underset{{\Theta }_1,{\Theta }_2}{\min }||\mathcal{F}(\mathbf{W},\mathbf{X})-\mathcal{F}(Q_w(\mathbf{W};{\Theta }_1,{\Theta }_2),Q_a(\mathbf{X},{\Theta }_2))||\text{\hspace{1em}(1)}$$
where $\mathcal{F}$ represents the mapping function for a transformer block, $\mathbf{W}$ and $\mathbf{X}$ are full-precision weight ant activation, $Q_w$ and $Q_a$ represent weight and activation quantizer, ${\Theta }_1$ and ${\Theta }_2$ are quantization parameters in LWC and LET. The idea of OmniQuant nearly all comes from AWQ.

3.1 Initialization of Quantization Parameters

3.2 Block-wise Quantization Error Minimization

3.3 Learnable Weight Clipping (LWC)

weight-only quantization：
$${\mathbf{W}}_{\mathbf{q}}=\mathrm{clamp}(\lfloor \frac{\mathbf{W}}{h}\rceil +z,0,2^N-1),\mathrm{where}h=\frac{\gamma \max (\mathbf{W})-\beta \min (\mathbf{W})}{2^N-1},z=-\lfloor \frac{\beta \min (\mathbf{W})}{h}\rceil \text{\hspace{1em}(2)}$$
where $\gamma \in [0,1]$ and $\beta \in [0,1]$, the meaning of $\mathrm{clamp}(\lfloor \frac{\mathbf{W}}{h}\rceil +z,0,2^N-1)$ is if a number big than $2^N-1$ or smaller than $0$ they will be sat as $2^N-1$ or $0$.Not be very special.
here is an example² for this:

Fig. 2.2 Asymmetric quantization

So, Θ 1 = { γ , β } \Theta_1=\{\gamma,\beta\} Θ1​={γ,β} in Eqn. 1.

3.4 Learnable Equivalent Transformation (LET)

$$\mathbf{Y}=\mathbf{XW}+\mathbf{B}=[\underset{\tilde{\mathbf{X}}}{\underbrace{(\mathbf{X}-\delta )\oslash s}}]\cdot [\underset{\tilde{\mathbf{W}}}{\underbrace{s\odot \mathbf{W}}}]+[\underset{\tilde{\mathbf{B}}}{\underbrace{\mathbf{B}+\delta \mathbf{W}}}]\text{\hspace{1em}(3)}$$
where $Y$ represents the output, $\mathbf{s}\in {\mathbb{R}}^{1\times C_{in}}$ and $\delta \in {\mathbb{R}}^{1\times C_{in}}$ are channel-wise scaling and shifting parameters, respectively, $\tilde{\mathbf{X}}$ $\tilde{\mathbf{W}}$ and $\tilde{\mathbf{B}}$ are equivalent activation, weight and bias. $\odot$ and $\cdot$ are elementwise multiplication and division.
Finally, quantization on transformed activations and weights are performed as given by
$$\mathbf{Y}=Q_a(\tilde{\mathbf{X}})Q_w(\tilde{\mathbf{W}})+\tilde{\mathbf{B}}\text{\hspace{1em}(4)}$$
where $Q_a$ is the vanilla MinMax quantizer and $Q_a$ is the MinMax quantizer with LWC.
The learning equivalent transform of the self-attention affinity matrix can be written as:
$$\mathbf{P}=\mathrm{Softmax}(\mathbf{Q}{\mathbf{K}}^T)=\mathrm{Softmax}((\underset{\tilde{\mathcal{Q}}}{\underbrace{\mathbf{Q}\otimes s_a}})(\underset{{\tilde{\mathcal{K}}}^T}{\underbrace{s_a\odot {\mathbf{K}}^T}}))$$
where $s_a\in {\mathbb{R}}^{1\times C_{out}}$ is the scaling factor in the affinity matrix. Similar to Eqn. 3, the quantized affinity matrix calculation is expressed a $\mathbf{P}=\mathrm{Softmax}(Q_a(\tilde{\mathbf{Q}})Q_a({\tilde{\mathbf{K}}}^T))$.
And Θ 2 = { δ , s , s a } \Theta_{2}=\{\delta,s,s_{a}\} Θ2​={δ,s,sa​} in Eqn. 1.

3.5 Quantization Operation

3.6 Training and Optimization

3.7 Iterative Calibration

3.8 Model Evaluation

Deployment

$reference$

---

1. OMNIQUANT: OMNIDIRECTIONALLY CALIBRATED QUANTIZATION FOR LARGE LANGUAGE MODELS ↩︎

2. RethinkFun 2024 模型量化一：量化基础 对称量化 非对称量化 极大值量化 零点量化 ↩︎