DAY4(Quantization&Tensor)

Quantization

model choose： quantize model architectures that are already efficient at trading off latency with accuracy.

represent weights to binary, ternary and use bit-shift to do multiplication.

• Quantization values(parameters) to do integer arithmetic operation.

• scheme: integer-arithmetic inference and floating-point training.

• equivalent to requiring quantization scheme to mapping integers q(bit representation, quantized value) to real number r
  r=S(q-Z)
  use a single set of quantization parameters {S,Z} for all values within each activations array and each weights array

• S(scale)-usually an arbitrary positive real floating number

• Z(zero-point)-quantities the value q to 0

multiple operation:

• for r3=r1*r2
  
• M=S1S2/S3, be in the interval of (0,1), and can be therefore expressed as normalized form
  M=2^(-n)*M0 (M0 is in the interval [0.5,1))
  takes 2N^3 operations
• q1 matrix to be the weight and q2 matrix to be the activations
• int32+=uint8*uint8
• Sbias=S1*S2, Zbias=0

Training with simulated quantization

• simulate quantization in the forward pass of training

• weights and biases stored in float when backpropagation

• weights are quantized before they are convolved with the input

• activations are quantized at points where they would be during inference

• biases quantization parameters are inferred from weights and activations (32 bit integers)

For each layer, quantization is parameterized by the number of quantization levels and clamping range

r is a real-valued number to be quantized, [a;b] is the quantization range, n is the number of quantization levels, [.] denotes rounding to the nearest integer
n is fixed for all layers. eg. n=2^8=256 for 8 bit quantization

Learning quantization ranges

• For weights:
• a:=min w, b:=max w
• make a tweak to range from [-127,127]
• For activations:
• ranges depend on the inputs to the network
• the learned quantization parameters map to the scale S and zero-point Z in equation S = s(a, b, n), Z = z(a, b, n)

Tensor

Tensor对象的3个属性：
rank: number of dimensions
shape: number of rows and columns
type: data type of tensor’s elements