Sigmoid function optimization

Sigmoid function optimization

Abstract

  Neural Networks is an emerging security paradigm. As neural networks are increasingly widely used in various fields, many applications for edge mobile devices have to use neural networks for computation. However, due to the restrictions on computational resources and computing resources in the edge, we usually need to optimize the neural networks before they can be applied on edge devices. In this article, I describe how to optimize a neural network by optimizing the Sigmoid activation function.

1. Introduction

  The $Sigmoid$ function, also called the logistic function, is used for the output of the hidden layer neurons and takes values in the range $(0,1)$, which can map a real number to the interval $(0,1)$ and can be used for binary classification. $Sigmoid$ is smooth and easy to derive.

The $Sigmoid$ function is defined by the following equation:
$$sigmoid(x)=\frac{1}{1+e^{-x}}$$
Its derivative with respect to $x$ can be expressed in terms of itself as
$$S^{\prime }(x)=\frac{e^{-x}}{{\left(1+e^{-x}\right)}^2}=S(x)(1-S(x))$$

  In the $Sigmoid$ function, there is an $exp$ function and a $division$, both calculations are very cumbersome and time consuming , which is very unfriendly to edge devices with limited resources. So it is a good idea to optimize $Sigmoid$ in terms of $exp$ function and $division$

2. Optimization Analysis

$$sigmoid(x)=\frac{1}{1+e^{-x}}$$
$$\Downarrow$$
$$\frac{1}{y}=1+e^{-x}$$
$$\Downarrow$$
$$\frac{1}{y}-1=e^{-x}$$
$$\Downarrow$$
$$ln(\frac{1}{y}-1)=-x$$
$$\Downarrow$$
$$-ln(\frac{1}{y}-1)=x$$
$$\Downarrow$$
$$f^{-1}(y)=-ln(\frac{1}{y}+-1)$$

# The original code
if Sigmoid(P[4]) > threshold then 
	do smothing

# The new code
if P[4] > desigmoid_threshold
    do smothing

3. Algorithm

def sigmoid(x):
	return 1 / (1 + math.exp(-x))

def desigmoid(x):
	return -math.log(1 / y - 1)

4. Future work

Although the computational efficiency has been greatly improved after $desigmoid$ optimization, exponential overflow can easily occur when the data volume become large, and we need to find an effective way to solve this problem. I am now using $log$ function to smooth the data, which reduces the probability of overflow to some extent, but I need to find a more efficient method to solve exponential overflow problem in the future.