How to Understand the Echo State Property of ESN
Introduction
Recently, I read some papers about Echo State Network (ESN) and echo state property which is a valuable property determining the performance of ESN. In this article, some of the understandings about echo state property are recorded below
Spectral Radius
Spectral Radius is a very important parameter, which determine the echo state propert in Echo State Network. Before the discussion, the essential defination of symbols are listed as following:
- W i n W^{in} Win : a N × K N\times K N×K weight matrix for h i d d e n hidden hidden to i n p u t input input weights.
- W h i d d e n W^{hidden} Whidden : a N × N N\times N N×N weight matrix for h i d d e n hidden hidden to h i d d e n hidden hidden weights.
- W o u t W^{out} Wout : a L × ( N + K + L ) L\times (N+K+L) L×(N+K+L) weight matrix for o u t out out to ( i n p u t + h i d d e n + o u t ) (input+hidden+out) (input+hidden+out) weights.
- u ( n ) u(n) u(n) : Input units at times n n n, u ( n ) = { u 1 ( n ) , u 2 ( n ) , . . . , u K ( n ) } u(n) = \begin{Bmatrix} u_{1}(n), u_{2}(n), ...,u_{K}(n)\end{Bmatrix} u(n)={ u1(n),u2(n),...,uK(n)}
- x ( n ) x(n) x(n) : hidden units or internal units at times n n n, x ( n ) = { x 1 ( n ) , x 2 ( n ) , . . . , x N ( n ) } x(n) = \begin{Bmatrix} x_{1}(n), x_{2}(n), ...,x_{N}(n)\end{Bmatrix} x(n)={ x1(n),x2(n),...,xN(n)}
- y ( n ) y(n) y(n) : output units at times n n n, y ( n ) = { y 1 ( n ) , y 2 ( n ) , . . . , y L ( n ) } y(n) = \begin{Bmatrix} y_{1}(n), y_{2}(n), ...,y_{L}(n)\end{Bmatrix} y(n)={ y1(n),y2(n),...,yL(n)}
- ρ ( W h i d d e n ) \rho(W^{hidden}) ρ(Whidden) : spectral radius of h i d d e n hidden hidden weight matrix.
The Figure 1 1 shows the basic architecture of ESN.
From wikipedia, the spectral radius of a matrix A A A is defined as:
ρ ( A ) = max { ∣ λ 1 ∣ , ⋯   , ∣ λ n ∣ } \rho(A)=\max \left\{\left|\lambda_{1}\right|, \cdots,\left|\lambda_{n}\right|\right\} ρ(A)=max{ ∣λ1∣,⋯,∣λn∣}
where the A A A is a square matrix of order n. λ 1 , … λ n \lambda_{1, \dots} \lambda_{n} λ1,…λn are the eigenvalues of square matrix A A A.
In many literature, they stated that the well-trained W h i d d e n W^{hidden} Whidde