How to Understand the Spectral Radius of ESN

Introduction

Recently, I read some papers about Echo State Network (ESN) and echo state property which is a valuable property determining the performance of ESN. In this article, some of the understandings about echo state property are recorded below

Spectral Radius

Spectral Radius is a very important parameter, which determine the echo state propert in Echo State Network. Before the discussion, the essential defination of symbols are listed as following:

• $W^{in}$ : a $N\times K$ weight matrix for $hidden$ to $input$ weights.
• $W^{hidden}$ : a $N\times N$ weight matrix for $hidden$ to $hidden$ weights.
• $W^{out}$ : a $L\times (N+K+L)$ weight matrix for $out$ to $(input+hidden+out)$ weights.
• $u(n)$ : Input units at times $n$, $u(n)=\left\{\begin{array}{c}{u}_1(n),u_2(n),...,u_K(n)\end{array}\right\}$
• $x(n)$ : hidden units or internal units at times $n$, $x(n)=\left\{\begin{array}{c}{x}_1(n),x_2(n),...,x_N(n)\end{array}\right\}$
• $y(n)$ : output units at times $n$, $y(n)=\left\{\begin{array}{c}{y}_1(n),y_2(n),...,y_L(n)\end{array}\right\}$
• $\rho (W^{hidden})$ : spectral radius of $hidden$ weight matrix.

The Figure 1 1 shows the basic architecture of ESN.

From wikipedia, the spectral radius of a matrix $A$ is defined as:

ρ ( A ) = max ⁡ { ∣ λ 1 ∣ , ⋯   , ∣ λ n ∣ } \rho(A)=\max \left\{\left|\lambda_{1}\right|, \cdots,\left|\lambda_{n}\right|\right\} ρ(A)=max{ ∣λ1​∣,⋯,∣λn​∣}

where the $A$ is a square matrix of order n. ${\lambda }_{1,\dots }{\lambda }_n$ are the eigenvalues of square matrix $A$.

In many literature, they stated that the well-trained $W^{hidde}$