Linear Layer

Definition

Linear Layer

In neural networks, a linear layer, sometimes called dense layer, is a function of a linear combination between weights $\mathbf{W}\in {\mathbb{R}}^{m\times n}$ and a bias $\vec{b}\in {\mathbb{R}}^m$:

$$\begin{gathered} \begin{array}{rl}{\mathbf{h}}^{(i)}({\theta }^{(i)};\mathbf{X})& ={\left[\begin{array}{ccc}{\theta }_{0,1}^{(1)}& \dots & {\theta }_{0,m}^{(1)}\\ ⋮& \ddots & ⋮\\ {\theta }_{n,1}^{(1)}& \dots & {\theta }_{n,m}^{(1)}\end{array}\right]}^{\text{T}}\left[\begin{array}{c}1\\ {\mathbf{x}}_1\\ ⋮\\ {\mathbf{x}}_n\end{array}\right]\\ & \\ & =\underset{{{\mathbf{W}}^{(i)}}^{\text{T}}}{\underbrace{ \\ {\left[\begin{array}{ccc}{\theta }_{1,1}^{(1)}& \dots & {\theta }_{1,m}^{(1)}\\ ⋮& \ddots & ⋮\\ {\theta }_{n,1}^{(1)}& \dots & {\theta }_{n,m}^{(1)}\end{array}\right]}^{\text{T}}}}\underset{\mathbf{X}}{\underbrace{\left[\begin{array}{c}{\mathbf{x}}_1\\ ⋮\\ {\mathbf{x}}_n\end{array}\right]}}+\underset{{\vec{b}}^{(i)}}{\underbrace{1{\left[\begin{array}{ccc}{\theta }_{0,1}^{(1)}& \dots & {\theta }_{0,m}^{(1)}\end{array}\right]}^{\text{T}}}}\\ & ={{\mathbf{W}}^{(i)}}^{\text{T}}\mathbf{X}+{\vec{b}}^{(i)}\end{array} \end{gathered}$$