Broadcasting (Tensor)

linear-algebra

Definition

Broadcasting (Tensor)

Broadcasting is a mechanism in tensor operations that allows arithmetic between tensors of different, but compatible, shapes. Whenever one of the shapes has a size of 1 (or the dimension is missing), that tensor is conceptually “copied” along that dimension until it matches the other array’s size. This eliminates the need to explicitly repeat or tile the data, while letting you write compact vectorised code.

Example: Let $A\in F^{5\times 1}$ and $B\in F^{1\times 5}$ be two tensors.

Intuitively, adding $A$ and $B$ together would result in a shape mismatch. However, with broadcasting:

• $A$ has shape $(5,1)$: it can be expanded along the last dimension to $(5,4)$
• $B$ has shape $(1,4)$: it can be expanded along the first dimension to $(5,4)$

After that automatic expansion, both match shape $(5,4)$ $(5,4)$, and the elementwise addition is performed. This neatly yields all pairwise sums of rows from A and columns from B, without manually calling a “repeat” or “tile” function.

Broadcasting was introduced by numpy and adapted by PyTorch and other tensor libraries.

Example

Let $A\in F^{5\times 1}$ and $B\in F^{1\times 5}$ be two tensors.

$$\begin{array}{rl}A& =\left[\begin{array}{c}1\\ 2\\ 3\\ 4\\ 5\end{array}\right]\in F^{5\times 1}\\ B& =\left[\begin{array}{cccc}10& 20& 30& 40\end{array}\right]\in F^{1\times 5}\end{array}$$

To add $A$ and $B$ together, $A$ and $B$ are expanded to have compatible shapes:

$$\begin{array}{rl}\text{broadcasted}A& =\left[\begin{array}{cccc}1& 1& 1& 1\\ 2& 2& 2& 2\\ 3& 3& 3& 3\\ 4& 4& 4& 4\\ 5& 5& 5& 5\end{array}\right]\in F^{5\times 4}\\ \text{broadcasted}B& =\left[\begin{array}{cccc}10& 20& 30& 40\\ 10& 20& 30& 40\\ 10& 20& 30& 40\\ 10& 20& 30& 40\\ 10& 20& 30& 40\end{array}\right]\in F^{5\times 4}\end{array}$$

Now, both can be added elementwise:

$$A+B=\left[\begin{array}{cccc}1+10& 1+20& 1+30& 1+40\\ 2+10& 2+20& 2+30& 2+40\\ 3+10& 3+20& 3+30& 3+40\\ 4+10& 4+20& 4+30& 4+40\\ 5+10& 5+20& 5+30& 5+40\end{array}\right]\in F^{5\times 4}$$