Compiling...
In the previous note, we discovered that standard "Vanilla" RNNs suffer from the Vanishing Gradient problem. Because they overwrite their hidden state (memory) completely at every single time step, they physically cannot remember information from early in a long sequence.
If an AI reads a 100-word paragraph, by the time it reaches word 100, it has completely forgotten word 1. To fix this, researchers had to redesign the internal structure of the neuron.
