> They accept an input vector x and give you an output vector y. However, crucially this output vector's contents are influenced not only by the input you just fed in, but also on the entire history of inputs you've fed in in the past.
If it helps, you can think of a RNN as being analogous to a finite state machine. But instead of a single discrete state, it's a continuous, high-dimensional vector. That has the extremely important effect that the output is a continuous function of the input, which is necessary for training using gradient descent.
http://hackage.haskell.org/package/machines-0.4.1/docs/Data-...
> They accept an input vector x and give you an output vector y. However, crucially this output vector's contents are influenced not only by the input you just fed in, but also on the entire history of inputs you've fed in in the past.