Advantage

• It can model sequential data in a much more natural way.
• Can process any length input
• Computation for step t can (in theory) use information from many steps back
• Model size doesn’t increase for longer input
• Same weights applied on every timestep, so there is symmetry in how inputs are processed.

RNN Disadvantages:

• Recurrent computation is slow
• In practice, difficult to access information from many steps back

General ideas

• flexible: multiple input, multiple output
• Can we use more historical information?
• Can parameters be shared?
• sequencial data: dynamic computational graph

Features

can regard it as a layered net that keeps reusing the same weights and structure

Many-to-many