Compare Transformers and RNNs and Derive Attention Complexity

Overview: Contrast Transformers with RNNs and work out the complexity of dense self-attention. Break apart projection and score costs, training parallelism, generation caching, long context, and streaming trade-offs.

Read the full Data Scientist interview experience this question came from.