Algorithm · ByteDance · Medium
Task Write the forward pass of Multi-Head Self-Attention. The original post mentions only Multi-head attention and gives no interface or constraints. The interface, simplifications, limits, and tests below are supplementary practice assumptions and are not recovered interview details. The inputs are a sequence matrix X of shape $$n \times d$$, four projection matrices W_q, W_k, W_v and W_o, each of shape $$d \times d$$, and an integer num_heads. Project the sequence into…