System Design · Amazon · Medium
Consider a layer that applies a linear projection, a GELU activation, and a residual addition: $$Y = \operatorname{GELU}(X W + \text{bias}) + \text{residual}$$. The model dimension is $$4096$$. The target hardware provides $$1\ \text{TB/s}$$ of HBM bandwidth, $$25\ \text{MB}$$ of on-chip SRAM, and a peak BF16 throughput of $$100\ \text{TFLOP/s}$$. Evaluate two batch sizes: $$B = 1$$ and $$B = 256$$. For each batch size, calculate the compute time, the memory time, and the…