MLP Compute-vs-Memory Bound Analysis

Fused GELU Linear Layer Roofline Bound Medium · Computer Architecture, Performance Modeling · N/A · Use the roofline model and check that the compute/memory crossover is near B = 108 A feed-forward block computes Y = GELU(X @ W + bias) + residual, where the model width is D = 4096. Assume W is a square 4096 x 4096 matrix stored in BF16, and that X, residual, and Y are also BF16. The weight matrix is read from HBM once per invocation, while the bias addition, GELU, and…