Dynamic Batch Inference

Dynamic Batch Inference Coding Machine Learning Engineer Problem Statement Build a Dynamic Batch Inference Engine that groups multiple token-generation requests into efficient inference batches. The exercise models, in simplified form, the scheduling behavior used by production LLM serving systems. Use the following simulated language-model interface: Implement BatchInferenceEngine with these capabilities: Receive generation requests together with completion callbacks. Group…