Design A VM Bandwidth Rate Limiter

A virtual machine bandwidth rate limiter is a host-level component in a cloud environment that governs the network throughput each VM can consume. Instead of counting discrete API calls per second, this system shapes continuous flows of bytes at the hypervisor layer. Each VM receives at least its promised base bandwidth, the total never surpasses the physical NIC capacity, and unused bandwidth is automatically reallocated to VMs with higher demand. The system acts as a bandwidth scheduler, not a request counter.

[!NOTE] This is not about classic API rate limiting. There are no distributed counters, no sliding time windows, no HTTP 429 responses. The challenge is live traffic shaping over a shared physical link. Every packet-scheduling decision must execute in microseconds, and the fairness guarantee it maintains is host-wide, adapting whenever any VM's traffic demand changes. If your first thought is “assign a token bucket per VM and drop excess,” keep reading to see why that wastes half the host’s capacity.

Requirements

The core problem is fairly dividing one host’s NIC capacity among many VMs without wasting idle bandwidth or allowing a single noisy VM to degrade others.

Functional requirements

• Enforce a host-wide bandwidth ceiling equal to the physical NIC capacity (e.g., 10 Gbps) across all VMs.
• Guarantee each VM a configurable minimum bandwidth (e.g., 100 Mbps) that remains honored even when all VMs are contending.
• Shape and throttle both ingress and egress traffic for each VM independently.
• Dynamically redistribute idle bandwidth to active VMs. The system must be work-conserving: if total demand is below host capacity, no VM should be throttled.
• Provide an operator API for managing per-VM bandwidth policies (minimum guarantee, optional maximum cap, weight).
• Enforce admission control: reject policy configurations where the sum of all VM minimum guarantees would exceed the host NIC capacity.
• Report per-VM bandwidth usage metrics for monitoring and capacity planning.

Out of scope (below the line)

• Distributed API rate limiting (Redis counters, sliding windows, HTTP 429 responses).
• Multi-host bandwidth coordination, cross-host traffic engineering, and SDN controller-driven allocation.
• Application-layer QoS: per-flow or per-application prioritization within a single VM.
• VM live migration bandwidth and billing/metering.
• Network function virtualization (NFV): firewall, NAT, load balancing within the shaper.

Non-functional requirements

• Fairness: Surplus bandwidth is distributed proportionally by weight among competing VMs. No VM starves while others have idle capacity.
• Low-latency enforcement: Packet scheduling decisions occur in microseconds. The data plane must not add perceptible forwarding latency.
• Noisy-neighbor isolation: A bursting VM must not push others below their guaranteed floors.
• Work-conserving: If total demand is below host capacity, no VM should be throttled.
• Dynamic responsiveness: Borrowed bandwidth is reclaimed within sub-second convergence when the lending VM resumes transmitting.
• Observability: Per-VM throughput, throttle events, and queue depths are visible for debugging and SLA enforcement.

Back-of-the-envelope estimation

The numbers that most shape the design are the per-packet decision budget (~1.2 us) and the 50/50 split between guaranteed and surplus bandwidth. The microsecond budget means the scheduler cannot make remote calls or take locks on the packet hot path. The 50% surplus means work-conserving redistribution is not an edge case—it is half the system’s value.