System Design · Disney · Hard
Design a data pipeline that ingests and processes clickstream events at a volume of one billion events per day, and that keeps up with that volume without falling behind. Follow-ups Handling lag: How would you detect and respond to the pipeline falling behind? Be ready to discuss SLA and performance monitoring, as well as autoscaling. Late-arriving data: Some telemetry events show up late. How do you handle them while still producing correct daily aggregates? Deduplication:…