System Design · Disney · Medium
Raw transactional records arrive in S3. Before they can be served from a feature store, they have to go through batch processing that cleans them up and aggregates them. Design this as a layered data pipeline. Your design should cover: The layers: how the pipeline is organized from raw S3 data through to the feature store. The transitions: what processing happens as data moves from one layer to the next. Data quality checks: how the data is validated along the way. Feature…