Design a Test Orchestration System

Sketch the design of a service that orchestrates tests across an engineering organization.

The service must let users define several kinds of tests, decide when those tests should run, run them on whatever compute is available, and keep track of what happened. The prompt is deliberately open-ended, so you are expected to ask clarifying questions, lay out the assumptions you make, and propose an architecture that holds together.

Your design has to cover at least the following areas:

• The way tests are defined, how they are versioned, and how they are configured.
• How a user launches a test on demand, and how a recurring series of runs gets scheduled.
• How work is handed off from the service to executors or workers.
• Where and how run status, logs, artifacts and results are kept, and how they can be queried afterward.
• How the service copes with failures and retries, enforces concurrency limits, and scales as load grows.
• What set of APIs and data models the service exposes.

Interview Notes

This question assesses system design and backend engineering ability, spanning distributed orchestration, scheduling, scalability, reliability, operational monitoring, API design, and data modeling for a test orchestration service. It sits under System Design and in adjacent territory such as distributed systems and CI/CD platforms.