Run org-safe online experiment for recommender

Design an online experiment to assess the file recommender in a production setting spanning multiple organizations, where collaborators may affect each other's behavior. Identify the primary metrics (such as file open-through rate and time-to-open), secondary or business metrics (productivity proxies), and guardrails (latency, error rate, privacy incidents, access denials). Select the randomization unit (organization, team, or user level) and justify the choice to reduce spillover; describe bucketing, stickiness, and holdout groups. Calculate the necessary sample size and minimum detectable effect under clustering/ICC assumptions; choose a variance reduction method (CUPED or stratification) and a sequential monitoring plan with alpha spending. Outline the ramp schedule, controls for novelty and carryover effects, and how interference will be detected. Specify the logging required to reconstruct exposure and attribution, along with a difference-in-differences fallback when only partial randomization is feasible. Describe the stop/ship criteria and how to protect against Simpson's paradox across tenants and roles.

Overview: This question tests a data scientist's skills in online experiment design, causal inference, clustered randomization, metric definition (primary, secondary, guardrails), sample size and MDE calculation, variance reduction and sequential monitoring, instrumentation, and privacy-aware logging. This question is drawn from a data scientist interview experience.